Honestly there is a range of things that could cause your servers to startup slowly, but these are ARK server related, not ASM.
Every server is different and every environment is different.
My only suggestion would be to create a brand new vanilla server, and start it up. No mods, No custom settings. Get a base line on how long it takes for a plain vanilla server to startup and shutdown on your hardware.
Then try introducing the same mods you have on that server and see what happens.
When you are doign this, try to shutdown all other servers, so you can do proper bench marks.
ARK Server Manager and it's creators are in no way affiliated with Studio WildCard or its partners.
ARK: Survival Evolved™ and its related images, trademarks and copyrights are the sole property of Studio Wildcard and or its partners and affiliates.
The ARK Server Manager is provided free of charge for all legal use. The source code is freely available on GitHub. See the Release Information forum for more information.