Map Server 2013 (AKA AIMS), now has a built in performance monitoring tool! And it's really simple to use and act on:
In the Site Administrator, click on Performance Monitor:
Click OK, then look at your results.
This map of mine only too a little over a second to load (1,025 milliseconds), which is pretty good. (and it should be - this is all on my desktop). I see that layers were 81% of the total, and that two similarly named layers (TOWNS_POLYM and TOWNS_POLYM (1)) are the bulk of that, at 39% and 25%, so there may be room for improvement there.
Checking the layer in Infrastructure Studio, I see that both are the results of a publish from Map 3D, and that other than the feature class, they are visually identical. To test, I removed the second layer from the Map (it still exists as a layer if I need to replace it) and reran the same test: