@Jim
While I agree the single largest issue for most SA's *was* provisioning, I'll note that the idea of 'immutable infrastructure' is *not* that the (system never changes). It is an alteration in the way one thinks of infrastructure. If one thinks of the (Physical host/VM OS, baseline services i.e. DNS, auth, mail, logging, ntpd, network, storage, FS layout) as infrastructure, they *can* become immutable.
You put the (cilent/guest/application/db/service) into its *own* box, separate from the infra box, then automation starts to be massively effective. -- no, one does not roll things out across one's prod systems without testing, but at least once one starts thinking in the context of managing 8K or more systems with only 32 people (and I'm talking Director, managers, PA's OS, platform, tools network/storage team), you learn to box things up nicely, separate what is test/dev/qa/prod, what can be handled in a herd and what needs to be snowflaked.
While it has been a while, we flipped the switch on DNS servers, migrating from an old set of hosts, to a new fancy shmancy cluster of application servers for DNS management as a single change to just over 5,900 *servers* and some 12k desktops. In about an hour, including final validations. Automation made the system changes. *and* provided the validation results.
the *testing* rollouts, executed 5 or 6 times prior, on limited targets, gave us the comfort to do that. And automation made backing out the *tests* just as easy. Total time ? about 18 hours, 11 of which were spent on the paper work and conf calls to get the change approved.
I'll guess that you've never had the opportunity as an SA to point out that a vendor is using *default dumb* in installation instructions for their application. I've found *VERY* few application vendors that have *default intelligent* installations. To make your environment manageable, as an SA you have to be an utter prick and override the "Install to root" "Run as Root" "well we have to do this because we don't know" crap. And willing to take the time to fix it so that it does work once in the box.