Re: Someone Else's Computer certification
It really depends who you speak to and there are a lot of factors at play depending on the type of site / service you're planning to run.
I personally prefer the manual scaling approach because it allows finer control over costs and despite having to pay for a person to manually scale things you will save money in the long run.
I have several clients with large sites that I manually scale. Sometimes I have to be up and about a 1am - 2am to scale things back down, and sometimes I have to be up at 6am to scale things up...but using previous information captured we are able to predict with a decent degree of accuracy what we need to scale to and from.
One of my clients I snatched off a firm that insisted on auto scaling everything, they were so adamant that autoscaling was the way, that they went completely off the fucking wall when I was asked to step in to try and reduce costs...they called me all kinds of names...essentially calling me an old fashioned dinosaur (but not as polite). I've had this client on and off for about a decade, but I've recently had them back for about 18 months, and in that time, I've reduced their AWS bill by around 75%. Which comes to approximately £90,000 a year. I cost them around £12,000 a year. It doesn't take Stephen Hawking to calculate that even though they are paying me double what the previous guys cost, who left the scaling to happen automatically, they are still saving a whole bucket load of cash.
The key thing though is proper testing...see not all of that saving was due to timing the scaling more effectively, I also trimmed a decent amount of fat by properly testing the site, finding the heaviest SQL queries and optimising them. I also tested out various tiers of EC2 instances and RDS instances to work out where the best bang for the buck was when choosing tiers to scale to. The previous guys were going straight to m5.12xlarge on all the RDS instances each weekend for 48 hours because they were lazy as fuck...with some testing, I was able to work out that the performance on the 12x was actually worse than the 8x, 8x is roughly half the price of 12x...not sure why, but I suspect the 12x instances run on different CPUs or something...I was also able to spot that the bottleneck on the database was storage...the original guys had only allocated a small amount of storage on bog standard GP2 storage, which meant no matter how much RAM or CPU you threw at the database, you would still be limited by IOPS. So I optimised the storage as well to bump up the IOPS (which is remarkably cheap compared to just throwing more RAM and CPU at the problem)...this allowed me to drop the RDS tiers another notch, which essentially halves the price again.
After all this optimisation, the scripts the previous guys wrote to "spin up more EC2 instances when needed" rarely kick in, because the CPU usage of the EC2 instances remains below the threshold that was set (60%) before additional instances are spun up...so we no longer stray into having tons of extra EC2 "app server" instances running. We end up with one extra box, at most.
Circling back to the "it depends" comment...never let your software developers determine the infrastructure that their software runs on. They will always throw RAM and CPU at the problem, they will rarely blame their own code or lack of optimisation. Always find an infrastructure specialist that also has some reasonable software development skills so that they can overlap with the dev team and understand the software...yeah I know, we are rare...but we are out there...yes, we cost quite a bit of money...but that cost is often considerably less than the money you will save and often we will charge you based on what we think we can save you. The kind of clients I look for are spending over £100k a year on AWS, turnover at least twice that and I look to save about 50% in costs, of which I take 10-20%. I aim to eventually pay for myself out of your savings, so that I essentially cost nothing.
At worst, someone like me will result in costing exactly the same as the "shitty guys" but the user experience on your website will be far better and result in less complains, downtime and problems and at best will slash up to 80% of your costs. At worst, it will take 6 months for you to start seeing the savings, at best you will get savings immediately simply by moving to a manual scaling model.
Automatic scaling can work well, but you will be paying through the nose for it up front, and you won't know if the resulting hosting bill is optimal or not. A lot of the time, you can end up with an autoscaling setup that is still profitable, so you might end up with no reason to optimise it, on the face of it, but a good infrastructure guy will still find you savings and make your business even more profitable...they are worth paying for.
Not only that, but if you are looking for further investment to take the business forward, an infrastructure guy can help you properly describe the functionality of the site, the layout of the infrastructure and the usage patterns of the site to help you make forecasts to take to potential investors...a software developer will never be able to do that.
Modern Software Developers come off the University production line, they are ten a penny. Infrastructure engineers like me typically don't...they are the guys that build stuff on the weekend for fun and have done for years because no matter how much you spend on going to University, you're never going to get your hands on a wide range of kit to experiment with...the only way you can build a decent knowledge in infrastructure, is to spend money buying second hand kit off eBay, messing with it, building actual infrastructure with it, then flipping it back when you're done.
I think most of us in a certain age / skill bracket, probably got into the industry this way.
This all said, not all Uni grads are fucking hopeless, they just don't know any better...I've taken quite a few "fresh from the factory" folks and turned them into machines...but you have to catch them early...because once they've been out there for a while, playing the same fucking dead song, it becomes impossible to tune them and they end up at "development agencies" churning out badly optimised, expensive bollocks.
Digital Agencies out there, if you want to keep your clients beyond the last milestone of your development "roadmap", you need infrastructure guys...the very guys waiting to pounce and steal your clients as soon as you launch the product...you are incredibly vulnerable because CEOs talk to each other and people like me talk to CEOs...so it doesn't take long to sniff out badly run projects then swoop in for the kill...arguing that your large team is better than a one or two man band is easily drowned out by saving £100k-£200k a year and you can't compare a team of 10-20 juniors with a two man crew of highly experienced engineers. We will stomp you in to the mud every time.