Developer pockets $2M in savings from going cloud-free The web software biz that decided to exit the cloud after racking up a huge bill says it has saved almost $2 million in its first "clean year" after making the switch to on-prem, and has already recouped the costs of the extra hardware it needed. A couple of years ago, 37signals was aghast to find it had run up charges of $3, …
Where are your cost alerts and someone to police cloud usage? I'm a firm believer in "horses for courses", you choose cloud, hybrid or onprem, but if you open cloud accounts to one and all then it's just giving an unlimited company credit card to all employees, don't complain when someone does something stupid with it!
We have sandboxes for our devs, max spend limits, we track costs and usage every 15-20 mins, we even restrict VM usage to max 2 CPUs and 16GB memory, max 2 machines per person. We don't allow databases, we dont' allow unlimited storage. Only about 6 people in the whole company are allowed into the test and prod cloud accounts out of 700 employees. We have hourly checks for costs exceeded, daily round ups, name and shame on spending, you spin it up and you're name will be on display for all to see so choose wisely.
Only 40 people in the company get a company credit card, like wise we don't let everyone in IT play in the cloud toybox!
It sounds like hell. Have you considered not having computers at all? Going back to pen and paper?
The problem for SMEs is that the cloud discounts are what makes cloud affordable for standard operations. Only the big players get the big discounts. Everyone else spends their lives chasing random cost instead of making new things (or fixing existing things). My own small cloud application (a few hundred TPS) costs next to nothing to run on GCP native services, mainly because it is part of a large GKE cluster and there are swinging discounts. If you're not operating at scale and you just lifted and shifted, you're almost certainly not going to get cost benefits.
Just moved a small low usage application off AWS to an independent, the typical £500~600.pcm AWS cost has been replaced by a flat rate £200 pcm.
Additional, support and maintenance savings will accrue as the migration took it off AWS services and onto our own LAMP platform (MariaDB instead of MySQL), which we can better manage the patch and version upgrades.
Wouldn't it be way cheaper to buy the user's an extra computer (128gb ram instead of 2 instances of 16gb each), and allow databases as well. Storage is cheap, CPU compute cycles are cheap, you'll save on network costs since it's local, you'll gain flexibility ... Seems like a decent investment.
And run some form of virtualisation on it so they can use it as multiple machines.
Actually, maybe get an even better machine locally and allow several people to run virtual machines on it - centralise that resource.
It's starting to sound a lot like a development server.
"We don't allow databases"
Screams senior management and IT making software architecture decisions to me.
"We have hourly checks for costs exceeded, daily round ups, name and shame on spending, you spin it up and you're name will be on display for all to see so choose wisely."
Presumably you have the same controls in place for senior leadership and their company credit cards, right?
Imagine treating your highly educated and skilled workforce like children, and bragging about it.
I've no doubt that in the same organisation you'll talk about innovation and discrupting the industry, while actively kneecapping your staff to a 1998 spec dev machine and policies that actively discourage such things.
What company is this? Just want to actively avoid them in future.
Businesses that run large, relatively stable, predictable workloads are likely to save money on-prem, long-term.
But they do sacrifice a degree of agility. If their customer base was to triple over the next three weeks, could they source, configure, and install enough hardware to cope with the additional load? Unlikely.
If they wanted to develop a new product with as yet unknown resource requirements, would they have enough spare capacity to do that, without affecting their production workloads, and without having to down tools to wait for new hardware to arrive? Because I've experienced this exact situation when relying on on-prem tin.
37Signals is literally at one end of the spectrum. The counter-argument at the other end of the spectrum is the UK charity Comic Relief, who provisioned serverless infrastructure that facilitated the processing of tens of millions of pounds in donations from hundreds of thousands of members of the public, for less than £100; their previous on-prem solution was IIRC 1000x more expensive, and sat doing nothing for the vast majority of the year.
Everyone else is somewhere in the middle. You don't have to run all of your workloads in the cloud; you don't have to run them all on-prem. There is a middle ground, but anti-cloud zealots like DHH seem to forget this and present their own experience as "look, this is true for us so cloud sucks".
"But they do sacrifice a degree of agility. If their customer base was to triple over the next three weeks, could they source, configure, and install enough hardware to cope with the additional load?
...
Comic Relief, who provisioned serverless infrastructure that facilitated the processing of tens of millions of pounds in donations from hundreds of thousands of members of the public, for less than £100"
Comic Relief hold an annual events on dates decided well in advance. If they depend on agility to plan for that they're doing it wrong.
I have, in the past, worked with marketing who probably did hope for something like a tripling of turnover in three weeks. Having failed to work out that their campaign had a small flaw they were disappointed. (It was intended to let them sell direct instead of through distributors; the customers needed to arrange an installion, however, and the installers were also the distributors.)
You and the other reply are missing the point about Comic Relief. It's a very bursty load.
That big burst of load for one day, if you went "on premises", requires buying a lot of servers sized for the max load, and having them sit idle the rest of the year. That is expensive.
Renting server capacity in the cloud is a much cheaper option. And using one of the cloud services such as serverless, which automatically scales up and down as needed, is an even better option because you don't need to predict your peak load as accurately.
As with everything in life, there is not one solution for everyone. Not cloud, and not on-premises. Use the best solution for the problem you're trying to solve.
> 37Signals is literally at one end of the spectrum. The counter-argument at the other end of the spectrum is the UK charity Comic Relief
Odd spectrum, I suspect if you ignore the annual telethon, their normal day-to-day operations load is boring and predictable. Thus, just like 37signals there will be few benefits from running this load in the cloud, although with charity pricing, I expect they will be a big user of 365 and cloud-based application as a service systems for accounting, customer relationship etc.
Also I suspect the annual telethon is such that they can get their cloud provider to donate the costs…
"But they do sacrifice a degree of agility. If their customer base was to triple over the next three weeks, could they source, configure, and install enough hardware to cope with the additional load? Unlikely."
If you have the right people, this situation can be handled on prem. I've seen Cloud scale ups go pear shaped. Nothing is guaranteed.
Remember that cloud computing is a for profit business, and they certainly won't settle for making 10 cents per dollar.
The "cloud" is just using someone else's hardware. There really is no benefit for an organization with a functioning brain cell in management to go cloud.
But functioning brain cells is a big ask for most corporate management...
For most organisations with some critical mass (and a baseline workload), cloud is going to be more expensive for that element of what they do, but for smaller organisations, who can't fill a modern server, cloud likely is a good shout.
For bursty or massive spike workloads, cloud can be a great fit for elastic capacity on a temporary basis, but the MO needs to be "kick the arse out of it, then get back home" as soon as the workload drops.
To my mind, sizing on prem for the "once in a year" workload if you're Wimbledon, or putting all workloads in the cloud just because you're planning to run a big marketing campaign once a quarter, are equally bad approaches.
You would think that cloud computing vendors would benefit massively from economies of scale - given the scale at which they operate. Those economies should come from a whole range of different sources from the purchase of hardware and services in bulk, through the ability to deploy massive automation, to the better efficiency resulting from being able to use one customer's idle time for another's peak need.
It's not clear whether in practice it works out like that for the vendors - but it doesn't seem to work out like that for their customers who mostly seem to be paying a premium for "flexibility" they mostly don't need and for administrative convenience (not employing their own staff) - which suggests the customers of their customers are paying a premium for a lack of genuine competition.
'You would think that cloud computing vendors would benefit massively from economies of scale"
They do - that's why AWS and Microsoft Azure make around $26bn and $35bn respectively per quarter and growing.
Why isn't it cheaper? Because they are still building their monopolies to the point where they are the only game in town for low latency, high bandwidth compute services
My current employer have been doing a costing exercise to see if on prem could save us money versus our AWS setup. We don't have "bursty" activity, just a steady steam of data processing and web traffic, so estimating the hardware requirements is straightforward. It also helps that every AWS feature we use can be replaced with an on prem equivalent - PostgreSQL, OpenSearch, Redis and storage to replace S3 essentially. Had we bought into the serverless snake oil like Lambda or other dubious AWS features I'm not sure we'd be considering a transition.
You make an excellent point, which I'm always surprised people miss - don't get locked in.
You'd have thought that IT had enough examples of people having buyers' remorse about lock-in being painful/expensive ***cough, ORACLE, cough***, that we'd have learned to stop doing that, but far too many people don't appear to pay enough attention to history and therefore are happy to take the short term easy option and suck it up/regret at their leisure when they're bent over by the vendor.
Pink fluffy handcuffs are still handcuffs......
IT people have experienced those examples. You can recognise them. They're the ones burying their heads in their hands when the decision makers meet the salesmen.
What always surprises me is that the manglement are so often snake -oil merchants themselves so why can't they recognise the same thing when they see it?
...the value of somebody else handling the operational complexity.
Take S3, he says they want to do this themselves? I'll assume he's not actually talking about multi-regional - even maybe multi-zonal S3?
So let's say you do this only in one location... do you use a SAN? Probably not because then you have another expensive vendor contract you didn't want.
So lets implement it as a service on utilising storage across your on-premise servers - I take it you've got all the servers RAID configured properly and monitored - or are you replicating across multiple machines instead?
Either way hardware fails, so you've got your on-call processes sorted, you've got your contracts with your hardware suppliers locked down because you know you'll be replacing failing disks with some kind of regularity.
You've either trained the datacenter staff or you have staff present near your datacenter location, and their access sorted to be able to go in and replace those failing disks.
Depending on how much you're leaning on the on-premise provider (in which case how much money are you really saving?) you need an entire ops team likely with a different skillset from your existing cloudops team.
"I'll assume he's not actually talking about multi-regional - even maybe multi-zonal S3?"
You can assume what you like, but considering his company provides SaaS apps as their product, and they tend not to work well when the infrastructure is down, I think the implied assumption he's going single site is bat-shit crazy.
He does actually cover lots of the concerns you raise in his article, and some others in the comments to his LinkedIn post.
They are still saving millions per year after paying for smart hands, architecting for in-site / cross-site redundancy and full support from vendors like Dell and Pure (so likely no disks to change - it will all be solid state, so most likely not failing with "some kind of regularity").
Finally, they didn't change the number of people supporting the infrastructure - so they're either super human, or the state of managing infrastructure has improved since you last did it.
Your "let's say" thought experiment is amusing, but I'm paying more attention to the guy who is already doing it for real and whose company is pocketing seven figures of savings per year (even before he does the same on the S3).
How so? If you do it yourself you have an extra layer - sure if you're on a cloud service like S3 you still have secure and monitor it but you're operating at a much higher level of abstraction.
You do it yourself, with the support of Dell, you still have to secure and monitor it, but you have to monitor more of the stack - because you're responsible for it, you'll also be responsible for much more of the implementation, such as things like fail over and replication.
Without reams of guarding in our codebase, the time the alert has come through (if indeed it ever does - most transient outages go unreported, except within our applications), the damage has been done.
But now we have retry loops everywhere we now get pointless alerts that have been resolved by one or more retries. So we can't win.
Oh - and to effectively monitor a monitoring system, you need correct IT staffing levels, and engaged staff. We have neither.
"somebody else handling the operational complexity"
That's the dream, isn't it? No complexity anywhere, or as Hans says in _Die Hard_: "We'll be sitting on the beach, earning 20%." And as everybody knows, there are no mosquitos in paradise. Whereas in the real world, managing suppliers is just as much work as managing employees. The complexity moves around, but it doesn't go away. This must be true, because if it were possible to outsource all the complexity, then anybody could be in the same business as you. Er, wait, that's what MBAs think already. Never mind, puts hand down.
I see some people still haven't learned - RAID is not a back up solution, ever.
Multiple independent boxes are cheap and easy to build. Segmenting the storage to different drives on each box should be part and parcel of designing the applications AND the hardware.
Anyone can build a couple of 100TB computers today with multiple fast network cards. Problem is, you'll actually have to treat the people running the show with a modicum of decency and respect - something management is not accustomed or predisposed to do.
As per their previous announcement of their savings, it's extremely misleading. Sure they've saved a load of money, but the headline figure being quoted of "how much they cut their cloud bill" is only half the story without knowing the extra costs spent. I can chop my cloud bill to £0 by buying loads of physical gear, the infrastructure and the people to set it all up. Whether or not that has actually saved me money can only be ascertained by comparing the savings with the costs.
This time around at least we get a little bit more detail about the costs - $700K's worth of servers. And sure that's a one-off cost (at face value) compared to the $3m or whatever in cloud costs.
But there's no detail of the time spent to spec and actually purchase those boxes. To install them. To keep them running. To repair them. To pay the staff to do all of that. There's no mention here of their data centre costs - the power and bandwidth which I'm assuming was part of the AWS cost but not mentioned here at all for their new environment.
And they claim they'll sweat the hardware for the next 5-7 years? That surprises me if so (and they'll likely have replacement costs during that period) but also bear in mind that with the cloud cost you've not sunk that $700K - aside from any long-term contract you may have signed up to, you can chop and change your cloud environment.
To be clear, I'm sure they did save a load of money by doing this. Relatively stable requirements with little need to scale is going to be cheaper on the whole with your own kit. But we simply do not have enough facts here to have a clear picture as to the savings - and they are certainly not the headline figure.
A company like this isn't going to spend time specing anything. Neither are they going to spend time repairing it.
The vendor is doing that.
Any vendor at this level is going to be able to take a dump of aws usage and function and give you a hardware spec. If the hardware goes wrong, the vendor is dealing with it. Everything else they will already have the people for.
AWS doesn't give you an IT team. You can buy consultancy at a price if you want but even if everything is in AWS you still need experienced network and firewall staff to drive it, same for AWS specific functions, OS install and configuration, specific product management like load balancers. All the people you need for on-premises you still need for AWS with the exception of the physical hardware lugging, which you get the vendor to do as has been mentioned.
What cloud gives you - they are all the same in this respect - is an easy way in for perhaps inexperienced staff to build a system which appears to be low cost but after a few years will be costing a lot more than anybody estimated. If you've got experienced staff it's going to be cheaper to build it yourself in a co-lo and use AWS perhaps as the DR failover. That would be my choice.
Or for very large enterprises that need global locations, like Spotify, but even for them, I bet they have a whole team just to keep pressure on cost and monitor exactly what's going on and the cost of AWS will be a board level topic on a regular basis.
Can anyone share any outstanding examples of great support from their cloud vendor when the "someone else's computer" bit went wrong and it was fixed quickly and competently?
Genuine question because my own experience is that it can sometimes take days to reach anybody with real technical knowledge (and this is via a large corporate contract). You can colour me cynical but I'm mighty weary of the marketing promises about what the cloud can actually do.
It seems to me that non-technical executives take great comfort in believing they've outsourced responsibly to cloud vendors only to find that when the proverbial hits the fan, they'll be harassing the poor onsite team who will have little to no option but to wait however long the cloud vendor takes to fix it as they have no control over it.
that cloud providers are going to invest in.... guess who going to pay for those?
If AI loads keep increasing cloud services inevitably become more expensive if only from demand/supply economics.
For quite a few firms with very conventional fixed workloads a smaller cloud provider (not in the GPU/AI game) or on-prem may well be considerably less costly than the global cloud providers.
I suspect there is a market for the supply of managed on-prem resources - pretty much the way IBM mainframes were once supplied.
SaaS vendor shifts source of cloud services from AWS to onprem "cloud"
Will their customers notice any difference?
What are the real benefits of "cloud" AKA "server-farm-in-unknown-jurisdiction"?
Low start-up costs, scalability, resilience and performance?
In theory. IRL those have to be planned in, which is a non-trivial to get right.
These guys have relearnt the old mainframe lesson.
Who owns the data owns you.
If you've got a global company supplying it IE Amazon, Oracle or Microsoft, can you be sure they haven't migrated your workload (and its data) to some third country? Cheaper rates, working hours better aligned etc.
Of course as they are all US based and THE PATRIOT Act still applies nothing you hold on them will be private from virtually any Federal official who asks for it*
*Exactly as the data would not be private to Chinese officials in China under the corresponding legislation.
Business realises that paying a middleman to do the same job you can do in house is less profitable for you and more profitable for them. Shocking.
Cloud is the most ridiculous over-selling of renting a few remote dedicated servers, and then being charged per API call, in effect.
I'm following a thread on Twitter about web developers who are only now finding out that a $5/month VPS can do what they've been paying an absolute fortune to do. Want redundancy? Buy three of them, from three different providers, in three different continents if you like.
For personal use, I'm chugging along with a £20 a month dedicated server, which I've done for over a decade. It does EVERYTHING. And there are no "usage" charges whatsoever. An unmetered connection. And I can literally go "Oh, a new version of Factorio... let me start up a server for it" and that's all I have to do to run it alongside everything else that's running on that machine. It's everything from my "GMail" to my "Google Drive" to my Plex to my TV etc.
And in work, I find the same. Most place just don't need three-continent-redundant, zero-latency, huge amounts of servers queued ready to go with a thousand containers on each.
If your company is vaguely technically and you've gone to cloud and somehow skipped the part in the middle where you just rent or colo a few decent (£500/month+) servers BEFORE you pay AWS or whoever that same kind of money... it makes me wonder if you really should be doing that at all.
Cloud is the right thing sometimes, on-prem is the right thing sometimes, hybrid is the right thing other times.
But if you build it for on-prem then blindely pursue a cloud first strategy throwing everything into the cloud without speding a time on dapting it to a different pricing model you will only guarentee the worst of both worlds.
Not only are cloud costs generally now much higher than hosting it all yourself on-prem, but the lack of control and reliability is costly too: at the utterly misguided circling-the-drain Switzerland-based insurance firm I work with, we have two cloud providers. Both have different patching schedules and forced OS upgrade cycles - essentially meaning regular downtimes and odd breakages, both are entirely self-service (so no customer service or help whatsoever), so every dev team - the few remaining who aren't actively job hunting - now has to be system admins too - a very different skillset. I can only presume both our cloud providers were chosen for their ability to provide hospitality to our CIO and her team - it certainly wasn't for value for money, reliability, responsiveness or customer service, anyway.
All our developers have become great friends with the retry loop due to the large amount off little transient network outages we now experience day-to-day - of course, the outages are not long enough to contribute to official downtime numbers (and hence refunds), but they certainly contribute to both perceived and actual flakiness. Everything takes longer to run and to do. Everything.
This has all been compounded by our senior IT management sacking all our specialists, and outsourcing as much as possible - after all who needs skills and in-house knowledge, when you have Microsoft? It's all going about as well as you'd expect, and I'm making careful plans to not be in the bus when it finally plunges over the cliff in the next year or so.
"You must be doing it wrong!" some of you will no doubt think, continuing to yourself "our cloud experience has been both endlessly joyous and entirely economical!" I'm not disbelieving you. But my firm has entirely handed over control and expertise for all their technical infrastructure to two third parties with seemingly no checks and balances. Hopefully yours hasn't..
Apologies for the whinge, and thank-you if you got this far.
Another issue with AWS or probably any cloud provider, is if they suspect you are spamming either e-mail or SMS using their IP addresses, they will cut you off. This happened to us, application level e-mails with occasional news updates, not spam, only going to addresses that had accounts on our system, authorised addresses with customers we had contracts with, AWS decided it was spam and stopped it sending mail. We had to jump through hoops to get it working again and in the meantime, had to use an alternative service. We now do it completely differently and no longer use AWS for anything.
If AWS think you're doing anything they don't like, they will stop it for you. You are on their servers. You can raise a ticket with AWS and firstly a bot will address it, then you might get a person responding but you won't be able to ring them to talk - that will only come later if they feel it's necessary and the problem is not resolved. You can't choose to escalate it ! Who wold you ring ? Do you even know which country the support team dealing with your issue is in today ? Will be a different country for another ticket.
If you want to tie your horse to a cruise ship when it's in dock, don't be surprised when the ship moves and nobody gives a sh*t about your horse.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
