Just because on-prem is cheaper doesn’t make the cloud a money pit With cloud costs expected to rise over the next year, you might be considering moving your applications back on-prem or to a colocation facility — after all, it worked for 37Signals. Earlier this year, the Basecamp project management and Hey email developer revealed the $3.2 million cloud hosting bill that spurred its decision …
Hosting your own hardware will always be cheaper in the long run than hosting someone else's simply because you're not paying the "someone else" tax, or do we think they provide service of the goodness of their (bleeding) hearts?
And don't try the "hired sysadmins are expensive" bullshit. If you think that then your product is pointless, waiting to be undercut by someone who knows better.
Sorry but I have to strongly disagree with you. There are plenty of ways that a Large Datacenter can operate significantly cheaper than housing your own hardware. It's called economies of scale.
Buying 1000 servers will cost you significantly less on a per unit basis than buying 1 server (I'm talking hardware costs here). Large Datacenters usually set up in places where energy costs are cheap, and can design their entire facility to minimise energy usage (on a per output basis), whereas for your own Server you will be paying whatever the local energy costs are, and what deal your company has accepted. You also will be unlikely to be able to modify your server room to a massive extent so it's usage will be significantly more than a dedicated facilities (again on a per unit basis). Large Datacenters also usually manage to scam better tax breaks/benefits from the areas they setup in, which means again they have savings you cant match. And then of course there is the personnel costs, unless your workload is absolutely optimal that one sysadmin is working 100% at capacity on just your server, you're going to have inefficiencies, they'll either be a bit underused, or be working on something else at your firm, so again you're paying full price for the sys admin. At a Large Datacenter, one sys admin might be administrating a dozen clients, therefore you're only paying for 1/12 of their time. So that will be cheaper.
It's the old case of manufacturing 1000 widgets in one go is cheaper than manufacturing 100 widgets in batches of one.
This is not to say a Large Datacenter will actually be cheaper or better for your specific usecase. I actually prefer having things in house. But to claim that in-house is always cheaper is just plain wrong.
You're missing the most important point, which is that economics of scale only really work in your favor if you're the buyer/operator. If you're just a customer however, it rarely does, as the bulk of any savings will simply go into the pockets of your provider/supplier.
So yes, large datacenters may well be able to achieve lower running costs but that doesn't mean they pass on the savings to their customers. A large cloud operator may well buy 1000 of nearly identical servers at hefty discounts (although the big cloud providers don't use standard servers but bespoke inhouse developed server systems, with manufacturing contracted out to ODMs, so it's not the same as a regular server), but they will still charge you close to MSRP (or the equivalent in leasing fees) because they can. As customer, you'll generally see little to nothing from that discount on your bill (also remember that the big server vendors like HPE and Dell already provide pretty good discounts to business customers even for low order numbers). And the same is true for any tax breaks, local incentives or other incentives the datacenter operator may get. They'll lower customer pricing only when absolutely necessary, and only far enough to stay competitive.
And as far as staffing costs go, you may well remember that even this short into the year we already had a number of Azure/MS365 outages caused by Microsoft staff making stupid configuration errors, which suggest that paying only 1/12th of their time I'm also getting only 1/12th of their attention (which is nonsense of course, because at the current going rates for cloud computing I'm paying a lot more than just for 1/12th of their admin's time, while getting a lot less than 1/12th of their attention).
The very fact that cloud costs have been very high and have been increasing further at an accelerating rate proves that services providers will forward little to no cost or efficiency savings to their customers (in case of Microsoft, they simply increase Azure attrition rates by simply killing off their on-prem software offerings).
Cloud is only cheaper for the service providers, not for their customers.
For any business contemplating whether to go on-prem or cloud, the cost savings the cloud providers see from their large-scale operations are completely irrelevant, the only thing that matters is how much they are charging their customers for their services. If that's more or less than what an on-prem solution would cost depends on the individual case but more often than not on-prem turns out to be cheaper (and comes with the added benefit of not being completely dependent on a nearly worthless SLA).
First, never underestimate the impact of u$ using their customers as their QA. If you think "Azure" when you say cloud, then--just don't.
Second, while it is correct that the primary benefits of scale accrue to the providers, if there is any meaningful competition at all, a significant amount of that savings will roll down to the customer. If competition is fierce (and it is not), almost all will.
Third, AWS & GCP are set up to provide something that you simply cannot do on your own until you are nearly a household name--reliability. Even four nines is EXPENSIVE. AWS & GCP can give you all the tools for it in a few minutes.
If you don't mind having your website down all day on Black Friday, then you are likely not to need their services at a much lower scale than if you do.
We're moving off Heroku this year. It's going to be interesting to see just how much it is going to cost us to do all of the things that they are doing for us. My prediction is that our costs are going up. It will be worth it, though, because of the added control. That is, we will have a gain in functionality.
Economies of scale often actually don't apply to middle scale operations.
Yeah you get a discount but you also pay through the nose for solutions which can scale to millions of clients when you might only have a few thousand.
Amazon is going to allow you to scale almost infinitely but that means they're vastly over provisioning for moderate sized workloads.
The per core cost or per GPU cost of a super computer isn't going to scale linearly, nor are you going to see a discount over buying a standalone GPU.
You can buy an Nvidia card in a machine and it costs 1x. If you want 10x Nvidia cards in a hypervisor that can split up that GPU cluster and allocate dynamically to different VMs... Prepare to now buy an Nvidia GRID license, you also need to pay for nvLink adapters etc etc etc. Maybe you got a $100 off per GPU for bulk, but now you're paying $100 a year in licensing.
As the article states you might save on economies of scale for labor if you're small. If you only need one server a $100k a year for labor makes no sense. But as you get larger and larger you need more and more layers of management. Your accountants need accountants to handle the costs of the accounting department... Etc etc.
Small is inefficient. Big is bloated.
It's more nuanced than you cite - and as the article mentions, it depends a LOT on your business and computing needs.
If you are very small, then yes - hiring a sysadmin capable of keeping up to date with "everything" is going to cost you, disproportionately relative to the size of the task - with the result that "sysadmin" is usually tacked onto the normal job of whoever was last to shout "NO" when it was raised in a meeting. Not that going cloud absolves you of that requirement, but it takes a lot away from it - "someone else" provides reliable power at scale, "someone else" provides cooling at scale", "someone else" provides security at scale, and so on. I've done that, in a small and a medium business, and it does have real costs and knowledge requirements (I have them, but many sysadmins really don't). As a simple example, neither of those businesses were large enough to justify a standby generator and the associated infrastructure - so we were reliant on larger UPS batteries to support a graceful shutdown rather than just ride out a couple of minutes while the genny (hopefully*) fires up. And of course, that does mean that when the power stops, so does your business.
If you are big, then yes I'd agree that paying someone else to "do stuff" and add a markup before billing you makes little sense.
In between ? Well "it depends".
* Yes, the genny should "just fire up", but I think we can all remember stories reported here in ElReg where it's not gone to plan - whether that's having the lift pumps by the fuel tank in the basement not connected to your maintained circuits, the basement being flooded and talking out the electrics down there, or the genny catching fire, or it relying on batteries which are dead and no-one noticed because there was always mains when you ran your tests, or the system was wired wrong and relied on mains power to start the genny (or run the controls needed to do so), or (long list fo other reasons) or ... you simply forgot to fill the tank. With a larger installation, there's more scope for having a parallel run/export capability so you can load test the genny, turn over your diesel supplies, and have more "professional" people managing it rather than it being a sideline to some smuck's normal job (who probably doesn't know what it is he's managing).
You need a competent sysadmin regardless of whether you're on-prem or cloud.
In fact, many would argue that having a competent sysadmin is more important for cloud because everything "cloud" is Internet-facing by fundamental necessity, and thus more exposed. Worse, you cannot implement a change freeze around a risky time for your business - cloud changes when the cloud supplier feels like it, not when you're ready for it.
And most of what you cite isn't a sysadmin job anyway, it's the building manager who looks after the genny, HVAC etc.
I agree that the sysadmin probably needs more competence, but...
1) That sysadmin is NOT going to be buying hardware. Or wiring it.
2) That sysadmin is now going to have the tools to get 4 or 5 nines of reliability with very little effort. (Of course, you actually need a team on call to deliver 4 nines).
Testing your generator and UPS arrangement using "the big red switch" is needed so you know that if the whole building's power drops, then it fires up and runs everything you need for at least a half hour during the test, where you walk around and chat to folks and see that indeed business is going on normally. At the end someone checks the fuel gauge has recorded the expected drop in capacity and you get more ordered as needed.
But so many folks are scared to really test in case it goes wrong, and you know what that leads to...
Other colours of switch are available, and you probably should appoint an Igor to do that task who know where it is and how to safely operate it on-load.
Indeed, and even that is not foolproof - there have been cases where someone has done that, but unknown to them, some vital bit of the system was powered from something other than the section they were switching off. Normally, the genny & UPS would only supply part of the site, and you'd never get permission to switch the entire site off to test just that part.
But yes, it's what you really need to be doing, and it's so risky that no-one would ever let you do it.
"Hosting your own hardware will always be cheaper in the long run than hosting someone else's simply because you're not paying the "someone else" tax"
That's very flawed logic, and it doesn't work with most other things. Consider your response if I said one of these things:
"Generating your own electricity will always be cheaper in the long run than buying someone else's simply because you're not paying the "someone else" tax"
"Growing your own food will always be cheaper in the long run than eating someone else's simply because you're not paying the "someone else" tax"
"Carrying your own packages from your house to your recipient, even if they live on another continent, will always be cheaper in the long run than using someone else's delivery service simply because you're not paying the "someone else" tax"
I'm guessing that you don't own a farm, power plant, and global mail system. You might own one of these things, but usually only if you're the someone else providing your service to people who would definitely not be doing better if they were trying to do it. This doesn't automatically mean that on prem is more expensive, and for many use cases it will be cheaper, but your reasons why it is are bad. If you use the wrong reasons to defend your stance, it may harm you when those reasons are proven invalid but there are right ones that could have done the trick.
-- "Generating your own electricity will always be cheaper in the long run than buying someone else's simply because you're not paying the "someone else" tax" --
Don't suppose you've noticed all those houses growing solar panels on their roof? Also, up here in windy Caithness, there are quite a few with their very own wind turbine.
Now what was your point again?
And at night, those houses just power on their personal back yard natural gas turbines to carry them through? Oh, right, unless they really have to use a battery, they're still using the grid. Solar panels aren't generating all your own power. They're making use of the solar energy that hits your house, which is great, but in many cases the panels are sending excess energy into the grid during the day and other sources are being used at night. The number of people who operate without the external grid is much smaller.
For the same reason, I've grown vegetables before but that does not count as providing my own food. At harvest time I get to eat some stuff I grew, and it can be tasty, but in the rest of the year I buy my vegetables from somebody else, and all through the year I buy every other foodstuff from someone else. If you use the electricity line coming into your house at least sometimes, you rely on the grid.
> And don't try the "hired sysadmins are expensive" bullshit.
Many companies hide the real system administration costs by passing it over to software developers in the "cross-functional teams", so instead of having a couple of competent system administrators you have a whole horde of backend developers with very limited understanding of running and maintaining servers setting up and running "servers" in cloud services.
The developers' time is wasted on effectively working as substandard sysadmins on top of often having to be database administrators as well, but it's fantastic because it's "cloud"
In a major public cloud you will not get the same performance as raw metal unless you pay $$$
There is a middle ground here, host your server at a smaller provider that specializes in such thing. Hetzer does have traditional cloud instances but also cheap raw iron. What they don't have is managed services like S3 though.
"On-prem still cheaper, but don't expect a cloud exodus"
If you're spending a lot on cloud services then on premise is much cheaper so, what is "exodus"?
Non-IT businesses have no choice as they must support the non-IT business model but, what is the excuse if you're an IT business? Lazy? Fear? FUD?
The forgotten: "Cloud" puts a lot of _REAL_ IT people out of work and for an industry that goes on and on about equality and pay... well, all that political correctness seems like hypocritical bullshit.
Non-IT businesses have no choice as they must support the non-IT business model
Not true.
Firstly, the same applies to an "IT" business - hosting a load of servers, and the associated support infrastructure probably isn't their core business, so there's (for most of them) still going to be an element of the hosting being an adjunct to their core business.
But mainly, regardless fo the core business, the same considerations will apply : At the scale they need, what will it cost them to have the on-prem facilities and support compared with cloud ? OK, and "IT" business will probably have some savings in terms of having staff who are already doing "IT" stuff and hence can take on hosting support as part of their routine job, but beyond that there's little difference. For example, does you average small IT business employ someone fully up to date (and potentially, depending on your jurisdiction, licensed) to specify, install, and maintain your electrical installation ? Similarly, does it employ someone fully up to date (and potentially licensed) to look after the cooling system ? Do you have the scale to justify the cost (and space and other considerations) to have a backup genny - or you you have to do a graceful shutdown and put up the "closed for business" sign if the power is off for longer than you have UPS batteries for ?
So it's not really about "IT business" or "non core activities" - it's about whether the benefits of sharing someone else's facilities and support outweighs the benefits of the DIY approach. That will depend on the skills you have in house (and how much free time they have after their other duties), and your particular computing workload.
” What's more, the analysts report that many of the same forces driving up cloud costs are negatively impacting colos and private datacenters too, which means the savings made from ditching the cloud may not be as large as enterprises are expecting.”
Savings will most probably be larger as we can expect cloud prices to increase at a faster rate than actual input costs. Remember the business model of the moment is subscription and companies need to keep increasing subscription rates to keep meeting the unrealistic expectations of market analysts….
As for the extra regulations, many will be more easily satisfied in a smaller and more local environment.
"For the time being, Uptime notes on-prem and colocation datacenter hosting is still less expensive than the cloud..."
Not just "for the time being", either. I think it should be obvious that if I am paying someone else to do something for me, it's likely costing them in the ballpark of what it would cost me, plus they're going to add a bit on top. The reason they can do it cheaper is because they have more expertise and/or volume than I do. The reason I'm happy to pay them more to do it is that it's worth it more for me to focus on my core business and not worry about the rest. It really is economics 101, straight out of Adam Smith.
"I think it should be obvious that if I am paying someone else to do something for me, it's likely costing them in the ballpark of what it would cost me, plus they're going to add a bit on top. [...] It really is economics 101, straight out of Adam Smith."
It's so entirely contrary to one of Smith's core points that this really needs countering. It's the concept of comparative advantage. I could generate electricity, but I don't have the skills to do it well, I don't have the scale to do it efficiently, and I don't have the business to sell it to anybody other than my own use. Hence, it would not cost me about as much to generate electricity as it does for me to buy it. Even if I don't value my time at all, I would have to spend significantly more currency per joule I generate. That's not automatically true of everything, but it is not only not obvious that the costs are the same, it's flat out wrong.
Smith made this point at length as he was arguing against the tendencies of governments to use this flawed logic, restricting their ability to use resources efficiently. To use the famous quote, it's not that he bought from the butcher and baker because he didn't want to spend the time butchering and baking, but also because they were better able to do that than he was; years of skill and having a town willing to buy helps.
This doesn't mean that you have a comparative disadvantage in running servers. You're posting here, so you're probably in the group that has an advantage. That is why your employer hires you to do it rather than deciding that "Well it will cost us about what they're going to charge and they're going to want a salary, so let's try to skip them". You're better able to administer systems than they are so they hire you to do it, and if you decided that a certain way of storing and operating the servers was most advantageous, it could be true whether you did the housing or hired someone else to.
I remember being the PFY, full of hope and excitment at working in the dynamic world of IT. I bemoaned the grumpy and wrinkled older guys who moaned that they'd "seen it before", or told me about the cycles, outsource, bring back, centralise, devolve... I scoffed that they were letting their greybeards cloud their thinking and that virtualisation and then cloud were game changers. They pointed to the mainframes and other technologies gathering dust - but largely still running "legacy systems".
I've worked on programmes with a "cloud first" approach and ignored the squeals of fellow techies when they claimed that X wouldn't work in the cloud... and written them off as luddites, only to also see their smug faces as they were proved right, usually expensively right too.
I am now that wrinkled old guy who has lived through enough battles to know that there isn't a one size fits all approach as I am about to embark on a new programme following a change of director and change of direction. He's not siad the actual words "cloud first" but came very close, I think my pre-emptive wince stopped him in his tracks.
To paraphrase the patron saint of IT workers "Jesus Christ", Give unto the cloud what is the clouds, but keep the stuff better run on prem, on prem". Hopefully another deity can help us better identify which systems and services are which. In the meantime, if the thing has given 2 contractors nervous breakdowns since management decided to let Keith go as he could be easily replaced then it probably should stay in house
And not putting all of your eggs in one basket. Keep a foot in both worlds. Years ago, the team I worked with wanted a Co-Lo, we scoped it out, visited the facility, and looked at cost. Not so attractive. A few years later, a move to the cloud for email, then a few more things, but it stopped there for that org. They still had a 3 node cluster of VMware hosting lots of services that could have gone into the cloud. A handful of physical servers doing local stuff, it worked well I must say.
I think email is one service where cloud hosting really makes a good deal of sense. It benefits a lot from economies of scale in things like training spam filters, dealing with IP blocks, etc. Running an in-house email server is a lot of work on a per-user basis, especially for a smaller company.
The issues I'm seeing lately mostly revolve around cloud storage. Free tiers are going away, and extra storage is getting more expensive. Where I work people who have committed data to services like Box and Google Drive are having to rethink their strategy. But of course for x amount of storage in house, you really need more like 3x when you take into account backups, so it's complicated.
and running your kit in there is still cheaper than the Cloud plus you're not subjugated to MS or AWS SLA's.
I agree that these days there is no way an IT Team (usually infrastructure) should have to deal with boring things like power, air conditioning, risk of flood etc. PLUS the inevitable "oh didn't the building owner tell you there was a power down test this weekend"?
Only the very smallest businesses should have local "in office" server rooms. The best thing you can do is stick your kit into a CoLo, especially if you're a Govt Organisation, Crown Hosting is so cheap you'd be mad not to.
Forget the technology of on prem vs. hosted or cloud for a moment and think about the money: if you're a beancounter, it's far preferable to pay a monthly/quarterly fee for your compute than it is to dig into the company savings and tie up a vast sum in buildings, hardware and people. Sure, if you average-out the costs over a few years of both models then the chances are good that the on prem will work out cheaper - but that's not how it works in finance. Operational costs are better on the balance sheet because there's there's less debt.
While they could bring the services back in house and lease the building space, rent the hardware and so on - that generally comes with lock-ins and, once again, that looks ugly on the balance sheet because it has to show as money you are committed to spending in subsequent years and you generally can't forward account for income on a similar basis.
That works until your business finds itself on the receiving end of a worthless SLA which is designed around protecting your cloud provider from having to take responsibility for service interruptions or security issues, which, if your business is fully reliant on the cloud, could quickly turn out very damaging or even fatal (to your business).
Good example is OMIGOD, a security hole in Azure's Linux management stack (which is invisible to Azure customers), giving attackers root access to Azure customers' Linux instances. Microsoft was informed about the security hole early on but didn't do shit for months while the hole was already actively exploited. Eventually, they came up with a fix, but at first not with a way to easily patch existing installations. It took Microsoft forever to come up with a workable solution.
Of course, OMIGOD was one of the worse ones but not the only security hole of that caliber. Any business which during OMIGOD has been an Azure customer and had sensitive data on an Azure Linux instance has been at very high risk of being powned. Depending on the nature of the business and the sensitivity of the data, that pownage could have easily caused major damage or even kill the business for good. And because it runs on someone else's computers, there is literally nothing the business could have done to protect itself (other than not using Azure of course).
And this is on top to the usual Azure/MS365 outages, which we already had several of, even this short into the year, and which make any at least half-decently run on-premises solution look like Fort Knox.
That's cloud SLAs for you. I wonder how much the beancounters [*] which prefer OPEX over CAPEX have factored in to compensate for the potential complete loss of control of what often is a company's most valuable asset (data).
[*] in reality, it's not accountants which prefer OPEX over CAPEX (they usually don't care), it's often someone higher up the food chain (usually someone who isn't "a numbers guy").
"in reality, it's not accountants which prefer OPEX over CAPEX (they usually don't care), it's often someone higher up the food chain (usually someone who isn't "a numbers guy")"
Yes, that's the creative finance people C-level suits who are only looking at next quarter's bonuses and don't care about the 5-year viability of the business, they're not planning to stick around that long anyway.
That's also why so many businesses and government departments rely on so many contractors, as they don't like to see large headcounts on the balance sheet, ignoring the fact that many of the contractors are essential to the company operations, that the company would grind to a halt without them, and therefore from a long-term cost perspective their costs should be treated same as that of employees.
” if you're a beancounter, it's far preferable to pay a monthly/quarterly fee for your compute than it is to dig into the company savings and tie up a vast sum in buildings, hardware and people.”
We saw the same arguments used for sale and lease back of business premises these past few decades. The trouble is this thinking bakes in higher overhead costs which have to be met regardless of trading conditions. As we saw with the risk of the web, where transactional costs were massively reduced, those with high fixed costs found themselves unable to compete and so failed…
With activist “investors” also encouraging businesses to take on even more debt just to pay them more, despite massive changes in the business environment, we can expect even more companies to fail…
That's true in business, less so in academia where money tends to come in one-time chunks. It's a lot easier to budget for the purchase of a piece of hardware (especially if you can write it into a grant) than it is to budget for ongoing subscription fees.
Businesses have ways to deal with large capital expenses, and they have to do it for all sorts of things. They can manage it for computers as well. It may look nice on the paperwork and they may be able to use it to make a specific financial statement look good when they switch on, but it usually doesn't bring that much to the business's financial situation.
The primary case where the difference becomes important is when a business needs to buy a lot of computers right now, and the cost of outright purchase would be higher than temporarily renting them. Or in other words, it's the spiky use cases with extreme scaling requirements that is the most obvious case where cloud is useful. In most other cases, there won't be much of a crisis to treat it as capital expenses; if it's small, the expenses aren't going to be very large, and if it's a lot of servers, then you can get the same kind of expenditure flow by buying replacement hardware on a staggered schedule.
Cloud can be useful and there are times when it can be cheaper, but how it appears on the statement isn't particularly important in most cases.
imagine a company that is so into leasing everything, so much that it's pissing £50mill up the wall in leasing per year.
goes bankrupt as sales went down but now they didn't even own the desks they work at.
I was involved in the take over, flipping shit back to on premise/owned stuff saved over £25mil, they are now profitable.
fuck cloud shit, only wankers go that route
Oh really ?
So now on-prem is attractive again ?
Does that mean that all those C-suits are no longer basing their bonuses on moving everything and its dog to The CloudTM ?
Because I seem to recall a not-so-distant time when The CloudTM was the cheapest, most reliable way to go.
Looks like the beancounters have brought a cluebat to the board's meeting room.
About time too.
Probably in ~75% of the situations out there. Maybe on prem is a money pit in ~25% of the situations out there.
Back in 2016, El reg reported on Snapchat's IPO filing where they admitted they were committed to $400M/year in google cloud spend over the next few years(the most extreme example I can think of), and I poked at their financials earlier today and they still seem to be losing upwards of $1B/year. Obviously an org that doesn't care about burning money.
Imagine what one could accomplish with $400M a year "on prem". Likely real on prem costs for them would be in the $50-70M/year range at the most?
Last company I was at was a "small" company (employee count wise), I moved them out of cloud in early 2012 when their cloud monthly bill was ~$70k/mo (and they had just launched about 4 months prior). Over the following decade they grew a lot, but never needed more than four cabinets. Company prior was even smaller, maybe 60 employees or less, and had a cloud bill at times over $400k/mo, they never moved out of cloud despite my efforts, I had everyone on board including the CEO and CTO but the board of directors wouldn't budge. Company is long since dead now.
IaaS is not for the faint of heart it requires real skills and experience to get going right, but even then you can fail (cost wise) as Basecamp has shown(most recently). Being an infrastructure person myself I realized this back in 2010. Perhaps the new normal of higher interest rates and no more "free money" will shift the tides heavily away from at least IaaS(I can hope anyway haha).
SaaS is a totally different beast of course, with a totally different cost model(for end customers), and in many cases if you want/need that software platform you may be stuck in using their model. I have seen time and again SaaS platforms launch over the years as an excuse to make poor (quality) software products that their customers couldn't otherwise hope to operate on their own. I'd like to think a small company I worked at more than 15 years ago was an early pioneer in that at the time (and we didn't even know it then). Company made a software/service for big telcos. At every major company meeting a common strategic goal for the company was to make their product so the customer could operate it themselves(that never happened while I was there). At one point it was my project to help demonstrate to the largest carrier in the U.S. that they could in fact host and operate our software on their gear. The customer agreed the demonstration was a success(took 1-2 months) and paid the company I worked for the $1M fee (or so I was told), but I could see the sheer horror in the eyes of their employees as they tried to understand how that super complicated software stack worked (by far the most complicated of any stack I've ever worked on, had two people on my team quit within their first few months because they felt they couldn't keep up, never seen that before/since). As far as I know they never took the software in house, always chose to operate it as a SaaS model from the company that made it, am guessing the usage of that software stack was mostly phased out a decade or more ago, as Google and Apple took over the markets it was targeted towards(though at least one of their applications is still online and shows the same name for copyright in the code as it did in 2005(except the date has been changed to 2002-2023, even though they got acquired in 2006, another one of their portals is online but branded with the company's name that acquired them, though the underlying host/domain of the site is the original from 2003, they don't even redirect to a new name).
I've been working on high availability internet hosted application stacks for 20 years (as of next month), and have never once yet seen a situation where IaaS (as sold by the big cloud companies at least) would be the right choice. I'm sure such situations exist, but they are few and far between(as I called out in 2010), UNLESS, of course, you don't care about burning money).
IaaS to me would be real useful if your workload is VERY variable. Needing thousands+ of CPU cores and terabytes of memory for short periods of time(guesstimate less than 7 hours/day on average over a period of time).
You can do on prem poorly/expensive, but it's my opinion it's far easier/more common to do cloud poorly/expensive.
If you've put you data storage and/or computing in the hands of a cloud company, there is little you can do if they go down. Unless you are a huge percentage of their business, you won't reach anybody during an outage (even if they haven't put all of their comms on the same feed). Too many companies will analyze the cost of going with an outside provider but fail to do any or much risk analysis. The reason a large manufacturer will pay tens of thousands to hire a private jet to ferry in parts if a regular shipment will be delayed is they've done the financial analysis that shows if the production line goes down, they lose hundreds of thousands an hour. At the same time, they might not have looked into how much risk they take on by going to a Just In Time delivery schedule with very few components on hand at the factory. It might be much cheaper to keep a few days of components and material on hand and certainly less stressful than having to go into panic mode. I know of a few companies that shut down for some time when dockworkers went on strike in California and shipments were sitting offshore in anchored ships and even when the ports opened again, the line at Customs was going to be long.
If a business plan is so tenuous that important functions can't take place in-house, maybe the business isn't viable. It's nothing to reverse engineer a product, but a company losing control of how they make the product at a profit, who their vendors are and details of customer discussions is deadly.
Risk management should be but often isn't a focus of the administrators and managers of any system, computers or otherwise. If you decide that cloud is your structure, you have to decide whether you want geographic redundancy (global outages are much less common than one-region ones), multiple cloud providers, or some cloud and some backup servers, or maybe you don't need any of those things but an outage could affect you in a worse way. Similarly, you have to consider most of the same risks if you're housing your own equipment (how much generator capacity do you need, what if the generator fails and you have a power outage, what happens if your DC is hit by a natural disaster, what happens if it's mismanaged). Some companies decide they really need two redundant datacenters in different locations with independent administration, and some don't. That can cause an outage as well and that outage can be as bad as a cloud-based one. Anything can break and anything important needs to have a book of plans for what is likely to break and what's going to happen then.
"Similarly, you have to consider most of the same risks if you're housing your own equipment (how much generator capacity do you need, what if the generator fails and you have a power outage, what happens if your DC is hit by a natural disaster, what happens if it's mismanaged)."
I agree, but in those cases it can be possible to throw money at the problem to get it solved. If AWS is having problems, you'll have no clue what those problems are and when they might be back up. You also can't throw money at them to get your operations back in place.
Having a third party backup provider can be a very good thing. If they are just backing up your data, you can keep it encrypted so if security is compromised you are still protected. If you are relying on a cloud provider to parse out that data internally or to clients, it can't be encrypted to the same extent.
"in those cases it can be possible to throw money at the problem to get it solved."
Not after the fact unless your problem has gone on for a very long time. If your power fails and you don't have a generator, throwing money at the problem will not get you a generator in the next fifteen minutes unless you happen to be right next to a generator store. You're still going to have to wait for somebody to show up with the equipment and fuel, and that's if you're willing to skip all the testing a new installation takes. Your money can probably speed up the process quite a bit from a routine install, but it's not going to be quick and easy. Similarly for a cloud failure. You're right that you can't pay them to just fix it already.
You have to throw your money at the problem before it happens by paying for a generator before your power fails. And the cloud-based alternative is redundancy. As I said in my original post, most cloud failures are specific to a region if not to a specific service in a specific region. The solution to that level of problem is geographic redundancy, which the cloud provider will sell you for more money. You have to have chosen that before they have a problem, just as you have to build your backups before they're needed. If you want even more redundancy, it is possible in the cloud just as it is on prem, but both get quite expensive as your need for fault tolerance goes up. Some companies need and therefore pay for multiple redundant datacenters with a significant amount of unused capacity on multiple continents with full-time admin staff, but I can guarantee that it's a big expense for them just as four clouds each with geographic redundancy would be.
We use the cloud for email and file sharing.
We use on premise for CI and code storage/management for all new developments. Most of our contracts require a tight control over access and where stuff is stored. At which point on site for all is simpler.
We have a legacy gitlab.com group repository but are looking at moving that on site.
All new tools we look at we have to make sure we can host...
If the servers go down it's not a show stopper. The site has a generator with 7 days power backup. The issue is usually the switch over from grid to backup and back to grid.
Either way at worst it's an hour and a bit downtime during office hours, bit longer out of office hours as it's dependent on if I can be bothered or we have a critical delivery to finish.
Of course all this is only really possible because we have symmetrically giga ethernet connection to the site. We can contract up to 40Gps.
We are of course mainly WFH or customer site.
There have been a few "cloud isn't always shit" articles recently, which shows the way the wind is blowing.
The cloud isn't always shit, it makes sense for stuff everyone uses, like emails, or stuff almost nobody uses, like a hyper-specialist product that only one company can supply. But there are so many uses cases where it's the wrong solution.
Interesting this nail is being hit on the head but as Leghorn Foghorn has said "...boy, I'm cutting but you ain't bleeding!"
Competent system administrators are hard to find and fewer are interested in entering the trade/profession.
When your services are in house the existence or competence of the system administration is either evident or assessible. In the cloud it pretty much has to be taken as an article of faith that your provider has engaged sufficient competent sysadmins to maintain the platforms your services run on.
After 30+ years I have to say Unix/Linux system administration is largely a thankless unprofitable career so I fully understand why potential recruits rapidly head in other directions. When ITSEC ceases to be sexy it will suffer the same fate. Management or coding are far more attractive to the polloi and largely more renumerative if mind numbingly vacuous or boring.
"Competent system administrators are hard to find and fewer are interested in entering the trade/profession."
It used to be a well paying position, but that's changed a lot. That also means that good operators aren't being trained up and the few there are will go where they'll be paid the best. That's going to mean the larger service providers.
The Cloud had it's chance to prove it could store data privately, securely, cheaply and reliably. OK, you can stop laughing now. There are so many reasons to store your own stuff anymore. As it turns out insurance for hacks and downtime is cheap. What are you waiting for then?
"As it turns out insurance for hacks and downtime is cheap"
Possibly, but Sony had some media get swiped from their servers before it was run in the cinemas. How will insurance pay for a new product or process being made public when it's that company's bread and butter? The cost of lost business is hopeless to try to quantify. I've seen a few Kickstarter programs fall on their face when a company in Asia obviously rips off their idea and is mass producing the product they are trying to get money to develop before the funding round closes. Translate that to a company losing an entire R&D directory full of stuff.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
