The IT world moves fast, so why are admins slow to upgrade? Administrators tend to be a conservative lot, which is bad news for tech vendors such as Microsoft that are seeking to pump their latest and greatest products into enterprises customers via subscriptions. SQL Server 2019 recently dropped out of mainstream support and is now on borrowed time, with extended support running until …
Despite the shrill exhortations to move to the cloud or switch to a continually updating subscription model...
Of course not.
I don't think that there is a company whose first priority is to pad the bottom lines of the likes of Microsoft, Oracle or SAP. Their interest is the health and well-being of their business.
Besides,change has to be managed and most companies will move cautiously, analyse the business, work our what it is they want to change and then submit a plan to the change board or whatever they call it.
Moving fast and breaking things might be alright for Zuckerberg but most managers want to see their companies prosper and not crash and burn just because some third party wants to sell them "the latest and greatest."
Exactly. I have "business consultant" morons to content with here who are pushing "move to the cloud" yet are unable to provide any tangible benefits to counter the "put our organisation critical data on somebody else's computer, have less control over this data and worse backups, less performance and pay much more for this"* stance. But then these are in the same category as the morons who when told that double checking SQL reports takes time and instead just repeat the idiot chant of "just use AI" as if the magical "AI" is capable of understanding anything but the most basic of data structures, let alone actually produce anything out of it that adds up and validates.
* Over five times the cost to host and for this we no longer have four hourly database backups, we would have a connection speed that is limited to Internet up/down bandwidth as compared to gigabit ethernet, less than half the compute and memory allocation and really not knowing who else has access to our organisational data.
"I have "business consultant" morons to content with here who are pushing "move to the cloud" yet are unable to provide any tangible benefits"
Benefits to whom? If they are not collecting kickbacks .... er, finders fees from cloud providers for the delivery of another customer, then they are indeed morons.
If they have their own yachts and private jets, then they are in fact quite smart. They are just not using that grey matter for your benefit.
"I don't think that there is a company whose first priority is to pad the bottom lines of the likes of Microsoft, Oracle or SAP. Their interest is the health and well-being of their business."
You've never spoken to accountants who would rather deal with regular monthly payments... Does not matter if it costs the company much more in the long run, They get to copy and paste July into August's spreadsheet and be done for the month.
I had to give this very serious thought before finally giving you a downvote.
Back in the 1998-2002 period I worked for a consultancy, and we had a couple of upgrades that had issues. Mostly due to something the client had neglected to mention or spot during the scoping & planning stages. All of them were surmountable, and we got the job done in the end.
Since then I don't think I've had any upgrade that had major issues. Any issues encountered were ones we had planned for, and either directly addressed in the plan or had a contingency plan for. Absolute worst case we had a small delay, but we always made sure our change windows were wide enough to cope with that.
I'm also excluding migrations to other technologies. To the business moving from one mail system to another may be sold as an upgrade, but it's actually a migration, and there are always unexpected issues with those. So those aren't - I'm assuming - what you're asking about.
Some may say I should have upvoted, but I think you're trying to determine if there are unexpected technical issues rather than ones that were created by the business or bad planning, hence a downvote.
I'm in this place too, but it's an old old app I wrote for a local library to allow searching of their collection of newspapers. Luckily it's a readonly DB updated from the master internal server and lives in the DMZ. Simple mysql, php, apache. Trivial. But it's on an old version of Debian, with an older version of PHP. And the date time library I use for dates (all before 1900...) is semi-broken and abandoned for newer versions of PHP. Sigh... And of course some of the PHP language stuff has changed as well, so I had to re-write some stuff to get around that.
So I'm not in a rush to replace it on my spare time. But I know it needs to move up to something newer. It's not sexy, but it works, I can spin up new versions and new interfaces to new a new DB of collections and the internal people know how to push updates to my server. Works a treat. Until it doesn't. But it's read only and in the DMZ, so it's not critical.
Too many systems don't have this mindset of keeping the production seperate from the reporting or query systems. Yes, I understand you want to be able to enter new orders and such into your production system and you need to do that. But for alot of things, have a query only system you can not care about because you can rebuild it quickly is "good enough" in alot of cases.
Upgrade issues: even later iterations of Ubuntu 22 LTS (ffs) had issues, such as changing the interface descriptors of PCIe connected Ethernet interfaces, adding a bunch of characters for no immediately obvious reason. Cue VM no longer working. Not long to fix but on an LTS distribution, please?
Counterpoint, how many people have experienced a problem because something wasn't updated? Maybe it stopped working with other things. Maybe it was impractical to maintain. Maybe it got broken into. I have seen a lot of people stick with not updating it because there were no problems until there were problems, and at that point it was a lot harder to maintain it. Occasionally, I've had to rewrite things from scratch to respond to it, and since I didn't write the first versions, you don't even get the benefits that generally come with a ground-up rewrite but you still get all the downsides like having to retest everything and it taking a while to plan, design, and implement.
We have many stereotypes of people who ignore things because they're not a problem now until it blows up in their face. It can be useful to consider whether we are doing the same thing but telling ourselves that it's different this time.
It can always go wrong. This is some time ago (it was an OS/2 product for a start..) but it stuck in my mind.
Went to do a 'trivial' upgrade. This was planned properly, before the upgrade everything was backed up to a Netware server. This took two hours.
Ran the upgrade that had been run on other systems many times before. It failed. Retries did not help. Rang the manufacturer support line, they did not have an immediate answer.
OK. Fine. The system needed to be operational by that evening. It took two hours to copy up, we have time in the schedule, copy back down - it'll be ready in two hours.
No. Try *four* hours. Finally got out of there at 9pm that night, everything restored to normal.
Following day the manufacturer had found the issue (weird licensing issue I think, but it was a long time ago). Two hours backup again. Upgraded in less than an hour. Job done.
Performing upgrades is only a small part of the job.
It is, I'll admit, one of the more visible parts. In many companies the other work sysadmins do is much less noticeable. Monitoring? Capacity management? Availability? Policy work? Risk management? Integration? Many of those are a lot less visible than upgrades.
Security has, thankfully, gotten a much larger presence over the last two decades - so when we're working on that it's often very visible.
I can see how folks would think we should be busy doing upgrades all the time, but that's like thinking that the only thing to running a shop is manning the tills or stacking the shelves.
And of course then there's whether or not the business wants to upgrade. All upgrades have a price, and some are more worthwhile than others. It's not always worth the effort.
I've dealt with a few systems over the years which had to be completely rebuilt rather than upgraded because they'd become so old there was no valid upgrade path anymore - we couldn't get hold of all the install media sets we'd need, and whilst an individual upgrade was quick the amount of time needed to perform about five upgrades with checks in between each one was simply prohibitive. So we built anew.
It's the sysadmin equivalent of developer's technical debt problem, and just as with the developers it's not solely the fault of IT. If the organisation won't prioritise keeping everything up to date, then the costs for moving to a current platform will only go up over time. Sadly, that's often not considered when people are looking at the costs for this year...
You missed testing. Functional end to end testing by people who know what they should see.
It's really, really expensive.
Most companies don't have the capacity to reliably change what they are using. Some go ahead anyway with mixed results. You might get lucky that the DB or server works similarly enough that there are no issues. You might not. If you are on contract to deliver the product, you blame the customer, see posts above...
> which is bad news for tech vendors such as Microsoft that are seeking to pump their latest and greatest products into enterprises customers via subscriptions.
Maybe latest... but greatest? Not bloody likely.
On MS SQL specifically... Our new "development server" is a higher spec than one of our production servers, because we're migrating to MariaDB, and it's cheaper (and more useful) to put the money into the hardware
Because new versions of Microsoft software always prioritise adding new features over fixing old problems. And those new features always bring even more new problems. So everybody has got into the habit of waiting years to upgrade in the hope that a lot of the bugs will be fixed by then. This used to happen by monitoring user forums and waiting until the number of complaints died down and it started to be seen as reliable. However, it's now so well known not to upgrade straight away to major new versions that there aren't so many first movers to watch experiencing the pain. This is entirely the consequence of prioritising rushing new product to market over quality engineering, and if they're finding that the reputation they've built up for that is now affecting their bottom line, they only have themselves to blame.
The rot set in wholesale a few years ago when Microsoft fired most of the QA staff. Such actions take a while to really take impact but it's been visible as less and less stable servers, systems and the declining quality of everything that Microsoft have released, even internally.
I think there are a couple of reasons admins put off upgrades. Many sysadmins have experienced upgrades breaking existing features. (I don't mean planned deprecation, just good old fashioned bugs) Secondly, vendors don't always make upgrades easy. I've come across one application where you had to export the config (as a text file) manually edit it, and load that config back into the freshly installed software. When the config is 20k lines, how confident are you that you've edited all the right bits in the right way?
If upgrades were easier and more reliable sysadmins would be more than happy to regularly upgrade.
Isn't the reason a combination of:
a) These new versions don't offer anything which most sysadmins (or the companies they work for) have any need for
b) Nobody trusts software companies (especially Microsoft) not to break a load of things.
c) New versions (or ongoing software assurance so that you get the new version licenses as part of that) are expensive in many cases (especially Microsoft server products)
The combination of those three can often lead to versions continuing in use until the end of extended support is close.
On the client side, take Windows 11 - there is also a fourth issue here: the system requirements. We have completed a move to W11, with the last old machines which didn't support it (and which remained on W10) now retired, but if it wasn't for the age profile of our machines (we moved to mostly laptops in 2020/21, so the bulk of the machines met the W11 requirements) it would have dragged on for longer. W11 hasn't given us much hassle, and the users have mostly coped fine - I did the obvious things like removing the crapware and left-aligning the taskbar using policies, to make it look very similar to W11. It's OK and I don't particularly dislike it any more than W10. Sure, there are a few annoyances, but W10 has some too. But it really doesn't offer any 'must have' or even 'that might be useful' new features and I can quite understand why many companies are sticking with W10 until the end of support.
I'm not convinced there's been much change in the fundamentals for desktop O/Ss in the past 20 years. Sure, they're being made more secure through better software engineering techniques, but actual features? The last biggie was hot-plug (Esp USB) which came in XP (And some vendors manually back-ported to Win2k!) And maybe TPM?
Otherwise, all that's happened is they've been painting fresh lipstick on the pig.
"Sure, they're being made more secure through better software engineering techniques ..."
Somehow I doubt that, judging by [1] the actually growing frequency of 'updates' and [2] numerous independent investigations of code quality over the years. Vendors would like us to believe that newer is more secure, but that's primarily so they can charge us for the next version -- and ideally on a subscription basis instead of a flat license fee. The truth is the IT market is saturated and its driving force is now economics not technical advance.
UAC is probably the last big one I can think of, so Windows 7*. The last few times I've had to do work on an XP machine it was the one feature I missed, invariably logged in as a standard user and realising I needed to switch into an admin account, rather than just run the installer and enter my admin creds when prompted. Powershell being universally available would be my second thing, but that doesn't come up as much when having to support old customer machines.
* I know it was released in Vista, but IMHO it wasn't polished until 7.
>>expensive in many cases (especially Microsoft server products)
If you get lucky and work in the right industry there are bargains to be had from Microsoft... £495 per server per year for Windows Datacentre is what I was paying last time we sorted the bill... quite a lot less than MSRP IIRC. Makes running a HA cluster quite cheap.
I did a software upgrade this morning. The application runs on its own system. That system is a virtual machine. One of the first steps in all of my upgrades, no matter how minor is to power the machine off and take a snapshot. The upgrade I performed was a minor one, upgrading from version x.y to x.y.1. It can't be any less traumatic than that, can it? Would you be surprised that I had to revert the snapshot, restart the server, and uninstall part of the application before it would successfully complete the upgrade, after which I had to re-install the part that I was required to remove. If you want to know why system administrators can be grumpy bastards...well this is one reason.
Normally this is handled by having clustered systems to provide high availability and during any upgrade (which will have been FULLY tested in a separate clustered clone environment) the first system in the cluster is brought down and upgraded, once it is ready to restart the other systems in the cluster may be paused or set to data only synchronising so there is no risk of contamination between the systems. If the first one picks up it's load correctly the others are brought down and upgraded, if not it is brought down and the changes backed out while the remainder of the cluster is re-activated.
If all that is not an option then a Beta cluster would be brought up with the upgrade installed and realtime data sync running. At a set time the pointers are moved from the Alpha cluster to the Beta cluster so Alpha can be upgraded
Other processes are available depending on the system loading/OS/complexity etc but this is not an uncommon scenario and every deployment has plans for this in place and regularly tested.
I run systems that are clustered in an environment which is appallingly difficult to arrange full system outages. They were designed to be clustered from the outset, and generally speaking it is quite possible to do most types of patching by moving the services around the systems that are running them. For most types of maintenance, the down times for the entire system for infrastructure work like OS patching is normally measured in a few minutes per year, mostly as very brief interruptions as cluster moves take place (although I can't say the same for other components in the entire environment).
But this only goes so far.
You will eventually come to a point where the clustering software itself requires patching, and there are certain break points where you have to have the entire cluster down to patch/upgrade this software. I've also had situations where certain OS operations cannot be completed while the cluster is running (an example was when the cluster software plugged into an OS monitoring system, and would not allow that system to be patched without stopping the cluster).
These situations are actually few and far between, but eventually you have them. But the longer you run the systems, the more likely you are to hit one of these break points. The design of the environment I mentioned above is now 15 years old, and has been patched to keep it as current as possible, but the end of the road of this state is neigh!
> The question is, why?
Because experience tells us that updates, patches and upgrades are quite likely to introduce new problems. Whether that means a new set of bugs that need to be analyzed and fixed, incompatibility with older software - requiring possibly a long chain of dependencies to be upgraded (assuming they actually can be upgraded) plus the possibility that after all those forced updates, the original bugs still exist.
And as we all know,every minute of downtime, every support call, all gets counted against the IT department. No matter what the cause actually was.
This is super-relevant. If you have an older version of a database underlying some custom software in regular use there's a good chance that an upgrade to the database will mean the software database needs an updated driver. Nine times out of ten that's fine, but the other time something in the driver has been deprecated and the software will need to be adjusted to work against the new version, or the driver will now depend on a version of a library used elsewhere in the software and now that has to be changed across the board. There's every chance that whoever wrote the software in question has moved on, so you're looking at a good amount of time for one of your dev team to figure out how it works and then a test and release cycle to get the new version up and running so it can be used.
Pretty hard to make a business case for this running-to-stand-still scenario - you've expended a lot of resource, and got the same outcome you were already getting (but with tool version 5.4 updated to 5.5).
It's strongly a "If it ain't broke, don't fix it" situation in the locations I've done off and on IT work for. So many places have E-Mail (which is usually GMail or some other web mail), web browser, and 1 to 2 applications they use but don't "push the limits" on where they'll benefit from the latest version of those either. If they do want the latest version of something (either something like QuickBooks, or some Photoshop-style package for artistic use) they go ahead and get it. But in general upgrading stuff ends up being 1) Risk of downtime with 2) No tangible benefit.
As a Linux user, at least for distros like Ubuntu and Debian they are downright paranoid about even a distro upgrade not breaking anything (in the worst case, keeping the "old" version of some piece of software, and it's dependencies, installed if the upgrade system determines upgrading that piece of software could be problematic.) But lets face it, most of these people are still stuck with Windows, there's discussion here of SQL Server, etc., and it seems like all to often people have fragile and bespoke software on there that blows up during upgrade, or at least the upgrade will keep one on their tows.
I *am* surprised about things like "post MySQL 5.7" type issues stopping people from upgrading at all. I mean, just like the rest, if MySQL 5.7 works well there may be no reason to upgrade. But, I've upgraded two items where they didn't upgrade "out of the box", and it was not painful to fix them. I think in one case it was using an old ASCII-only datatype (and for some reason didn't like using the replacement Unicode-compatible one, even though it was only storing ASCII data in it anyway...for some reason), but newer MySQL maintained a ASCII-compatible data type, so I switched that one item to that type and everything was fine. The other was setting some like ISAM (database engine)-specific "go fast" settings which were simply obsolete (like, they probably were already NOOPs in 5.7...) and could be removed.
The real question is : why is it that the rule "always wait for version 1.02" is still valid ?
Because Borkzilla & Co push out "bright" ideas ideas without a clue as to what the consequences may be.
Admins, on the other hand, have to ensure that the business continues running because, if everything comes crashing to a halt, it's their ass on the line to get everything running again.
In that kind of situation, you get burned once and you've got the lesson.
Borkzilla, on the other hand, appears to never have learned any lesson.
Most Admins I know (and myself when I was an Admin) tend to upgrade the OS, and back-office software like SQL Server and Exchange, when they replace their hardware. From what I have seen the same seems to hold for VMs as well. If we jumped every time MS released a new whatever we'd have to learn to bloody levitate, so we ignore the marketing blather and stick to our renew cycle. About the only time that anything gets upgraded outside the renew cycle is when a new feature in, or new version of, whatever software our users are running requires a later version of whatever. Most software I know and use can happily run on 15 year old versions of SQL Server and really don't need the latest whiz-bang version. We don't run it on versions that old, but that is mostly to maintain support. Sorry MS, but no matter what your marketers, management or shareholders think, none of us give a crap about your margins or dividends. Our job is to provide a reliable service to our users, and more often that not your software (and especially your Updates) are the cause of what little unreliability slips past us.
and more upgrade.
For some of us, any upgrade is out of the question. its not the down time or anything like that.
The systems may be running software built against a specific OS and is only verified against that OS.
So lets take our windows controled industrial gear worth 3 million quid and running a severe risk of injury or death to the operators if it goes wrong and install windows 11 over the old windows 7 installation.
Any luck it will just die and refuse to boot up, or it could boot up and try a variety of incidents that result in 1/2 the staff being deep fried in molten metal. at which point m$ says "not us 'guv" and the machines manufacturer says "not us 'guv either".
But we could buy a new 6 million quid machine equipped with win 11 to replace something that 5 yrs old and not even in finished paying off the loan taken out to buy it in the first place.
Note: the stuff I use has a minimum 10 yr lifespan, and more usually 15 yr+ . our last windows machine is still on XP
... is that lack of available, supported-by-the-OS hardware can render an otherwise-working control system useless. I've seen industrial control systems run by PDP-8s, PDP-11s, MicroVAXes, S-100 bus computers, Apple ||s, and PC XTs.
Good luck getting working motherboard replacements for those.
True, but consider the options.
Wait for it to physically die and deal with the capex then, or borrow that multimillion figure now?
If you have some idea about your hardware, then you can watch ebay et al and stockpile a few key spare parts for a few hundred USD.
That is likely to give the machine another five to ten years, during which there may become a reason to upgrade it that the business actually needs. The capex is far more justifiable when it's "the new machine does X, so we can sell Y"
In cases like this, where possible, the most appropriate route is to get these systems on a subnet disconnected from the internet for security/compliance reasons. And ideally have that entire VLAN unroutable from other nets.
If they also need to be managed remotely 24/7... Well, you're kind of outta luck for some certifications but well managed firewall rules and conditional access restrictions go a long way.
In most organizations I have worked for, it boils down to a risk/money conversation.
What's the risk of doing an upgrade?
What's the risk of not doing the upgrade?
What's the dollar impact of both of those risks? (impact versus likelihood)
What's the dollar cost of doing the upgrade and new licensing costs? (OpEx Capex type analysis tied back into either existing budget or new budget ask)
What else have you got on the go that are must do items, regulatory, legal, stay in business type items?
How they do they stack up priority wise with the upgrade?
What technical debt are you taking on and can you live with it? (Bit subjective but should feed back into the risk questions above.)
The elephant in the room can be promises made to the powers that be to deliver project x,y and z by year end and taking on an upgrade could jeopardize the delivery dates, always a gnarly problem and may require some intestinal fortitude if the right answer is do the upgrade.
Once you have answers to those questions you have a pretty solid argument for upgrading or not and if you are upgrading, when to do it and how much effort it will require. Any PM worth their salt will document these to a reasonable level so that the decision process is recorded and there for anyone who later wants to say "I told you so".
Unfortunately for MS the business case for the upgrade doesn't stack up, so until the risk becomes too large, due to security or lack of patching or lack of support, the more often than not answer is stick with what you have for now, and revisit the decision as and when the risk profile changes.
Was going to say something similar, but would add:
What's the benefits of the upgrade?
Plus, software companies tend to push out their software without full testing these days: They're in a hurry to start recouping the costs for the software, and may push out a 'new' version that's basically previous version with some new features that could have been released as an 'add on' to the previous version.
M$ are a typical culprit for this: SQL 2014 was an actual engine change, but 2016 was mostly tools and stability fixes, 2017 wasn't much better, 2019 had a few engine tweaks and some long overdue features, 2022... nothing special. 2025, bolt on AI by the sounds of it. Hence: What do I get for the extra cost that I actually need? Cries of 'security' is just marketing pressure: Older versions will have matured and vulnerabilities both found and patched while the new version? Nope, not yet.
Basically: Software companies release new versions way too quickly to be worth the time and effort to invest in keeping up with the latest 'bling'. Better to stick to a pattern of upgrade after x years and if you are getting the 'latest and greatest', then accept that it may not work as well as you'd prefer.
Does anyone really need new "features' and vastly increased attack surfaces?
I'd be really interested to see solid stats which show just which of the new, whizzy shiny features are used voluntarily by users because, pound to a pinch of shit, I reckon almost all are just tinsel and stuck on glitter to attract the gullible.
If a new version doesn't actually do anything new that we want, and in some cases removes features that we do want (which seems to be more and more common these day), why on Earth would we upgrade? Especially with the existing version being in some sort of support for a long time yet.
It'd be like replacing your car because the new one can now go 300 miles an hour, but you still only need to go maximum 70.
Dealt with any number of CIOs. Only ever heard "next year". I kept asking, because at least then I know i'd done my job.
O/S out of support? Check. DB out of Support? - Check. ERP - same. Security flagging it - no change of direction. External auditor raising it as a risk? - Noted, but not approved.
Twas not rocket surgery, but just needed a plan, time and approval. Lots of dependencies - what upgrade of serious tin and software doesn't have them? Plan was there, the other two were not.
Great plan, so long as it doesn't blow up on your watch.
The article ALSO makes it seem like IT Admins are to blame. Sorry, we're the monkey, not the organ-grinder.
A better question would be "The IT world moves fast, why don't business decision makers prioritise upgrading their systems".
Before you try passing all the responsibility, consider how many admins you know, including those who post here, who are not advocating updating things. I don't see many comments saying "I would upgrade, but I didn't get permission". It's undoubtedly one of the causes, but it is not the only cause. Sometimes, we are in fact the organ-grinders, because if we decide that updating that server can wait and never asked for permission, that decision is on us. If we take responsibility for that choice, we can better decide whether it is the right decision or not.
Lots of reasons for not upgraing and none of them are deep or complex.
"If it ain't broke don't fix it", is a totally reasonable statement. Many SQL instances are not internet facing so it's quite resonable to have them in a less secure state allowing assets to be sweated and reducing downtime. There are risks in applying any patch.
Change can be hard. Many large businesses will have bespoke configurations which might actually depend on some loopholes or bugs in particular versions or a library that no longer is supported or a quirky 3rd party integration. Redevelopment to use alternative features can be difficult and very expensive. Some system will only support older versions and suppliers may not allow you to upgrade.
Upgrades to new versions aren't free, we're all aware that MS et al like to force upgrades every so often depeciating systems which are perfectly usable and supportable.
A SQL version update to a major system is a huge piece of work, it's not like moving to a new version of Word
"Never mistake motion for action." (Hemingway?)
The most degenerate form of futile movement beloved of manglement is of course spin.
If the endlessly spruiked progress in IT were always an unequivocal improvement on what was replaced we wouldn't need backups. Would we?
In my experience we are ever more frequently playing host to Mr Cockup who is forever in rut over those systems that have so far resisted his overtures.
Because it is their ass on the line if an update takes down the business.
Given the quality of recent MS Patch Tuesdays, IMHO, this sloth like behaviour is warranted.
Those that rush in and update may well have to repent at leisure after they get canned when it all goes TITSUP.
I'm currently mad at Firefox because they've changed the default home page to not show URL just the site icon and title. This is going to lead to increased phishing as lots of users default to that screen and click the icon with the title that matches... if they have the choice of two icons with the same title, they won't check the URL just blindly guess.
That's a new one on me. A bit like an Early Adopter?
My very antithesis when I ran vendor support for a real-time DCS control system on hazardous plant. Rare for us to recommend an update or upgrade unless we'd really nailed the cause of a problem, and even then were more likely to bug the devs for a specific patch. Big version changes were a hair-tearer (what I still had).
A rule I learned was, the more developed a version number was, the more trouble there had been before. The number 6.3.7 is engrained in my brain, and was eventually ok, though we did make 6.3.11 before the 7 upgrade. The current Thunderbird revision history is a warning to me, 24 versions of 128.x.x (39 on 115.x.x, which I took a risk with) and they've just gone 136.0 on 4th March. No thanks just yet!
Because decades of abuse by software manufacturers has shown us that there's no such thing as a seamless upgrade.
It always introduces new restrictions, new licensing, new syntax, new options we have to go and turn off, one-way data migrations (so you're forced to v2 but have no choice about rolling back to v1 if it does any of the others in this list), loses features, introduces new bugs, deprecates things that were in use, and so on.
If upgrades were just seamless, people would upgrade.
If they were able to be done on production safely. If they were zero-downtime. If they guaranteed compatibility. If they weren't tied into other features so you had no choice but to upgrade but didn't want to. And so on.
Take Apache. Pretty much I can just upgrade it, while it's in use, while people are browsing sites running on it, across a whole raft of machines, there'll be a little downtime but after a few minutes it'll still be running with all the same previous config untouched. Except... even that's nowhere near perfect and I've had it bug out because of a config syntax change, etc. and just refuse to restart itself.
Hell, SSH I can do that while there are active connections and it just keeps servicing the active connections, upgrades the main process, and then all new connections are on the new version and all existing connections stay on the old version without any interference (so that you can, for example, upgrade SSH remotely in a safe manner without getting kicked out of the server that's mid-way through updating). That's how you do it!
SQL Server - ha. Take the server down for the day and pray it works when it comes back up.
For decades the attitude towards updates, even security updates, has been "Oh, well, you just have to accept downtime and if it fails, well clearly you didn't read the documentation, changelogs, rejig your entire site-wide config for this version (can't do it earlier, old versions don't support it, so it's change all your config, cross your fingers and pray it works first time) or need those features we decided you don't want for you".
And that's an attitude from software developers that stinks so bad that you can taste it as anyone managing any number of machines.
I think people here are getting confused between upgrading and rolling out new features.
I take a measured approach, I'm not a stick in the mud but I'm not entirely gung ho either.
On a hardware level. I don't upgrade on a whim, but I do upgrade regularly if only to allow me to sleep at night. Three years is about as old as I allow things to become. 5 years if it is a server. Sometimes I will upgrade server level stuff just to save running costs...if upgrading a full rack brings down the energy and cooling costs, I will upgrade the whole rack. Because Capex and Opex are not the same thing. Usually negotiating some capex for fixed assets isn't a big deal, but negotiating opex for a bunch of subscription based licenses is. If you want to be in a position to be able to spend more on licenses or new software, you need to be in a position to bring down the operational expenses...i.e. power bills, space etc to account for it...where capex is concerned, if it is possible to sell some of the old kit after deploying new kit (i.e. it's not too old and worthless) then upgrading hardware doesn't sting as much...in fact you can time an upgrade so that it coincides with an uptick in price on the old kit if you do it right...I've had situations in the past where it was possible to sell a load of kit for more than it originally cost which made an upgrade insanely cheap.
Client side stuff just isn't worth messing around with. Always keep the people at the coal face happy.
In terms of software...depends on what it is and how often it goes through major versions. We've all been there with systems that haven't been upgraded in a decade and a simple upgrade turns into a migration because there is no "one stop" way to upgrade...take Laravel for example...if you have a site based on that, you pretty much have to stay as current as possible otherwise it's going to be a ballache further down the line to upgrade.
Where cloud infrastructure is concerned, you absolutely need to keep on top of that...because the older an instance is, the more it costs...especially on Amazon...all you have to do there is change the instance type and boom it's upgraded and your pricing will be cheaper. I just halved the cost and increased the performance for one client that uses AWS simply by switching to a new instance type. Software is the same, OS is the same...but the underlying instance is newer tech and Amazon charges less for it...about half the price actually...and even though the core count and RAM allocation is the same, because the underlying infrastructure is newer, has higher clocks, better RAM etc...I got a 20% performance increase which has led to scaling up less often...so following a quite 20 minute change to the setup, I've brought their monthly bill down by more than a half...this particular client has a pair of RDS databases (MySQL, which were on 5.5 which Amazon is currently charging an extended support fee for on top of the underlying instance costs), so I upgraded to the newest available version of MySQL and I swapped the underlying instance from an M5 based instance to an M7g based instance...boom cost just disappeared.
Where rolling out new features is concerned, that's based on demand...if a customer wants it, I'll deploy it after some discussions about securing it, whether it opens up some operational issues etc etc...I'm entirely agnostic about rolling out new features if it has been properly discussed and managed. I won't do it on a whim, just because some user demands it, but I will do it if it has been carefully considered and there are organisation wide benefits and precautions taken to monitor the roll out and mitigate any potential security risks that might arise.
Because "move fast and break things" may be good for tech CEOs that win billions of dollars but get you fired of IT jobs.
Upgrade only when you have to and the stuff has been tested enough otherwise you end with many update and or upgrade disasters.
Also as mentioned many times in this same site, companies do not want to invest in their own cyber security, or waste a lot in upgrades more so if those are going to make things worse.
The fact Windows 11 now has any pre year 2020 computer in a blocklist does not help. Yeah you can bypass that but that risks Microsoft getting nasty in an upgrade and breaking everything.
They already break stuff in the "allowed" hardware list after all.
AC because not a lot of folks doing what I do.
Mission-critical software where people won't die but lives are made miserable for those who must use and support it, due to multiple software dependencies which are never in-sync is a real problem:
My clients are finance IT groups who are providing reports and data for their CFOs and in support of required US corporate filings (10Q etc). Failure to report in a timely manner can bring millions of dollars in fines and cause market instability. One of the biggest software providers for these services is Oracle. Oracle's tech stack requirements for the products are many and varied, and often contradictory and incomprehensible. For instance, one may use MS SQL but not SSL-enabled connections at initial configuration, but may be able to enable those connections at a later date. If you're on Azure and using MS' managed database solution, always-on SSL is a feature which cannot be disabled. Oracle could be more with the times and allow for this if they weren't inherently disincentivized to do so and instead concentrate on compatibility with their own DB product.
The tech stack relies on "Fusion Middleware" which for our purposes is their version of Apache and WebLogic, with Java in the mix. Java is 1.8.xx so you can imagine the security group is unhappy most of the time. Quarterly updates to patch these are available but unsafe to apply until the quarter following that release because the downstream components' coders don't have those fixes in time to check for compatibility problems. A client went to open up their 10K report and got.. nothing, because Java was updated. Wait a few months for them to release the fix, or roll the whole of the patch bundle back to the previous version isn't a great choice.
Browser incompatibilities run rampant - this quarter, Firefox works but Edge doesn't. Next quarter, use Chrome but not Firefox, Edge seems to be okay. This could be because of changes in the browsers, or due to issues with the Middleware code, likely a mixture. So that's fun for the end-user to navigate.
More companies are performing frequent security scans for vulnerabilities on the servers themselves, and additional pressure brought on the organizations to be more up-to-date with Java, OS and Middleware versions. However, those requirements aren't going to increase the Finance IT budget to allow extra hours for the thorough checkouts required post-patching, or provide the redundant environments required to ensure the patch cycle (dev-stage-production) is followed without impacting end-users. So either they're 6 months behind in fixes, or they're taking calls from the CFO asking why their pretty reports are now spitting %NULL% the day before they're due to report to the market.
Out-of-support costs: lack of support for old OSes, DB servers/licenses, and support (patches) from Oracle for software farther than 4 patch cycles behind (1 year) means Finance IT is being dragged kicking and screaming into the bleeding-edge of software whether they want to be or not. Granted, no one dies if the software stops working, but for the poor people who have their livelihood attached to corporate financial reporting, it's not helping.
Thank you for attending my T.E.D. talk.
"Turner said the reasons tend to fall into one of two categories. First, there are the mission-critical applications that can never be offline for any reason. Upgrading is, therefore, highly problematic and likely to be a business risk."
The third reason: because the new version (e.g., maybe, version 8.0 does not deliver like-for-like performance compared to version 5.7) is worse may be more common than both of them but together.
"When your application database version is multiple updates behind the rest of your systems, this adds to the workload and increases the challenge."
That's not true, if you can upgrade straight from version 5 to 8 then doing it in one go rather doing upgrades from 5 to 6, 6 to 7 and 7 to 8 a year apart reduces workload.
If you can't go straight from versions 5 to 8 then you are better off if the team doing the 6->7 and 7->8 upgrades have recent experience of doing the 5->6 upgrade
"Why aren't database servers being upgraded?"
It's a frigging database. If it works, why mess with it?
Punting out a new version will attract interest if it does something new and genuinely interesting, which will almost certainly require some code tweaking, or does exactly what it did before but more efficiently. It would have to be quite a lot more efficient to warrant the disruption of replacement.
Is the latest "upgrade" offering double the performance on identical hardware?
-A.
