Patch management still seemingly abysmal because no one wants the job

Patching: The bane of every IT professional's existence. It's a thankless, laborious job that no one wants to do, goes unappreciated when it interrupts work, and yet it's more critical than ever in this modern threat landscape. So color this vulture surprised to learn that, in the decade since he left an IT career for wordier …

[Bebu]

A couple of things strike me here...

Having security, system administration and operations siloed invariably means the duck gets well and truly shoved.

From my own experience dealing the various concerns of each as a unified whole leads to more optimal prioritization of rectification task with some symbioses - shooting more than one duck with a single round. Also a single reporting structure limits the quantity of duck shoving possible.

The complaint about the ever increasing number of applications installed on "end points" outrunning the capability of securing and maintaing them also makes me question whether the system architecture is well matched to these objectives.

In this context "end point" seems to mean both desktop and server machines (cf OSI End System (ES) v. Intermediate System (IS)) but the desktop systems are likely the most problematic which leaves me wondering whether dumb clients that basically just do I/O over secured channels might be ready for a renaissance. Places almost all the load on servers (these days likely to be cloud/cluster anyway) but does significantly increase manageability and potentially security.

[StewartWhite]

Let's just ignore the elephant in the room shall we?

What the article and all of these reports completely fail to even mention is that patching should be an exception to the rule. Why have we (IT as well as "the business") allowed ourselves to think that having endless patches is the answer to the problem? Microsoft et al are allowed to get away with writing dismal quality code that has systemic flaws (e.g. let's require AV tools that mess about at kernel level because security is such a tedious subject - CrowdStrike anybody?).

I attended a cybersecurity/finance conference a few weeks ago which was profoundly depressing as all of the vendors were boasting about how much more money they'll be making as they add yet more levels of snake oil (AI just being the most obvious) whilst casually stating that the security situation was only going to get worse but they didn't care, just think about those lovely $$$$$!

How about people in the IT world stating some home truths the next time somebody in the deified "business" asks for something stupid to be completed by yesterday. If they whinge to the board, just mention "Blockchain" (substitute AI or another mot de jour as appropriate) and ask them exactly how much money they spaffed on it for absolutely zero return.

The current setup isn't working and is getting worse with no possibility of a solution unless we opt for radical change, e.g. let's put Agile out of its misery as it's just a cargo cult for "design" by committee.

[sketharaman]

Re: Let's just ignore the elephant in the room shall we?

Great point! My customer told me that his son's Fortnite on Xbox does 2GB updates every few days. I'm old enough to remember a time when we played computer games off of a 360KB floppy diskette and cannot understand why a video game should occupy 7GB but my customer is a digital native and half my age and even he's speechless why a game update should be 2GB.

[Jou (Mxyzptlk)]

Re: Let's just ignore the elephant in the room shall we?

Make the game update greater 100 GB, and need 300+GB of free space. Will make it to the news :D.

[stiine]

Re: Let's just ignore the elephant in the room shall we?

Is this because the game runs as a vm and the update is a new vm image?

[stiine]

Re: Let's just ignore the elephant in the room shall we?

You're at least 30 years too late.

Can it change? Yes. Will it change? I don't expect io change.

[Dagg]

Re: Let's just ignore the elephant in the room shall we?

What the article and all of these reports completely fail to even mention is that patching should be an exception to the rule. Why have we (IT as well as "the business") allowed ourselves to think that having endless patches is the answer to the problem?

It depends on the cause of the patch. I worked in the retail pharmacy industry in Australia which has public health care and we had to deal with endless government legislation changes, national health service changes, Pharmaceutical Benefits Scheme changes, drug vendor changes. Most could be managed by configuration updates but some needed code changes especially around dispensing rules. We also need to be aware of changes to doctor dispensing software and script switching services.

There was also the completion between the different pharmacy retailers. So the whole thing was very much like the Red Queen in Alice In Wonderland.

[DrkShadow]

Re: Let's just ignore the elephant in the room shall we?

Curious, with what permissions should an application run that watches/logs all Windows-level API calls, looking for function-call patterns in detecting malware?

Is this anti-virus? Can it run in user-space?

[ChoHag]

Your rate of internal systems churn necessarily follows the rate at which your vendors improve (or just fix) their products. How could it be any other way? If you choose a vendor which requires a weekly cadence that's on you.

[ecofeco]

I can neither upvote not downvote you.

Choosing a vendor is often not in the hands of IT, but in the hands of the C-suite.

However, I have seen IT also making bad choices.

[Lee D]

Push the patches, it breaks the system, you get blamed.

Don't push the patches, the system gets compromised, you get blamed.

The only way to break that deadlock is PATCHES THAT DON'T BREAK THE SYSTEM.

i.e. vendors need to test their damn patches and not release them, and patches need to be the absolute minimum necessary to resolve the problem to ensure minimal impact - NOT bundling new features.

They also need to be "zero-downtime" patches (i.e. not rebooting the entire machine just because I updated Word with a security update). We also need proper rollback that works - i.e. with the recent Crowdstrike nonsense, why oh why couldn't Windows detect that it was in a bluescreen/reboot loop and revert back to BEFORE THE LAST CHANGE THAT MADE THAT HAPPEN? It's supposed to be able to do that and it doesn't (yeah, yeah, "3rd party", etc. etc... well that 3rd party has an in-kernel device driver... and that's the OS supposed to be managing that!).

We have all this. It exists. Other OS can do it. We used to be able to do it on older OS and different OS. We can still do it now. We can definitely arrange for it to happen in the future.

But you know who needs to pull their finger out? OS providers, especially Microsoft.

[Julz]

I

Think a more interesting question is; why are there so many patches? A combination of poor release and testing systems combined with a high threat area and intensity are probably two of the main factors. Perhaps those areas could be improved upon which would lead to a much happier life for both the poor sods applying the patches and the end users experience of the system.

StewartWhite: just read your post. I agree, we should not just tolerate the current mess.

[Headley_Grange]

Re: I

I don't know why there are so many patches, but I can guess. MS releases Windows before it's ready; this earns them money sooner, creates a hype about a new releases, creates a market for new devices which forces an unneccesary refresh cycle on potential competitors that helps keep them poor.

There's no correlation between performance and choice in this sector, partly because, realistically, there's no competition and partly because government and huge corporate players buy Windows which locks a bunch of suppliers and customers into it too. As a result of this Windows has a great range of business-focused apps; accounts, timesheets, data cubes, circuit analysis/synthesis, ERP, MRP, EDM; some of them MS proprietary but most of them not. This isn't because Windows is a great OS, it's because Windows is, effectively, the only OS. The fact that it needs patching every week is just a cost of using it to businesses and no different from having to put petrol in the car and top the oil up every so often.

There's no real competition because companies are lazy. FOSS operating systems are various and from a board-of-directors view look expensive and risky. If a company uses Windows then it doesn't need to train most of its workforce in its use. If it goes to Linux then it's got a problem with the UI, then app availability/compatibility and that's before the MD complains that he can't make his home Hi Fi Bluetooth dongle work.

I don't think there's an answer. Splitting MS in two to create competition would make the situation worse because new and shiny sells better than old and reliable so both companies would race to release new stuff before it was ready.

[Jou (Mxyzptlk)]

Re: I

It is money. That is the god it boils down to in the end.

[Mike 137]

"That's not where you want to be from a compliance perspective"

"Compliance"? I thought patching was about security (it should be).

Given the incidence of patches across the technologies in use, patching should be a dedicated job function, and should be conducted continuously with reference to criticality. So instead of patching "immediately", "within a week", "within a month" or whatever ("compliance"), patches should be sorted into a prioritised stack as they're released, with the most critical at the top (e.g. by CVSS rating), to be pulled from the top of the stack and deployed as fast as possible conversant with proper testing (security).

[Headley_Grange]

Re: "That's not where you want to be from a compliance perspective"

I think you're spot on, but I bet that there are insurance policies out there that wouldn't pay out if you weren't running the "latest" version of the OS when some sort of problem occurred.

[fnusnu]

Re: "That's not where you want to be from a compliance perspective"

Good luck trying to do that in an enterprise with hundreds, if not thousands, of applications.

[Giles C]

First job when i started at the company

After coming back to my employer (previously was a contractor) I had to sort out the Cisco estate patching, which in some cases had 7 year old code or worse.

It took 6 months but there is now a process in place to ensure all the sites are upgraded once every three months (if required) and everything runs on gold or gold-1 images.

It was a lot of work but worth it in the end.

Fortunately windows Linux etc are out of my scope.

[Khaptain]

Bitten too many times

I would make a bet that any admin on this forum that has been in the job for the last 10 years has been bitten because of patching on multiple occasions..

I can only say thank you to the God of Snapshots for giving some breathing room.

[Curious]

When will software companies stop bundling "features" and "ui changes" (requiring redocumentation and full revalidations) with their security fixes?

And Microsoft, trying to push new Teams and new Outlook as per-user-profile maintained apps does not help with these huge-attack-window applications (Teams shows as the highest impact threat in MS defender m365 Inventories despite it's autoupdate.).

[MatthewSt]

Especially when Microsoft have already got multiple distribution methods available to them. If they actually made use of the Store for their apps then patching, disk usage, bandwidth are all taken care of automatically. Even supports zero downtime patching (new version is installed side by side and executes next time you start the app)

(new Outlook uses the Store, new Teams is MSIX but not store)

[Throatwarbler Mangrove]

All of this and . . .

There's a wide variety of software, all with its own foibles and requirements for being updated. With Windows, of course, the installer itself is so complicated that a patch management tool has to carefully trap errors and output to determine whether a patch push was successful, but Microsoft has actually done a lot of work to try to consolidate patching of their applications into a coherent structure. If you're running a full Microsoft stack, Endpoint Manager or whatever they're calling SCCM these days is not the easiest product to learn, but you can at least manage updates from a single location. Try updating an Oracle application or a more obscure third-party application or Cisco firmware and see what a joy that is! Our network infrastructure at my last job tended to run ancient versions of Cisco code because the network admins were afraid of the consequences of upgrading; a network upgrade that goes sideways is, of course, a disaster. You'd think that Cisco, controlling both the hardware and software stack and having a relatively limited number of configurations compared to a general-purpose OS, would be able to deliver software updates that were bulletproof but apparently not. It certainly seems as though many technology vendors subscribe to the notion that there's more money to be made by creating problems than solving them.

[Anonymous Coward]

More than one vacation…

…was paid for by the need to come into an multi-floor office building and ensure all the PCs were turned on to accept patches, one weekend a month.

We got paid a full day at 1.5x even if we smashed it out quicker. The thinking being that we’d just stretch it out to get the full day anyway, if needs be.

God bless patching!

[harrys]

Pragmatism pragmatism pragmatism....

Systems are imperfect will always be, once I lowered my expectations my work satisfication increased

At the same time my prgmatism then motivated me to find solutions that allowed me to carty on having an easy life WHEN not IF things messed up

Pragmatism also told me no one likes spending money on "insurance" so just implemented things peicemeal on core assetts first

But main thing is you have to play politics, if you want something done then ends justify the means .... the bigger the prick the bigger the ego, but massaging it will get the what you want :)

Just have a shower later and talk it out later with non work collegues how you manipulated a total arseole at work today to get what you needed

Never never underestimate the vanity of people in power, thats the qualities that got them there in the first place :)

[sketharaman]

Awesome article on a rarely-covered topic. On the back of incidents like CrowdStrike, it's easy to say "all software must have latest updates", "you should not allow vendors to update your software", "you should test all fixes before applying them on production" etc. But this article makes it clear why it's not so easy to follow that advice.

[Anonymous Coward]

Its not so cheap, either.

[ecofeco]

I don't about passing the buck

But I do know about no budget or time to focus on it.

As far as the C-Suite is concerned, the IT dept should not even exist. After all, the vendors told them their IT products and services were perfect and carefree!

[Ryan D]

And get it done by yesterday

From the front lines, true story.

I constantly hear the demand to patch faster and to get it done is as short a cycle / cadence as possible. Why give us 7 days when they can demand 5 instead.

I mean, what could go wrong with rushed, under tested patches?

Oh, and do that in our two production environments which are mirrored and need to stay live 99.999999% of the time and , and , and. Yeah, 2500 servers (mostly RHEL, but yes there are Windows boxes in the mix) with varying support from the LOB app owners. Who need to sign off on every damn one before we can do final packaging and predeployment. What could go wrong?

Oh, and for bonus points, do this while being audited for PCI compliance.

[Uh, Mike]

Quiet Heroes

I used to love going to SANS,

but then returning home was disappointing,

to spend all my time pushing patches,

instead of chasing intrusions.

Still, we had the luxury of an integration lab

where we could certify the patches.

Eventually we got the authority

to audit and enforce patch application.

The places that didn't go down

are where the heroic work took place

before the disaster.