Log4j and Omicron: Brothers in harm, mothers of invention

Infosec takes many cues from the human immune system. The analogies and metaphors are useful and apt: viruses, vectors, evolving a defence where learning is part of the response... It's not unreasonable to suggest that immunology and cybersecurity could learn a lot if they talked more. Sometimes, though, the parallels are far …

[disgruntled yank]

log4j

If only I could get a fifteen-minute answer on log4j.

[Unoriginal Handle]

Re: log4j

Here's a three liner:

1. Identify systems in your environment - and prioritise Internet-facing ones - which are using Log4j.

2. Check traffic to and from those systems, and allow only communication to and from trusted IPs

3. Block outbound connectivity to LDAP and RMI-IIOP services (port if you have to, application-based of you have a new-fangled application-aware firewall).

4. (Spanish inquisition time...) Change Log4j config to mitigate the threat, or upgrade...

[Doctor Syntax]

It's not difficult to find out what packages (or libraries) are installed on a Linux system. Debian and, I suppose, others have a mechanism for seeing what's installed. Debian has several options including synaptic which gives a sometimes useful commentary. It tells my that the Apache package isn't installed and that 2.17 is already available if I wanted to install it. It also tells me that I've got a Java successor, Logback, and a C++ API modelled after log4j which raises the question as to whether they might be bug-for-bug emulations of the original.

[Clausewitz 4.0]

Problem is the bundled JAR files with vulnerable packages hidden

[LDS]

Even on Linux, not everything that is installed comes from a package from a repository. Especially for commercial applications that are not distributed in that way, but even for open source ones that doesn't use that mean of distribution.

Maybe they are downloaded directly from GitHub, maybe they download packages from other repositories when installed.... and some of them may have forked some library or just copied libraries to "avoid issues with breaking changes"....

And they you have your dependencies' dependencies - maybe you never used log4j, but there's something that was automatically added that does...

[Doctor Syntax]

A simple find will locate any.

[doublelayer]

No, it won't. You may think that every Log4J jar is called that, but not only do you have the case problem, the version number problem, and the it-might-be-in-an-archive problem, but it can also be named something different if they wanted to do that. If you have a program that is already packaged in a jar file, it can contain dependencies inside of it and it's an archive file so find won't find it unless you also unzip everything first recursively. If you have a program that runs from source, the JVM might be running off a compiled version which has internal names. The files are there for you to find, but they won't be obvious unless you already know what their names are.

[Ian Johnston]

It's not difficult to find out what packages (or libraries) are installed on a Linux system.

Well, that's how it used to be. Now every snap, flatpak or appimage on the system comes with its own libraries which are only updated if an when the application maintainer thinks it's worthwhile. Because of course they all have time to follow every dependency of every dependency of every dependency of their work and rebuild the whole thing when one of them changes significantly.

Number of snaps on my system: 0

Number of flatpaks on my system: 0

Number of appimages on my system: 0

[Ilgaz]

Elephant in the room is Play Store

Seen update frenzy at Google Play Store? I didn't too.

[Richard 12]

Re: Elephant in the room is Play Store

Most of those things just don't log anything at all.

[Howard Sway]

Emergency response team

It's true that the virus pandemic and Log4J bugs are both serious problems, which require an emergency top level response. So I've decided to follow the same steps the government took, in order to reuse their experience in tackling an emergency, by following best practices.

Therefore my programmers have suspended all their current work, and have been ordered to spend all day this week on the urgent task of having parties and getting blootered on wine and coke instead, as this is apparently the expert way to act to solve a problem, and they're all "hard at work" when doing so.

[Wally Dug]

Re: Emergency response team

Surely the correct response is to:

10 Deny the existence of log4j.

20 Admit the existence of log4j - but it's only for web servers and we don't have any.

30 What? We do have web servers?

40 Well, it's not that serious. Is it?

50 Ah, well, we may have a problem, but there's no need for alarm.

60 There's no need for a lockdown of your systems.

70 Lockdown your systems.

80 It's fixed. Carry on, nothing else to see here.

90 What do you mean a variant of log4j?

100 GOTO 10

[Chris G]

Re: Emergency response team

As far as Omicron is concerned have a look at today's Telegraph, there may be reason to celebrate.

[Anonymous Coward]

Re: Emergency response team

@Howard Sway

"Therefore my programmers have suspended all their current work"

Shows how important 'your" programmers are. The world won't stop because of them will it?

The only thing that I.T. people are working hard on is finding a way, indeed anyway, to put the blame on someone else.

But to compare a deadly virus with a problem caused by I.T. is sick. I guess all the people who are suffering from the loss of a loved one are saying "if only it could have been Log4j instead of covid".

Moral? This is a tech page NOT a political pisstake page.

[Anonymous Coward]

There is always a price

to be paid for "free". And what happened to "many eyes"?

[John Robson]

Re: There is always a price

Because there has never been a security bug in closed source software.

[Ian Johnston]

Re: There is always a price

At least commercial developers have an incentive to improve things. Open sources ones don't. In fact it's worse than that, because in closed source support is a cost while in open source it is the only profit centre. Open source developers have a perverse incentive to do things badly.

[spireite]

Re: There is always a price

In my long career, I can count on one hand how many times I've seen commercial companies/programmer iappearing to have an incentive to improve things.

Answer: They generally don't

[Snake]

Re: never a security bug

"Because there has never been a security bug in closed source software."

That reply is avoiding the topic.

The point of the OP posting about the elephant in the room is that, yes, closed source has always had bugs. But for decades we've heard that the open source concept squashes bugs because of "many eyes on the code".

Yet some of the most serious bugs lately have been in open source projects. And the occasional response?

'But closed source!'

There should have been a "mea culpa" in regards to the blind belief that open-source human frailty would triumph over closed-source human frailty.

[Anonymous Coward]

Re: There is always a price

@John Robson

Of course there have been bugs in closed source software. The problem is that the "free software" people refuse to accept that there are bugs in free software.

I give you log4j as an example. I will probably be downvoted into Australia for saying this, but how old is this software and what happened to the many eyes thing?

At least with paid for software there is someone to sue...

P.S. I used to know a John Robson who lived in Keighley. Are you him?

[jtaylor]

Re: There is always a price

"At least with paid for software there is someone to sue..."

Is there? License agreements usually state that the product is sold "as is" and explicitly disclaim all liability. I'm not aware of any (US) statute that would support a lawsuit of the type you mention.

Have you heard of any lawsuits against suppliers over software vulns? Were they successful?

[Sandtitz]

Re: There is always a price

"And what happened to "many eyes"?"

I'm more interested in what happened to the many eyes in so many IT companies that use Log4j in so many products.

I've been dealing with several Log4j affected products and many companies just provide the usual mitigations: define log4j2.formatMsgNoLookups=true and/or remove JndiLookup.class. If it's so easy - why have you included the poorly thought defaults and/or unneeded Java classes?

Practically all of the big companies use Log4j but it seems that not in single one did anyone evaluate the software from security perspective.

What irks me as well is how companies have jumped on that first perfect 10.0 CVE vuln, but they're not necessarily tracking the additional vulns that popped up afterwards. When they are publishing updated firmware packages and mitigation instructions, you can't be sure which CVE's are addressed.

[spireite]

Re: There is always a price

Because they are too lazy to check.....

[tonkei]

Re: There is always a price

Lazy? Pssch.

There's no financial incentive to check _before there is an issue_ if there is an issue.

;D,

[Ian Johnston]

Re: There is always a price

Those would be the "many eyes" which entirely failed to spot a gaping hole in ssd for more than twenty years?

[spireite]

Re: There is always a price

To be honest, some of the flaws in closed source are just as bad.

The difference behind open and closed is that open probably get addressed sooner in a lot of case.

Downside is the number of forks etc that have the issue but are never maintained.

I suspect that you are more liekly to hear the flaws on OS sooner on average, and be able to react sooner

[Doctor Huh?]

Is anyone else seeing this? White Hat Log4Shell probes outnumber Black Hat probes

One odd thing I'm seeing in my server logs. Over half of the door-knocking probes for the Log4Shell problem seem to be coming from obvious white-hats, presumably looking for bug bounties/honoraria. I'm actually struck by how competitive that market seems to be.

Is anyone else seeing a similar trend?

[Clausewitz 4.0]

Re: Is anyone else seeing this? White Hat Log4Shell probes outnumber Black Hat probes

@Doctor Huh?

Maybe the Blacks are after more valuable targets, and prefer to do more basic scan / reconaissence before firing up an exploit.

[Penguin Of Evil]

Nobody Ever Acts Anyway

So today it's log4j, last year it was openssl, and on it goes - and none if it ever really gets fixed

Today for example:

- None of the major anti-virus/malware protection/security warn you about things like apps containing ancient openssl and the like thus pressuring the offenders to update their apps

- A Microsoft subsiduary appearrs to be shipping obsolete OpenSSL code in a major MMO title according to the strings in it ("OpenSSL 1.1.0e 16 Feb 2017")

Even worse the file listing the third party packages used is a long list of obsolete things and is bundled in plaintext with the package

Nobody creating software is looking

Nobody scanning software is looking

Nobody installing software is looking

Nobody creating software is acting

Almost Nobody using free software is investing in it enough (ditto proprietary though)

because nobody creating software is going to get sued.

I worked in a large tech company. The small areas of the business that were subject to liability law (drones, medical, self driving) had rigorous and well managed compliance and analysis, code scanning, security policies and pre-release steps. The rest of the business had a totally inadequate (but still better than industry average) whackamole and attempted policy enforcement group.

We need laws placing liability on those who sell, rent or similar software and services just like we do with physical product. Only then will this mess get fixed

[SecretSonOfHG]

Re: Nobody Ever Acts Anyway

That's an interesting point, but be aware that the liable areas are so well covered because of their... liability, and that's because something going wrong in those has significant impact on people's lives or wellbeing.

And we have a much, much wider range of applications where the impact of a security incident is likely minimal to small annoyance to actually none. Making these services liable on the same scale as the others will likely put them out of business in short time.

[Penguin Of Evil]

Re: Nobody Ever Acts Anyway

Liability law is not the same as making people automatically liable. I agree entirely that making people automatically magically liabile for something is bad.

Liability law is about showing you acted with reasonable care. So for example if you ship a product built to good standards of desig and safety and someone manages to kill themselves on it you are not automatically liable. This is true in the aspects where liability law already intersects software. Shipping the latest current well regarded OpenSSL and a bug being found isn't going to get you successfully sued. Shipping a 2017 era OpenSSL that didn't get updated on the other hand you deserve a kick up the backside. Nothing else is going to change the behaviour of the IT giants.

Another problem is that it's going to become a national security issue. When Ukraine gets invaded or China finally eats Taiwan you can be sure that the resulting cold war is going to involve them exploiting every hole they can and every control of European/US software, and of course vice versa. There are reasons China doesn't use US chips in security roles, and the Russians have things like Elbrus.

[Anonymous Coward]

The log4j-detector project on Github was invaluable for us, we found all our nested instances in Prod in a couple of hours last Monday and ultimately removed the JNDILoookup class after the initial mitigation proved to be a false dawn. Took the Dev Team 3 days to come up with the info, and even then they had missed a couple of instances.

[Dude-named-Ben]

Mothers of invention mmmmmm

Second time today somebody references Frank Zappa and the Mothers of Invention

Wasn't their first album called "Freak Out"?

[Filippo]

> IT doesn't take itself as seriously as a responsible production engineering discipline

That is absolutely true.

Unfortunately, creating software that is actually as safe as a house takes something like 5x more time and money, compared to the "eh-good-enough" level of quality that we are accustomed to.

Good luck finding customers that will spring for that, unless you make it compulsory by law.

[Penguin Of Evil]

There are good studies that show the "it costs five times as much" claim is false. There is a significant initial investment but after that the cost of writing software properly is actually quite low because the bug rate is massively lower in the first place. Force it to happen at scale and it'll be cheaper still.

The law thing absolutely - right now we have what economists call a "lemon market" for software. The bad drives out the good due to pricing and the inability to tell crap from quality.

[Chris G]

I wonder what the final cost is, of buying affordable unfinished software that will depend on patching and updating throughout it's life.

Windows and flash ought to have taught the lesson but it seems it is better to get the new shiny stuff out there and allow the users to fault find so that the product and users can be 'saved' by the caring seller, who undoubtedly takes the security/privacy/wellbeing/money of their users extremely seriously.