Anatomy of OpenBSD's OpenSMTPD hijack hole: How a malicious sender address can lead to remote pwnage

The OpenBSD project's OpenSMTPD can be potentially hijacked by a maliciously crafted incoming email. Infosec biz Qualys discovered and this week disclosed CVE-2020-7247, a root privilege-escalation and remote code execution flaw in OpenSMTPD. It can be exploited locally by a normal user to execute shell commands as root, if …

[-tim]

How?

If a system program needs to run another program, it should not use a shell as an intermediary.

I've been working with email systems for decades and it is amazing that the same bugs have shown up in so many different programs. My advice to anyone writing a email client is to go get a list of the top 5 major exploits of all the top email programs and make sure your code doesn't do any of them.

[Phil O'Sophical]

Re: How?

If a system program needs to run another program, it should not use a shell as an intermediary.

I wish I could upvote you more than once. I've seen this lazy programming technique cause so many security issues that I've lost count. No amount of string sanitizing can make it 100% safe.

[GnuTzu]

Re: How?

Well, I'll add to the votes then. It's like one of those fools writing a CGI program with Bash and not validating parameters... or with Perl and failing to use taint properly.

[Anonymous Coward]

Re: How?

Isn't that the *nix philosophy though? Small programs that just do one thing, with a shell joining them together to do more complicated things.

[GnuTzu]

Re: How?

When your an administrator writing an init script or parsing logs from a properly authenticated shell, to a point, yes. When your writing a service for which a TCP or UDP port is opened to the entire Internet... well, to what degree do you want to be low-hanging fruit?

[Anonymous Coward]

"Isn't that the *nix philosophy"

Unluckily, yes. It did made sense in 1970 when you had to chain programs because memory was measured in kilobytes, and development tools were primitive. Sometimes traditions just become bad habits.

[Cem Ayin]

Re: "Isn't that the *nix philosophy"

You have it backwards: it is not the component-based design of the Unix UI as such that is to blame. Quite the contrary: reusable components are an efficient and highly successful concept. Where the age of the Unix shell(s) is showing is the poor way of "glueing" the components together: by passing all data as strings, which requires serialisation and parsing to be implemented over and over again - which in turn means asking, if not begging, for trouble.

(If only I had a $ for every quoting error I've encountered in my professional career...)

Hate to say it, but the PowerShell is the one thing that Microsoft did get right in terms of OS interface design.

[Dan 55]

Re: "Isn't that the *nix philosophy"

I think you mean, if only you had a \$.

[vulture65537]

Re: How?

too right, and common knowledge since at least 1994

http://sunmanagers.mrbill.net/1994/0509.html

[Brian Miller]

Re: How?

I am simply aghast at this rookie code. Seriously, there are two errors here: spawning shell each fscking time an email comes in, and trying to sanitize the address. Here is how to sanitize an address: do not do it, encode it, and keep it encoded, or hash it, and use the hash.

But, oh, spawning /bin/sh ... they should be put into stocks. Really.

[Dan 55]

Nested ifs

This is why I don't like them as it's more difficult to follow the logic, especially with clever ands and ors in them to save a line. Better to keep the structure flatter and return when a check fails/succeeds and carry on if otherwise.

Which is what the patch does.

[Rich 2]

Re: Nested ifs

Completely disagree. A function has one entry point. It should also have one exit point - the bottom.

Bailing out at arbitrary places so often leads to resource and memory leaks, and similar issues, and contorts the logic.

The code is complete crap, but for many many reasons not related to using 'if'

[Dan 55]

Re: Nested ifs

I'm not sure why a function must have only one return point if in turn it means it becomes more difficult to understand, you've just seen an example with this patch. If there are 4-5 levels of ifs, I know which is logic is more contorted.

For bailing out at arbitrary places we can avail ourselves of a goto (HERITIC!) to tidy up at the end in C or RAII in C++.

[Rich 2]

Re: Nested ifs

You don't have to resort to very deeply nested code to follow a rule of one exit point. And even if you do, the logic is still sound; you can easily back-up the nesting to see what conditions were necessary to get to where you are.

Bailing out early means that the logic is subverted; it is almost akin to using 'goto'. I also dislike 'break' (except in a switch) and 'continue' for the same reasons; they mangle the logical flow by effectively behaving like 'goto'. They are also much easier to overlook than a correctly nested condition which makes reading and modifying the code more difficult

[Dan 55]

Re: Nested ifs

I guess it's all the same to the compiler in the end, if not to humans. The Stack Overflow peanut gallery (the answer highlighted gives my favourite example) favours returning early and often, though.

[bazza]

This is why command line interfaces should have no role other than for direct interaction with a user in a terminal. The parameters passed on a CLI are untyped, uncounted, open to interpretation by other things such as the shell processor (which is what’s happened here), with the emphasis for parameter validation placed on the caller not the callee.

With this particular example I don’t like the philosophy of it’s use of environment variables to say what’s valid, what’s not. Effectively they’re saying “over to you, admin, and well not tell you why it matters unless you read our source code”. Dangerous content is a function of what the system’s shell processor does with characters like ; and whilst those are well understood, the information as to what constitutes correct configuration lies outside of the control of the opensmtpd project.

So if the bash or sh or tcsh or whatever project goes and makes a change, that could open up new holes in opensmtpd that the opensmtpd project are unaware of and the shell project is under no obligation to guard against. Oops.

We’ve seen this elsewhere with hdd encryption in Linux, one instance resulting in the effective password being empty.

[Luke McCarthy]

Unfortunately, Unix does not provide a way to call another program, or send a message to another program, in a type-checked way. (If it did, command-line syntax parsing could be automatically generated instead of hand-written). In other words, the OS level API is too low-level, burdening every program with the task of correctly serialising and parsing data which crosses a process barrier, a hairy and difficult task even for the experienced developer.

[Phil O'Sophical]

The problem isn't with the destination program parsing the data, but with an intermediate (the shell) parsing it with unexpected side-effects before the data gets to the desitination. If one program needs to call another, and pass parameters, it should do it directly with the standard fork/exec model.

The downside is that you don't then have direct access to some shell features like redirect, and lazy programmers prefer to invoke a shell to do it rather than manage the IO themselves.

[Luke McCarthy]

Yes, true in this case. I was speaking more generally. Invoking the shell is the only reliable way to parse command lines exactly how the shell would parse them, since there is not a standard library function to do this. This mda_command appears to be plucked from smtpd.conf:

> deliver to mda program [ as user]

> Mail is piped to the specified program, which is run with the privileges of the specified user or the user the message is destined to. This parameter may use conversion specifiers that are expanded before use (see .B Format Specifiers .)

I guess someone decided they would like to put arbitrary shell syntax here, to allow for variable expansion and such, and correct interpretation of quoted strings.

[Claptrap314]

You are assuming that the command line is the only way that a program should be receiving data. THAT is hugely wrong, and is a common limitation of a lot of unix programs. Newer ones provide better (and much safer) interfaces for IPC.

[bazza]

Unfortunately, Unix does not provide a way to call another program, or send a message to another program, in a type-checked way

Well it does, there’s the idea of dynamically loading a library, which is how an awful lot of stuff works. The problem is useful functionality being built into an app, and not into a library that the app (and indeed any other app) can use to provide the functionality.

Then there’s fairly common things like DBus, which allows communication between applications so that one can ask another to do something on its behalf.

[Claptrap314]

There are fifty ways to pass a message

Drop the data in a file, Kyle

Send it on a socket, Crocket

Put it in shared memory, Stephanie

...

[TimMaher]

Re: There are fifty ways to pass a message

Hop on the bus, Gus?

[Phil Endecott]

Re: There are fifty ways to pass a message

Hop on DBus, Gus...

[Anonymous Coward]

So much for the vaunted OpenBSD security

If a simple shellcode injection attack can be made against its mail program!

[Dan 55]

Re: So much for the vaunted OpenBSD security

There will be a post-mortem soon posted to the mailing list on the mailing list and just as other problems have spawned better solutions on BSD which have been ported to Linux (e.g. a better random number generator, OpenSSL -> LibreSSL), I'm sure this one will do too.

[_LC_]

When will we get rid of this malady?

That's the real culprit:

> execle("/bin/sh", "/bin/sh", "-c", mda_command, (char *)NULL, mda_environ);

The issue here is "the typical Unix problem": Programs are starting other programs to do things. The correct approach would be putting the code in libraries. The shell commands would then only be handling the command-line options, passing them on to the library. Programs could use the libraries "as part of their code". This is the safe way. This is the efficient way. But they never learn...

[Anonymous Coward]

Re: When will we get rid of this malady?

What I don't understand is how it is possible something like that was in OpenBSD, which supposedly puts security first, does careful audits of all code etc.

I'd say that doing a grep for "/bin/sh" would be a pretty obvious step for looking for stupid insecure code written by someone with a 1980s everyone-on-the-internet-can-be-trusted mindset. If something this simple was missed by them, how many other less obvious security flaws are out there? Perhaps the reason why OpenBSD has comparatively few security issues found is that hardly anyone uses it so attackers don't even bother looking for its holes.

[Anonymous Coward]

Re: When will we get rid of this malady?

The issue is their claim to fame is remote holes in the default configuration. Well, that’s pretty easy to claim when by default you don’t do anything!

Eg, they don’t claim this bug as an issue since it’s not the default configuration.

[Michael Wojcik]

Re: When will we get rid of this malady?

The OpenSMTPD project claims it's "part of the OpenBSD project", but OpenBSD itself lists it as an "associated project". It appears to be primarily the work of two developers, neither of whom is Theo de Raadt. I do think it's unfortunate that OpenBSD adopted OpenSMTPD without challenging it on this very poor architectural decision, though.

The first of OpenSMTPD's stated goals is "Be as secure as possible". Exec'ing the shell with tainted input on the command line is not compatible with that goal, regardless of how much whitelisting and escaping you try to do.

I also find it disturbing that this bug was reportedly introduced in 2018. There was an OpenSMTPD update in 2015 that fixed various security holes (and looking at the diffs is not encouraging, frankly). Then sometime over that five-year gap, someone decided to make a change that created a severe vulnerability. Where was the code review for that? What improvement was that change meant to deliver? Public-facing network services are the most prominent facets of the attack surface, and should receive the most scrutiny, but this 2018 change doesn't seem to have registered on the OpenSMTPD project website.

Also, I'm curious to know what's supposed to justify OpenSMTPD as an alternative to, say, qmail, or a new project based on qmail. Was writing a new MTA in C really the best idea?

And, seriously, any decent static-code analyzer with data-flow analysis should have been able to catch this. A dynamic-analysis tool that explores untested code paths - even something like AFL - should have been able to catch the offending case too. Seems like the OpenSMTPD team isn't making use of tooling to help catch vulnerabilities. That, too, is a failure to live up to their own goals.

All that said, using this (really quite appalling) error as an excuse for a blanket condemnation of OpenBSD is simplistic to the point of uselessness. OpenBSD has addressed many other vulnerabilities, and no non-trivial system is perfectly secure. We may hope that this incident leads the OpenBSD team to turn a more critical eye on their associated projects.

[-tim]

Re: When will we get rid of this malady?

The worst part of that execle system call is had they left out the 1st 2 parameters, not combined the command and it's argument in a string, the code would have been smaller and correct.

[iron]

The Morris worm strikes again!

It is worrying that malware techniques from when I was at school are still applicable today. By now we should have found a way to prevent them.

[phuzz]

Re: The Morris worm strikes again!

The technique they used wasn't really a vulnerability. They didn't have space in the email address to store their payload, so they put it in the body of the email, and retrieved it once their initial attack had succeeded.

There's presumable already protections to stop someone executing code from the email body, but once the researchers had got root access, the protections were worthless.

[Michael Wojcik]

Re: The Morris worm strikes again!

Oh, we've found ways to prevent them, or at least make them much harder to exploit. The problem is that many developers aren't interested in using those approaches.

[Rich 2]

Crap code?

When OpenSMTPD was first released some years ago (when I was still using OpenBSD), I took a look at the code and commented on one of the OBSD forum threads at the time that the code was ...errr how do I put this politely? ...crap. Unless it's changed significantly over time, it was very poorly structured, had almost zero comments in it, etc etc ..really just everything that could be bad about it, WAS bad.

I might add, that I wasn't just slagging it off to be a troll (and I'm not now). My point was, should OpenBSD be including such poor code into the base system. Not at all surprisingly, I was resoundingly condemned on the thread and told that if I thought there were problems then I should submit patches. The problem was the patch would have consisted of "delete everything and replace it with something else".

My only surprise is that it's taken this long for a bug like this to surface, and I despair at the exceedingly low bar that is set for code quality in some open source projects.

[whitepines]

Re: Crap code?

I despair at the exceedingly low bar that is set for code quality in some open source projects.

While this is not in dispute, at least you can see the lack of quality and make a decision to use something else. Proprietary codebases are often even worse (offshored to the lowest bidder, budget allocated to marketing, etc.), and there's nothing you can do about those except disconnect the system from the Internet.

[s. pam]

Robert Morris Jr. Anyone?

Gosh golly its like I just fell into a wayback machine and am back at UC Berkeley in 1988.

And that's NOT a good thing. I thought the OpenBSD folks had more of a clue. I guess their clue bag has a hole.

[Chris Gray 1]

With great power...

(By chance, just watched Into the Spiderverse last night...)

The "Unix model" includes lots of very powerful tools, such as being able to use a shell from within programs. In cases like this, the tool is *too* powerful. Using a full programmable shell for process invocations not much more complex than using vfork()/exec() is overkill, and dangerous, as others have mentioned.

It shouldn't be hard to write a *much* simpler mini-shell, which is intended for such situations. It would take time to wean programmers off of using "bash", etc. but if you give it a few "modes" for the kinds of shell-like things it will do, it shouldn't be hard to handle most cases, and would be far safer. Yes, its slightly more total code to be maintained, but worth it.

[vulture65537]

Re: With great power...

You mean it's time someone invented smrsh - the sendmail restricted shell?

https://www.linuxtopia.org/online_books/linux_system_administration/securing_and_optimizing_linux/chap22sec182.html

[Claptrap314]

Re: With great power...

How about just fixing the programming model instead?

[Claptrap314]

Many (most?) unix commands are !#/dev/garbage

Thinking about this bug, I cannot help but be upset at the core unix model: many, many programs have a LOT of functionality that can only be controlled via the command line. As other have noted, this is okay-ish if we are all friends. But we are not all friends.

Programs do not exist in isolation. They are part of much, much larger ecosystems that exist in order to make certain people happy. Given that it is impossible to predict what purposes users might want to make of any given application, or in what environments, ethical programming requires that the programmer exercise care to ensure that their programs are easy to invoke safely and produce output that is easy to use.

If a program cannot receive its control input via stdin, it is broken.

If a program cannot receive its control input via config file, it is broken.

If a program can reasonably expect that its output might be used for input, but only after further processing, it is broken.

If a program has command line options that require some sort of Turing-complete system to parse, it is broken.

If a running process cannot have its config updated either via an api call on a port, or by reading a config file after a sig 1, it is broken.

If a long running process does not dump its config (on startup and on change) to a log file in an easy-to-parse fashion, it is broken.

The command line is simply too dangerous an environment for arbitrary data to pass through. In secure mode, programs should refuse to take data on the command line that can be arbitrary text.

[_LC_]

Re: Many (most?) unix commands are !#/dev/garbage

Even more so, the programs can change (take the old mkudffs, for example, which only recently received an update breaking scripts). If you're lucky, your program will stop working then. If(!) you are lucky!

[cs9]

OpenBSD: over 2 days without a trivially exploited remote root vulnerability

Give the developers a break they were blinded by light reflected off their tinfoil hats when they wrote that code.