Cloudflare gave everyone a 30-minute break from a chunk of the internet yesterday: Here's how they did it Internet services outfit Cloudflare took careful aim and unloaded both barrels at its feet yesterday, taking out a large chunk of the internet as it did so. In an impressive act of openness, the company posted a distressingly detailed post-mortem on the cockwomblery that led to the outage. The Register also spoke to a weary …
the updates are pushed out to a small group of customers "who tend to be a little bit cheeky with us" and "do naughty things" before it is progressively rolled out to the wider world.
Nice!
Still, if you are a cloudflare customer who likes to be on the bleeding edge, simply do "naughty things"!
Unless you can step through the parsing engine and test every possible input, it is very hard to spot issues with non-trivial regexes.
This is a problem that has been known about for a long time, and there is a nice discussion of it in Larry Wall's "Programming Perl" where using lookaheads in an expression in a certain way can cause it to never complete execution (at least not in the expected lifetime of the universe).
I'm not saying not to use them, just be very, very careful when you do, and make sure you've double-checked you understand what the expression is actually asking for!
If it caused 100% CPU on every machine, then it can't have been a particularly obscure or rare string. Presumably their testing didn't include a representative set of data to test on, otherwise it should have showed up straight away.
It's one thing making a mistake that only cocks up occasionally under rare circumstances, it's another to immediately peg every CPU to max...
Without knowing the nature of the regex in question, it could have been anywhere between common and obvious to rare and subtle, and without being a master of reading regexes (like SQL execution plans, it's more an art than a science), it may still have been easily missed on visual inspection.
It could, for instance, have been a pattern that causes an infinite loop under certain circumstances, which aren't found in the test environment (because of a missing test case) but are common in the wild, and its possible that only a single instance of the offending pattern was required in the wild to trigger the regex in that way. Again, without seeing the regex, it's impossible to say one way or the other.
Of course, it could be that the regex in question is longer than this entire comments thread, and it'll take you a couple of weeks to work out what it's for, let alone what it's actually doing.
Case in point - google "email address regular expression" and look at some of the examples.
Some of them are utterly "out there" to be sure - every book I've got on perl or just things like "mastering Regexes" point out that this particular job is not possible to correctly implement with a regex, and if you really want to validate email addresses, you should use some programming in whatever language you're using and only use smaller regexes for the parts they serve best.
I think 100% of the references warn of this, or at least, all the one's I've seen. It's like "this is an example of what not to do" in the ones I have read.
"reading regexes (like SQL execution plans, it's more an art than a science)"
I wonder if it's possible to have a regex parser compile in test mode to give an execution plan. But I suspect the result would be a lot harder to understand than an SQL execution plan.
My guess is the pipeline is something along the lines of:
- write a rule
- validate rule and add to rulebase
- check rulebase in monitor mode against pre-canned test traffic
- check rulebase in enforce mode against pre-canned test traffic
- check rulebase in monitor mode against sample traffic containing items to block
- check rulebase in enforce mode against sample traffic containing items to block
- check rulebase in monitor mode against production traffic for select customers
- check rulebase in enforce mode against production test traffic for select customers
- deploy to production
This is based largely on (historical?) Google checks for firewall rule changes. As long as the hit counts/device health stats don't show anything scary, everything should be good.
I wonder if the canned/sample traffic didn't trigger CPU usage in quite the same way (i.e. in small doses the checks remain in CPU caches but as traffic rises above X it starts to cause high latency with memory reads and the CPU is least waiting) or resulted in additional CPU to fully process the rule with some production traffic (repeated calls to a script possibly?) that wasn't fully considered.
Seems to me like what they need is a limit or timeout on how long regex's can take - alerting and terminating if they go over.
You don't need fancy tech. What you need is not to swamp 100% CPU on huge multicore devices when you could have just said "Has this regex taken more than 10ms to execute? Then kill it and tell the admin so we don't fall over globally".
The problem here, is that if the purpose of that regex is to trap malicious code, then malicious code authors would exploit this to slip their payload past the regex, by deliberately putting something that would cause it to time out in the script. Ironically, in attempting to increase security, by closing off part of the attack surface, this would decrease it, by opening another (potentially bigger) hole.
Whereas at the moment, the attackers can only DoS their entire infrastructure with bad source data on a poorly-written regex. So much better!
You write the regex so that it's written properly. So that it doesn't matter what data it's given, it can resolve it within a set time. If it can't do that, then you can' t use it anyway as it will introduce *so much* latency into the system that it turns into a DoS and becomes useless.
You're confusing "source data" (hacker controlled) with "regex expression" (Cloudflare controlled). If the regex can't deal with the source data in time, it should alert. There's a clue in that word... alert.
If it alerts on every damn page you go on that has a bit of Javascript, it's useless anyway but at least it didn't bring half the globe down with it.
And then realise that maybe, just maybe, regex hunting is no better or different to AV signatures - which also exhibit this same problem.
If a malicious attacker can control the data in the page to the point that they can make your regexs timeout, then they can do a lot worse anyway. Hell, "give up" and return an error in that instance. You'll still have *much less* impact than taking down your entire CDN because of a multitude of over-running regexs from a handful of sites. You'll just have a handful of sites that don't work, rather than an entire international company service.
the point I was trying to make here is that if you have a bit of code that "scans" something for dodgy scripts, using a regex, then if an attacker knows that this will time out and "pass" the script then you are opening yourself up to an attack where the attacker crafts a bit of script that will time-out the search and in doing so, stops the whole scanning process, allowing other "nasties" to get through. This presupposes that teh same bit of code that times out is also responsible for doing that, but if we're talking about doing stuff with regexes, it's a possiblity that this is how it is working, or alternatively, a timeout is set ont eh total time to check something, in which case the regex times out and prevents anythnig else after it from running, allowing things to slip through.
The better solution is not to use a regex at all, or use one that is so trivial that it is not possible to feed it input it will choke on. If you must use something like this, you should never allow something to pass through after timing out. It should be an instant fail. The flip side of that is, that if you get it wrong, and make it inadvertantly CPU-expensive under certain circumstances (as here), then even if you do have a timeout, you either have a trivial DoS channel, by flooding with things that will time-out, or you have a timeout so short that legitimate things get flagged up, and you get flooded with false negatives.
And CI/CD, which seems to be yet another pup sold as a way to get rid of people. "I know, let's automate everything, then we can get rid of all those checks and balances that keep our systems up! Those checks and balances cost money you know!".
Who needs 5 nines anyway, 4.5 nines is great, 3 nines is bigly, sure our parents were delighted with 2 nines...
One trick company tools and processes are not appropriate for complex large businesses.
Honest, yes. But there are a lot of questions about the rollout scripts that matter.
But it sounds like this was an edge case that was missed. Might have only happened one query in 1000000. Still would have taken 100% of their servers out in seconds.
If the regex was actually generated, then you get a whole 'nother set of difficulties in catching these.
When it comes to content delivery networks -- what Cloudflare does -- there aren't really other good options. Unless you plan to scatter your own servers all over the globe, and make sure you buy enough of them for your absolute peak use case (which means most of the time a lot of them will be idle.)
While Cloudflare was refreshingly open about the technical cause for the outage, they have not touched on the process issues.
There's always a possibility that a change, however thoroughly lab tested, has unintended consequences in a large-scale production environment.
That's what good old phased rollouts are for. Why have they make an immediate, global change? That approach should have raised a very big, crimson red flag.
I can appreciate that for a business such as Cloudflare "we may be under attack" is the normal first-out-of-the-box reaction to a sudden problem. However when a change has just been rolled out the possibility that the change may have been responsible should take precedence. Rolling back the change, assuming that can be done quickly, should be a fairly obvious - and prompt - response. At the very least it triages the most likely cause over which you have control and in the worst case leaves you no worse of than before you rolled out the change.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
