Happy Friday: Busted Barracuda update borks corporate firewalls

A firmware update pushed to Barracuda firewalls has knocked out boxes in large firms and crippled networks, we're told. The change was sent out to customer gear by Barracuda on Friday morning Pacific Time, according to a source familiar with the matter. The update is automatically installed, and promptly caused the devices to …

[Nate Amsden]

not understanding

Is this literally a firmware update that was pushed out(as in OS upgrade or something) ? (didn't know any vendor did that for any kind of product) Or was it just a config update/change ?

[diodesign]

Re: not understanding

Yes, this was new data pushed out to Barracuda devices from Barracuda.

C.

[theblackhand]

Re: not understanding

It's likely to be a data file updaye akin to the GeoIP databases that you can use to classify traffic.

It shouldn't be too hard to QA the data....

[Anonymous Coward]

Why even use hardware firewalls in the first place?

It's something I never really understood. I can see ease of use and how you might be able to quickly (time = money afterall) set up one set of rules and propagate them. But you're still left with a box which you don't fully control. Call me paranoid, but I still recall those stories about the NSA gaining access to hardware routers because of known exploits and such.

And it's not as if a software firewall can't do the same thing. In fact, I'd even argue that it'll be a lot more flexible also allowing you much more customization.

Personally I'd take an OpenBSD run firewall over "hardware" any day of the week.

[Nate Amsden]

Re: Why even use hardware firewalls in the first place?

openbsd firewalls are nice (I have been using openbsd with pf since about 2004, and freebsd with ipfw before that I think) but hardware appliances typically are capable of a lot more, mainly in layer 7. If all you need is basic layer 3/4, then openbsd can be fine depending on support requirements(installing it is still a pain for me but i don't do it very often). If you want deep packet inspection with rules to be able to handle that, the commercial boxes tend to have those features in a more user friendly form.

Looks like OpenBSD is still limited to 1 CPU for PF https://www.openbsd.org/faq/pf/perf.html which is too bad I would of thought that had been addressed by now. With such powerful multi core systems on the market it would be pretty cool to see. My first "big" openbsd firewalls were in 2005, a pair of dual socket single core I think they were, with pfsync running between them with about 8x1Gbps interfaces. Though actual throughput was limited to around 500Mbit (I think because of interrupt overhead?? CPU never got closed to being pegged)

500Mbit is of course plenty for most internet connections. That particular use case was a bridging openbsd firewall between internal gig network segments.

[Pirate Dave]

Re: Why even use hardware firewalls in the first place?

"500Mbit is of course plenty for most internet connections."

Maybe in 2005, but not anymore. Bandwidth is cheap now - a full Gig (over fiber) runs less than $5k/month even way out in the boondocks where I am, and is much, much cheaper in the big cities. So a 500 Mbit firewall won't cut it.

Besides, most of the "hardware" firewalls are just vendor-labelled white-box 1U Intel servers from China anyway. Most that I've seen don't have anything particularly special or esoteric in them as far as hardware (although I think my sonicwall NSA boxes supposedly have MIPS processors in them). Maybe Cisco, Foundry, F5, and the other "big iron" switch and router guys put some special-sauce hardware in their firewalls, but things like Barracudas, Sophos/Astaro, and Sidewinders are all just Intel boxes running somebody's hacked-up Linux or BSD.

[Nate Amsden]

Re: Why even use hardware firewalls in the first place?

yeah like in most things these days much of the value is in the software. And for sure at least many Sonicwalls do not run x86 CPUs(I want to say none do but that may not be accurate).

With something like a sonicwall being able to scale to 96 CPU cores(high end), while OpenBSD is stuck at 1 is obviously not a good sign of progress on the software front.

I don't know if Linux is any better, I use linux on 99% of my systems, though my (personal) firewalls run OpenBSD, my work firewalls have been sonicwall for the past 5 years(no complaints).

While last I checked F5 used Linux underneath (and Citrix uses BSD), both run pretty custom networking stacks to get high performance. F5 was limited to a single CPU up until about 2008 or 2009 I think it was, they had SMP boxes before that but the network traffic couldn't scale beyond 1 CPU(2nd CPU could be used for 3DNS or something)

[razorfishsl]

Re: Why even use hardware firewalls in the first place?

Hardware inspection.

a decent firewall has to inspect & move a data packet rather than just "route" it.

[scott.ramsdell]

Re: Why even use hardware firewalls in the first place?

640K is more memory than anyone will ever need, right?

[Nate Amsden]

Re: Why even use hardware firewalls in the first place?

Do you recall the story of the FBI putting a backdoor in the BSD IPSec stack ? Someone recently told me about that. I'm sure I heard about it at the time but forgotten until recently.

http://www.theregister.co.uk/2010/12/15/openbsd_backdoor_claim/

To me it's impossible to tell for any given vulnerability if it was deliberate or not, would be difficult to prove either way.

[Lee D]

Re: Why even use hardware firewalls in the first place?

The setup I use is thus:

- Leased lines through to vendor routers, modems, etc. (which we usually can't remove or change).

- Ethernet cable from each to managed switches.

- Managed switches force all the external connections onto specific VLAN's (ignoring any existing incoming tags, obviously)

- The only other device on such VLAN's is a virtual machine running Smoothwall (or, in a pinch, any VM capable of network routing), where each VLAN is presented as a different network interface.

- That VM also has the "normal" network VLANs which it routes as appropriate.

If something like this goes wrong, you roll the VM back to a previous snapshot.

If you can't rollback, you restore the VM from backup.

If you're in a deep mess, you boot up anything like an Ubuntu disk in a VM and just NAT the right VLAN interfaces.

If you're suddenly pushed off-site for whatever reason, you just add a VLAN to the same VM or change one to reflect whatever external connection you can get.

If you're really in a pinch, put the VM back onto physical hardware and plug in one network card and cable per interface.

Unlike all the hardware firewalls I've had, this allows as much expansion as you like, serious amounts of processing (for VPN, web inspection, SSL decryption, reverse proxy, etc.), bandwidth, failover, logging, RAM and interfaces. And it's all contained in one place, configured in one place, logged in one place, and that one place fails over. A central pinch-point for management, filtering, QoS modification and control without reliance on any one set of hardware.

And because the VLANs can be tapped into network-wide, if some line goes down, or there's a network split for whatever reason, it all stays up in the working remainder.

Despite a load of lines coming in all over the site, and bunches of VLANs all doing different things (e.g. a telephony VLAN also running SIP over the net, etc.), it's easy to understand and manage.

And engineers who come visit can just unplug their Ethernet cable and test what they need to on their equipment direct without messing anything up (and if they plug it back into the wrong place, the switch configurations will stop it opening up the connection to the whole network).

But a firewall having to be hardware is quite an archaic concept. And you can quickly outgrow anything your budget runs to, especially if you're offering outside services, VPN etc.

But then, I was deeply involved in the Freesco project from many years ago - which was a single bootable Linux floppy that did everything a "Cisco router" could do for you, so I haven't relied on a hardware firewall / router, even back in the dial-up modem days.

[Fred Flintstone]

Re: Why even use hardware firewalls in the first place?

The setup I use is thus:

Nice to see an architecture where someone has put in some effort.

[Sergiu Panaite]

Re: Why even use hardware firewalls in the first place?

Quite so - unless you're looking at highly latency sensitive high rates of traffic, microbursts, want to avoid jitter in that scenario, etc. But yes, in all honesty that's not the kind of scenario that most people look at, so non-hardware firewalls are far better suited than most people give them credit. The real issue is that a lot of admins want something they can easily set up and requires very little maintenance, hence the success of brand-name software firewalls where you effectively buy peace of mind and the privilege of "not caring"; then again, they go ahead and do this kind of thing...

[Version 1.0]

Re: Why even use hardware firewalls in the first place?

It's the easiest way to keep the management and auditors happy. Show them a software firewall and they get all panicky, "Someone has to audit the software", show them a hardware box and it's sorted.

[Anonymous Coward]

Re: Why even use hardware firewalls in the first place?

It's the easiest way to keep the management and auditors happy. Show them a software firewall and they get all panicky, "Someone has to audit the software", show them a hardware box and it's sorted.

Yes, that's the disadvantage of auditing as a profession: it has mutated into yet another tick box process where skills are supplanted by a certificate. I have audited some places that could have a direct effect on some pretty critical trading commodities (and in some WERE the exchange) and if I had been a tick box follower I'd given them a clean bill of health. However, my brief was not the tick box, but to make them safe, and they were not. Now they are, because they understood the difference and wanted the security more than just the tick box.

When I audit I insist on helping with fixing the issues - that ensures the place is safe when I walk out. Not interested much in the other type of audit, that's too simple (which means I'm not working in banks much anymore :) ).

[Anonymous Coward]

Re: Why even use hardware firewalls in the first place?

A hardware firewall would be harder to hack and would have higher throughput than a software emulator of one.

The Barracuda firewall provides a number of functions such as 'Stateful packet inspection' and 'Full user-identity awareness'. Of course in order to provide such functionality the firewall has to break SSL and in the process dilute security on the local network.

See also: Stateful packet inspection and forwarding, Full user-identity awareness, Intrusion Detection and Prevention System (IDS/IPS), Application control and granular application enforcement, Interception and decryption of SSL/TLS encrypted applications, Antivirus and web filtering in single pass mode, SafeSearch enforcement, YouTube for Schools support, Denial of Service protection (DoS/DDoS), Spoofing and flooding protection, ARP spoofing and trashing protection, DNS reputation filtering, TCP stream reassembly, Transparent proxying (TCP), NAT (SNAT, DNAT), PAT, Dynamic rules / timer triggers, Single object-oriented rule set for routing, bridging, and routed bridging, Virtual rule test environment

[Flakk]

"Good news: no one was ever exposed from a security perspective," a spokesperson for Barracuda told us.

"In fact, with Barracuda devices knocking customers completely offline, their networks have never been more secure," he could have concluded.

[redpawn]

Feeding Frenzy

Like sharks they tend to bite shiny things and have lots of sharp teeth. Keep your fingers clear of their mouths when you work on them this weekend.

[Amos1]

PR people have no soul or conscience

"A Barracuda spokesperson has been in touch to add: "We were able to proactively identify an error in the application definition file..."

So they proactively identified an error after it borked every firewall they pushed it to? Perhaps "proactively" means "Before a customer figured out what we screwed up and told us."

Or maybe "proactively" means "Before every customer who ran VoIP or Office 365 for email could contact us because nothing worked."

[Hawkeye Pierce]

Re: PR people have no soul or conscience

Furthermore:

>> "The problem was quickly resolved and we are working with impacted customers to ensure all firewalls are updated with the correction."

No, the problem was that you pushed out an update that knocked out a number of your client's boxes. The CAUSE of the problem may have been quickly resolved, but the fact that you are working with impacted customers shows that the problem was not quickly resolved.

[Your alien overlord - fear me]

And this is you should just let the NSA look after your firewalls, they're not going to bork any data streams !!!!!

[patrickstar]

There's this incident when they broke the router for all internet traffic in Syria when trying to backdoor it...

[Anonymous Coward]

Same Barrucuda as the backup?

Someone suggested I look at Barracuda backup so I tried the online demo, was pretty laggy and I just couldn't get much from it as it seemed to hang. Next day a regional reseller phoned me offering a proper engineering demo, long story short engineer tries to show me some aspect and it hangs, tries again and it hangs, shows me some other bits then goes back to first bit hangs again. Decides to show me their local backup, and how it lists backups only to find it is not listing backups today, to which he says "oh I hope it is backing up" needless to say I haven't jumped at it. They offered a 30day trial and I asked if at the end of the trial if I declined, what level of secure wipe did they offer? "you reset it to default and send it back" (I.E. no level of secure wipe)

[Anonymous Coward]

Update

> A Barracuda spokesperson has been in touch to add: "We were able to proactively identify an error...

Huh? Where I come from "proactively" means fixing it before it becomes a huge problem.

Or are they talking about another bug that would have led to all their firewalls getting hacked if that lucky geoip glitch hadn't crashed them?

[cd / && rm -rf *]

Duh.

Everyone in the IT industry knows you don't push out major software updates on a Friday with the weekend looming.

Either this was a to fix a serious 0-day and they couldn't wait, or clueless PHBs ordered the techs to push it out regardless. Or both.

[Arctic fox]

@ cd / && rm -rf * Re: "Duh."

Indeed. Apart from anything else pushing out such updates on a Friday risks costing their customer companies even more (due to increased personal costs, weekend working (overtime payments of time off in lieu) and so forth) than the disruption would have cost them anyway. Sort of adding injury to insult (so to speak).

[Snowy]

A time when

Turning it off and on again was a bad idea, quick fixes sometimes are not quick and do not fix it!