OpenWrt forums lost as hardware failure again crocks open Wi-Fi router

Open source Wi-Fi firmware project OpenWrt says a hardware fault has taken down its forums, which appear not to be recoverable. The forums disappeared with a simple, and distressing, message: The OpenWrt forum is currently offline due to a hardware problem on the hosting machine. Unfortunately we neither have access to the …

[Dan 55]

That's bad

A lot of the "documentation" was in the forums and it never made it over to the wiki.

Volunteer projects can't afford to have one person in charge of something with the rest unable to get to it if it goes wrong, only to find later on they weren't doing their job (backups) or they got run over by a bus.

[Roland6]

Re: That's bad

>Volunteer projects can't afford to have one person in charge of something with the rest unable to get to it if it goes wrong

Trouble is that is probably the case for the majority of projects and business IT 'departments'; business continuity is more than fully redundant hardware...

[Anonymous Coward]

Re: That's bad

But how do they not have a backup at least consisting of having a crawler go over the forums every month or two and some project member sticks them on old hard drive is in closet?

Come to think of it, surely Google and/or Microsoft has a copy...

[Anonymous Coward]

"send them an automated invite"

Better get a move on - GDPR kicks in in 10 days...

Remind me - anyone else getting "click here to continue receiving updates from us" messages from people you've never had any contact with? Suspect it's the latest ploy to "install" malware.

[John Brown (no body)]

Re: "send them an automated invite"

"Remind me - anyone else getting "click here to continue receiving updates from us" messages from people you've never had any contact with? Suspect it's the latest ploy to "install" malware."

I've not seen any myself, but I have got some from companies/forums etc I've not dealt with in years and have un-subscribed from

[TrumpSlurp the Troll]

At least

As far as we know it wasn't a router software failure.

[tin 2]

backups weren't under their control? WTF? even if I was in charge of such crappy infrastructure (and didn't have the motivation to change it) I'd take some kind of backup - locally - quarterly or something.

To have nothing at all seems... odd.

[Velv]

A perfect example for Business on why you need to make sure of the background, resilience, security and support for any software you deploy.

Open Source can be cheap to install and utilise (TCO is rarely zero), but what happens when the “Vendor” fucks up? “Oops, my bad, we’re broken and all our support material is gone. Might get it back, who knows...”

Open Source can be very good, just make sure you’ve done your due diligence before recommending it to the company.

[P. Lee]

Nothing to do with open source

A full ms share point/ exchange stack would make no difference.

This this "small volunteer organisation" behaviour. I would have hoped, however, that IT people would know better than to trust IT.

[DropBear]

What kind of a joke of an outfit are these clowns running where major pieces of infrastructure (like, oh, the entire community communication backbone) are only accessible for maintenance by one. single. person...?!? You don't need a full complement of gaffer boys and personal assistants and catering crew in an open source project, but there's no excuse for not having at least one or two other people trusted with admin access (including the data backups), who only ever need to do anything at all (as admins) in exactly these kinds of situations when the admin-in-charge is MIA at the worst possible moment.

[Crapectomy]

In the open source world many participants defend their little area of expertise, their little kingdom, their server like it was their own turn rather fighting for the community. So if there's only a single person with access that's probably intentional, at OpenWRT and other projects.

[handleoclast]

Seems like a project killer to me

If their grasp of basic principles is so flawed that they allowed this to happen, then what chance their software is any better? Worse, this is the second time it happened, so they don't even learn from past mistakes.

This is the sort of thing that makes me look at the bargepole with a condom on and think "Nah, not long enough." Yeah, the project itself might be far better, but I see no reason to take the risk.

[Anonymous Coward]

Re: Seems like a project killer to me

"If their grasp of basic principles is so flawed that they allowed this to happen, then what chance their software is any better? Worse, this is the second time it happened, so they don't even learn from past mistakes."

True that, I remember now ! I think also this is another symptom of the fork. If you remember, the Lede fork was justified by the aging tools, and possibly also their "under my desk at home" nature.

[Loud Speaker]

Re: Seems like a project killer to me

Not used OpenWRT for a while, but I put LEDE on my BT provided hub, and now it doesn't crash every few minutes, my downloads are 20 times faster, and my uploads are three times faster. On top of that, the UI is actually documented (although you would have to be fairly technically knowledgeable to configure it).

BT sent me a replacement hub when I complained about the crashing - it is exactly the same, and my son's at another location is similar (crashes only about once an hour instead of several times).

BT do not have any meaningful documentation on their "home hubs" and that on the Business Hubs appears to be a limited script on how to do what they think you want to do.

LEDE is not for everyone, but for some, it is wonderful. Or, maybe, BT Hub software is complete shite, but, IME, the other vendors are not much better.

The OSS would be massively better if the hardware manufacturers were to allow people to read the documentation on their products. I think they should be forced* to release all docs on any product that is EOL's to minimise landfill.

* This might need to be at gunpoint in the case of Oracle and Cisco.

[Anonymous Coward]

From the partially restored openwrt forum:

"At the moment we managed to recover about 38% of all posts in about 20% of all topics, with a strong tendency towards newer content. We are aware that this is not a perfect substitute for the complete forum content, not by a longshot. But it may help until we can get access to the original forum content."

GASP

It seems indeed backups were not working at all. Reminds me of this fella who once phoned me in angst, one sunday.

HP Tech had totally wiped the autoraid (yes, they were used to do that, and actually, they did it twice for this particular customer, evidently to make it perfectly clear they didn't have a clue how to update the firmware).

Tuned out the backup script was always returning 0, without checking the result of the tar command. Turned out as well one tape was not enough to contain the whole email system, and this, for a *very* long time :)

A nice technical conversation after on the likelihood to restore the email DB with a truncated backup, and then, admission of the truth: 1200 factory user maiboxes lost.

[ldir]

Progress

Some of this is why the LEDE fork took place in the first place. The re-merge has similarly re-encountered the original intransigence. My personal perception is that progress was being held back by one developer who held the keys to the original forum & original email list and despite repeated requests never quite got around to handing things over. The problem has become self-limiting though as the machine in question, running the original forums & email list & domain services has gone AWOL/died/burst into flames, thus DNS was switched to services running under the auspices of SPI. The LEDE based forums, lists, bug trackers etc have not suffered any outage and many devs have keys to these boxes in the event of issues. The fact that the email list situation has dragged on for so long is ridiculous and in my opinion showed where the heart was on the part of at least one ‘key’ person in openwrt land, they’d rather confusion reign instead of actually merge. Fortunately this attitude appears to be a one off as agreement and cooperation with others is thriving: In the meantime, today sees branches for a major release 18.06 being made, something that would have happened a week or so ago had the machine issue not occurred.

[J27]

I wouldn't trust these idiots with my router. Failing to keep up to date backups is something that's totally unacceptable in any software project.

[Loud Speaker]

I wouldn't trust these idiots with my router.

Unfortunately, most of the closed source alternatives are seriously insecure and barely functional.

[elvisimprsntr]

Given how inexpensive NAS servers are these days, there does not seem to be any excuse for this.

1. Two separate RAID servers with geographical separation

2. Real time RSYNC between the two

3. A domain/DNS provider which will automatically fail over to the backup server

4. Grandfather-father-son backup strategy, with periodic checks the backups can be restored.

5. Multiple sys admins