Backup failed, but the boss didn't slam IT – because his son was to blame The silly season is close, but not so close that The Register can't find another instalment of On Call, the Friday feature in which Reg readers share tales of tech support triumphs achieved under uncomfortable circumstances. This week, meet a reader we'll Regomize as "Ray" who told us about his time working as the network …
My veterinarian's child was responsible for taking down the local Mom and Pop ISP for a weekend back in ~2000.
The kid pulled a prank on one of his friends, convincing him that he'd gotten his hands on 'Chrono Cross' for the Dreamcast with a trick involving a VCR and some sleight of hand. He then gave his friend a FTP link to a ~600MB CD of /dev/random garbage, expecting him to spend a two days tying up the phone while downloading it and a week trying to get it to work and wasting many, many CD-R discs in the process.
The friend, however, went straight to IRC. "Someone got a hold of some Bleem! code! <link> == Chrono Cross, fully running, saw it in action myself!"
That was at 5pm on Friday night. In under an hour the FTP server fell over and took the network gateway, running on the same machine, with it.
Remember how I said they were a Mom and Pop? Well, Mom and Pop were on vacation until Monday.
Happens quite often in smaller companies: Another internet connection is ordered, son sets up the router, connected to company LAN. Of course a home-router, which assumes it is DHCP ipv4/ipv6 master, has no rival anywhere, and any rival must be by design ignored and bombarded with "I am the only one here!" logic.
Result should be known to Reg On-Call / Who-Me readers. Including who has to sort out what is going on.
is that what happend ? its not quite how I interpreted the story but thats by the by.
I've certainly come across what you describe, I've found a router someone ahs brought in from home presumably to use an adapter to turn on cat5 socket into many. It was busy trying to allocate IP addresses
When I worked in the UK we had someone do this so they "could get Wifi for their phone"; it took out parts of a research building for an entire day.
An interesting side effect of this was so we could see what what a balls up the outsourced networking provider had made of the separation between office and lab subnets...
Our IT dept ended up writing some scripts to sniff out rogue DHCP servers - we use them when running dev software on our kit (although it should be running through a separate NIC on the PC)
Some of the engineers are less....disciplined in setting then up, shall we say, especially when setting up the correct interface on their laptop. Helping track down the PC responsible was fun, I think the IT manager still owes me a few of these ->
Indeed they are easy to find. But the company size I talk about does not have an internal IT, at least not on that skill level. So they called us (i.e. my former employer to be exact) to sort out the "network is broken" problem. And some customer types blamed us in advance of course...
My worst backup incident occurred when I was on holiday, and was thanks to the boss! I may have recounted this before on here.
Back in the days before cloud backups, when off-siting meant taking a set of tapes home with you, I was managing a 486 SCO Unix HP box with a DAT drive, running a number of terminals. Everything was fine when I went away for a week but, just in case, I took the "mobile" phone with me, this was a Motorola car phone attached to a large lead acid battery pack (see https://www.tvfilmprops.co.uk/det/4016/Motorola-4800x-Partner-Retro-Mobile-Phone/ for a picture). I'd left clear instructions on swapping the DAT tape each day but in the middle of the week I got a call, "The backup tape isn't ejecting, what should we do?" My clear instructions were to leave things as they were, don't do anything and I'd sort it when I got back a few days later.
The general manager wasn't happy with this so, without telling me, he called in an engineer from the parent company's support people who was only familiar with DOS PCs and clearly had never encountered Unix before. Apparently without even looking at anything he just powered down the box and proceeded to fix the tape drive, which he did successfully, before powering back on again. At that point he got a load of error messages and not much else. With no idea what to do he just left, basically saying it wasn't his problem as he only did hardware!
I returned the following week to find a server with a trashed file system and the most recent working backup was a week old. It took a few days to rebuild everything, restore the last available data and re-enter nearly a week's work. Did I get any thanks or an apology? Of course I didn't! A while later they relocated the operation to the parent company's offices and I turned down all offers to go there.
As every school IT manager should know, SIMS defaults to storing the backup a second partition of the system drive, so the first job is to change this to a seperate server, preferably an off-site one.
I was once called into a school that had a fire in the comms room, it took out the SIMS server, and naturally they'd not changed the back-up location, so they lost everything.
I, along with several other SIMS admins from the area, spent a few weeks rebuilding the student data from paper files, then rebuilding the timetable, the class lists, the assessment and attendance data, everything we could get from paper records.
Naturally, that school got an Inadequate grade when Ofsted visited a few weeks later, and they were handed over to an academy group, last I heard they were down to half capacity and in danger of closing...
If you had the slightest clue how much admininstration goes on behind the scenes at a school, you'd wonder how we got any teaching done at all.
I spend half my life ensuring the data is correct for the termly school census, the meagre funding we get is entirely dependent on getting those censuses right.
In the good old days, the local authority had a department that ran all the schools in the district, since the Tories decided to force schools to become academies, each school now has their own data manager, IT support, HR, finance department, burser, timetabler, exams officer. Collectively costing the DfE far more than it did for the town to run those schools.
Then schools merged into Academy Trusts, they have the same staffing, but now have a shit load of cash creamed off to run the trust.
As for your point about Ofsted grading; by law we have to provide an accurate student roll, attendance data and assessment data. If you can't keep that data safe, then Ofsted are not going to look kindly on anything else the school does.
Once upon a time the local authority managed such stuff for the schools. Because they managed the schools Not a perfect system, no system ever is. But there were enormous economies of scale, shared services like school libraries, SEN support teams or resource centres and a significant ironing our of anomalies and idiosyncrasies like the head teacher/governors deciding to spend the Special Needs budget on a new cricket field. But in the spirit of "Local management of Schools", competition ( because the market is always right, yeah) and Tory determination to control labour run councils all that was flushed away. Nicely aided by the BBC running headlines about "greedy" LEAs withholding funding to pay for some of those service. Yes, they actually said "greedy" as if the councillors were taking the money to pay for their Christmas parties. But of course that remaining few percent was paying for the School Library Service, a little museum of local artifacts for the primary schools to use, an IT support and an SEN team- all far more cheaply than if the schools bought that stuff in individually from a commercial contractor.
I don't understand how my local council can say that giving up the central catering body and transferring responsibility to each school can save money. Even if the plan is to outsource, surely the Council could negotiate a better outsourcing contract than all the schools in the County could individually.
It saves the council some cash when the money used to pay the staff negotiating, monitoring, charging for and paying the centrally managed catering contract isn't delegated to the schools. i.e. the schools have to carry the cost of the admin and are only funded for the cost of the catering so that. it costs school money to do all those things.
Sorry, when I went to school administration was a paper register, entrusted to the kid least likely to get lost, and taken to the school's central administrative function = a middle aged lady in a cardigan with tissues up the sleeves and an electric typewriter.
Of course we didn't achieve 11 A* in key stage assessment level alpha
This post has been deleted by its author
This post has been deleted by its author
This post has been deleted by its author
I was involved in the remote diagnostic of a bank, and it's ATM performance problems. During the week everything was fine.
Every Saturday the performance died at about 10am. What could be the problem?
I did some traces in my product, and I could see that the network time was about 10 ms. At the weekend, it went up to 100ms, and every thing queued up.
This was very useful information
He came in on Saturday morning, and walked around this site, and met someone in "operations". They discussed the problem and the operator said
"Ahh we now do the cross site backups on Saturday morning". They stopped the backups - and the performance was good - restart backups -it was bad.
They spoke to their networking people who set the transactional traffic to be high performance.
I still have nightmares about backups that won't backup and/or restore, 36 years after working for a DCS vendor whose proprietary system ran on MTOS. When the (frequent) need to upgrade a client site arose, we often found the pre-upgrade backup failing, or the pre-upgrade restore test failing. We never did find out why but I've been anti-tape ever since.
It matured. There is, practically, only one tape backup technology left for more than 10 years. All others died out, and for 80% of them for the good reason to be unreliable. The same happens with tape backup software. Famous example: The amount of weirdness and bug accumulated in backupexec over > 30 years is astonishing, I don't know why that software still exists which even the numerous vendors, this software went through, cannot fix or work around.
I prefer backup to disk, and tape as secondary (daily or weekly). That setup was more expensive in the past, is cheaper now than other solutions.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
