Run a demo on live data? Sure! What could possibly go wrong? Hang on. Are you sure that's not working? Welcome, dear reader, to another installation of Who, Me? in which we recount for your schadenfreude indulgence tales of Regizens who create havoc – either by their own design or others – but mostly emerge unscathed. This week's hero, whom we shall Regomize as "Sedgwick," worked as a systems engineer in the R&D department of a …
Apologies for the lack of detail, I wasn't present in the room but did see the effects. One Thursday, there was a lot of grumbling at the end of the day when an unfortunate IT engineer took down the main data centre by accidentally hitting the big red button, causing people to lose unsaved work. The following Monday morning, the same thing happened again, with similar grumbles.
I later learned that the second incident happened *because* the engineer in question had fitted the big red button with a cover. They were proudly showing it off to their boss by sharply hitting said cover, which hadn't been properly fitted and gave way just enough to trigger the button again.
Accidents are one thing, but there are some people who do this sort of thing deliberately, as told in this tale of our own Father Dougal
Don't knock the big red button..
In the Uni computer lab I used to work in, there was a big red button on the wall that killed the power to the student computers. I *think* it was there from when the room hosted several large, and potentially dangerous machines, it's just when the electricians installed the new electrical mains, they ran them through the same BRB.
Now, when closing the lab, I always gave warnings half an hour before closing, then ten minutes, 5 and finally 2, so I'd argue they did get fair warning.
95% of students would leave after the first or second warning. TBH, by the time you got to the third, you were down to the hardcore students who wouldn't leave no matter how many warnings you gave.
After a few weeks, I changed the final warning to "If you don't pack up and leave within 2 minutes, I'll go to the big red button, hit it, and you'll lose any work that isn't saved". Weirdly, no one ever tested me on that..
When doing my electronic engineering degree several centuries ago (that's how it feels from this distance, anyway) it became apparent that a lot of my fellow students were affected by dyslexia to some extent. I felt they'd deliberately chosen the course with the fewest essays, as it was maths-based. I've met many intelligent people who couldn't write a comprehensible sentence but could easily speak it.
I think the quality of English here is excellent!
I worked as a remote support tech that supplied biometric till systems for schools. For weeks we had the end of day process go offline at the same time on Friday so the school in question couldn't close there trading week and devs started to panic,(what fun)
Turned out that the woodworking shops closed at that exact time and turned off all the electric with a big red button, which some genius had wired the till power supply
I remember working at a customer site, and by law they had to have some security software installed. The manufacturer was really pleased to quote their defect rate ... single figure defects per year, installed on thousands of systems.
During my visit an audit was taking place. I heard the conversation went a bit like this.
Auditor:Do you have this security software installed
Customer: Yes.
Auditor: Is it enabled
Customer:Not as such
Auditor:Please may I see it working, and that it stops someone without authorisation from doing something.
Customer: Not during production hours
They turned on the security system on a test system, and the performance dropped significantly, there were abends and other errors, it did not scale to hundreds of users, it just about did what it said on the box (after a lot of complex setup).
A short while later the Government updated the law to say the software had to be installed, enabled, and working.
The result was so many defects and problems were found the manufacturer had to withdraw it as it needed a redesign.
This post has been deleted by its author
Nothing so drastic, but I did have to travel to a customer site to set up a room for on-site training. It was a last minute request (only about a day's notice). I was informed that everything was ready. It had to be done on a Friday, and this was on the Isle-Of-Wight. The company shut up shop early in the afternoon.
So I get there and start my work - only to find the requirements weren't met. They were missing software. Queue people rushing around trying to
a) Blame someone else
b) Find someone who could authorize the software purchase so late in their working "day"
Luckily I am covered (I simply produced the statement from them that all was ready and the requirements were met), and they managed to find someone willing to cough up for the software. Install done just in time, and I got a trip down memory lane (had to ride on the chain ferry - haven't done that in decades).
I realise this isn't always feasible, but this is why I normally advise people, if they are doing development work on a system, to have a test install of the system running somewhere, probably with slightly out of date data.
A system I'm working on at the moment runs on two VMs. One for the production system, and one for the test system. The test system uses a copy of the database from the main system that is about 24 hours behind. We do all our development, testing and demonstrations on the test system, safe in the knowledge that if the company hosting the systems is doing their job (and generally, they do) properly, our data is safe.
That said, another system I'm working on, we didn't pay for a test/development server, so I've learned to be *very* careful on it..
Agreed - but would go further...
Make sure it's easy to distinguish between the systems (e.g. change the backdrop image and/or colour schemes, fonts ... for the test system).
I've seen an ohnosecond panic when the penny drops that the change was made on the wrong system -- fortunately nothing was lost apart from the demonstrator's** dignity
Typically I used some hideous colour scheme, one that nobody (apart from Lawrence Llewellyn Bowen) would sanction for delivery to users, on test systems. Taste doesn't matter if you're making it blindingly obvious whether it's "safe to play" or not.
** Thankfully not me ! :)
Make sure it's easy to distinguish between the systems (e.g. change the backdrop image and/or colour schemes, fonts ... for the test system)..
This.
That's why my user level terminal has a blue background. If not blue, danger. I don't set a red colour for a higher privilege level because sometimes that may not work or not be configured so I rather fail safe. Also, my working prompt is different.
As for dev and test, we give those their own VMs on networks that can't even reach production. Might be a tad OTT, but we prefer several layers over guesswork.
Not always easy.
I work on a system for customers and I do change the colours and typically have the same set for the instances I am on. Often I have limited access to their prod instance too.
A recent customer had their prod instance set to my "dev" scheme and they had granted me full admin. I didn't do anything wrong, but occasionally would go to look for a record / update and wonder why it was not there and then remember "Oh I am working for acme" and this is NOT their dev instance.
forget it having words of "Prod" / "Dev" in its title, I mean who look at that when you have nice colours to identify
Excellent idea, if you can get the funding for it. It may be easier to do so if you can arrange for the Test system also to be part of the Disaster Recovery/Business Continuity plan. Of course, if Production and Test are just two VMs running on the same bare metal, then it'll be difficult to be convincing about the DR Plan.
That was, unfortunately, many years before Youtube.
But here you'll find the Investigation Report, see section 1.5.1.4.2 Unintentional retraction of the landing gear on the ground on page 24. And yes, that report was not about the unintentional retraction but about the unintentional touching the ground with sudden deceleration done about ten years later by the same pilot.
There was a crash in Pakistan a couple of years ago. The pilots we're having a big discussion about covid while coming in to land and only realised they'd forgotten to put the gear down when the engines scraped the runway.
They pulled up and went round again, but when they ran the engines up on the second approach, both failed and the plane landed on a load of shanty houses. There were only two survivors.
That's Pakistan International Airlines Flight 8303.
PPRUNE thread, currently 84 pages.
Why didn't your friend use theirs?
Moderately experienced pilots tend to be at highest risk of jumping the checklist.They are certainly no beginners anymore and develop an overconfidence: "I know what I do!"
What it was in this case, I certainly do not know but I really wanted to reply to a dog replying to a cat.
My friend was also horribly prone to cycles of massively overwork followed by a collapse. I suspect it was during the latter part of one of those cycles.
I really wanted to reply to a dog replying to a cat.
Shouldn't you have been a horse or similar sized animal rather than an auditor?
Some timeago I worked for a software development company that had a product to archive data from relational databases. It worked by running a search for candicate records to be archived and then found all the releated data. So, you could archive customers who had not ordered anything for five years, and it would also find all the invoices, payments, shipping, correspondence that related to the customer(s).
As the tech person I always insisted that the client test their scripts on a test sustem before running it on production.
One client proudly proclaimed that they didn't need a test system. Nor a development system. They just ran production. So I asked how they did development of the application and specifically new database tables (etc). The answer was "Oh, Malcolm here just gets on the main console and keys the changes directly in to the database. On production. And if it goes wrong, we are front page news next day. Ha!" We were out of there so fast as I explained to the samesman that no amount of profit was worth the risk that if anything did go wrong we were bound to be the fall guys.
Oh, yes. The system? HMRC's tax computers.
Which "tax computers"? There are many systems in HMRC, most of which had discrete Live, Clone and Dev environments while I was working there, most particularly those involved in the assessment or collection of tax, such as COP, BROCS, CESA. I wouldn't be surprised to learn that there was a system somewhere in the estate that ran on the basis you describe, but I would be surprised if it was one the main revenue systems.
(Unless it was from the former Customs and Exercise part, some of the things that were done in Southend were tantamount to piracy).
It was run by a large, internation, computer services team. I prefer not to expose them as my desire to end up on the wrong end of a defamation law suit is pretty low! This was decades ago, in the mid 1990's. And yes, if I recall, it was Customs and Excise. But it wasn't Southend we visited.
I am sure that they are a lot more professional now. This would all have been refreshed as part of Making Tax Digital. I mean, it must be so, mustn't it!
"customs and excise computer". It's all clear now. Anyone ex-Inland Revenue was very clear on the difference between taxes and customs and excises. One was calculated and collected with precise accounting, a practice dating back to the Domesday book, while the other was gathered by freebooting Johnny-and-Jilly-Come-Lately privateers with a more swashbuckling approach.
I'm guessing that it went something like:
> list processes | find "important"
important business process do not stop this
> start lowlevel process "kick me"
> sudo start highlevel process "importent process - for demonstration"
> list processes | " find "kick me" | kill
Killed "kick me"
> list processes | find "important" | kill
Killed "important business process do not stop this"
Edit... or maybe that was the "bug", an auto spell check??
> list processes | find "impotent" | kill
Did you mean: important
Killed "important business process do not stop this"
The typical "This cannot work because I don't have the rights" - applied to Active Directory, Fileservices, Webservices, Volumes, you name it.
Luckily the AD was new enough to have a recycle bin, and it was activated. The fileserver "delete test" was stopped fast enough to use robocopy to get the missing data out of the snapshot. The volume only contained replica-VMs which are on standby when the main server room would be burning or otherwise dead. The Webservice kill is not my experience, but I've seen others doing that. The older you are, the longer such lists.
A worthy article for Who-Me.
This post has been deleted by its author
is my friend... it keeps me warm at night with the high blood pressure, the tremors and the cold sweats.
Especially when the ohnosecond is not the time to think "Oh my whats gone wrong?" but more like "&%^*$^ 7Kg of tool spinning at 6000 RPM is about to exit the machine and play merry hell with my running legs...."
Deleting data sounds so much safer... if just as expensive in the end
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
