Used to work for Midland Bank back when they were still a thing. We had a power failure in London after a JCB dug through a mains cable, PDU and generators did their thing, but we soon realised that whoever installed it hadn't hooked the aircon up to the PDU so there was no cooling available. Luckily this was noticed almost immediately so we were able to shut down all but the essential systems, and open all the doors & windows before things got too steamy and we survived for what turned out to be a 5-6 hour outage. Sadly the JCB driver wasn't so lucky.
Posts by Martyn 1
52 publicly visible posts • joined 30 Jun 2009
The time that Sales braved the white hot heat of the data centre to save the day
Beware the three-finger-salute, or 'How I Got The Keys To The Kingdom'
Back in the days when I was a VMS operator I just happened to look over the shoulder of one of the trainees as they were typing:
$delete [*...]*.*;* /before=-90
(delete everything on the entire disk older than 90 days)
I just stopped him before he hit <CR> and asked what the hell he was playing at "Eric told me to delete everything older than 90 days...", I stopped him in his tracks and went to ask Eric (the boss) about it, his response was "We're short of disk space and the users don't need anything older than a couple of months so we're gonna play tough and delete anything older than 90 days and then we'll have loads of space", I pointed out that as well as user data that disk also held the executables for the production system and he would end up deleting the entire code and the whole world would come crashing down and a heap of brown smelly stuff would hit the fan.
It was pure chance I was looking over the guy's shoulder at that moment and we were only a <CR> away from compete disaster. No doubt about it - Eric would have been on the streets if it had happened, but although he was an idiot he was a nice bloke and I didn't want to be around trying to pick up the pieces anyway.
IT protip: Never try to be too helpful lest someone puts your contact details next to unruly boxen
And be careful where you leave your own contact details.
Many years ago when I was VMS system manager at Big Bank they had a data feed to/from another organization which stopped working, there was no documentation and everyone involved in it's implementation had left, and eventually it landed on my desk, I fixed a shonky bit of code and made the mistake of commenting my changes and adding my name to the comment!
Roll forward a couple of years and I'm working for DEC, and I get a call from customer support manager saying that Big Bank had a problem with some code falling over, nobody had any idea what it was doing and so had logged a call with DEC support who found my name in the code so it was now my problem and I had to fix it for them even though I wasn't working on customer support!
First call I made a call to Big Bank I explained what the code was supporting and their response was that they'd decommed that service about a year ago so didn't need it fixing - thank you very much.
I tend to initial my comments nowadays rather than giving a full name :-)
When the satellite network has literally gone glacial, it's vital you snow your enemy
Re: Wind and rain...
I was contracting for a bank in London in the '87 storm. The only people who made it into the office were us contractors 'cos a day off costs money. All the permies took one look out of their bedroom windows and rolled over to go back to sleep, so there were no managers in. There was a power failure so we spent most of the day in "The Banker" (pub) with instructions to the nightshift ops (who had to double shift 'cos they couldn't get home) to come and get us when/if the power came back on.
Behold the perils of trying to turn the family and friends support line into a sideline

Being an OpenVMS and then Linux guy I have the get out of jail free card of "I don't do Windows" so mostly avoid such requests. A few exceptions were two neighbours who each barfed their laptops which they used to run their small business.
One was fairly trivial (about 10 mins work after googling the error), but the other came to my door looking particularly ashen 'cos it wouldn't boot up at all, I said I'd have a look and before starting asked "Have you backed up all your work stuff?" and the blank look on his face told me all I needed to know. So booted a live distro, backed up everything to CD, fixed the bug introduced by a recent windows update, and then gave a tutorial on backing up his stuff to CD. Got a couple of bottles of vino out of that one :-)
I also used to get dragged into fixing the Mother-in-law's laptop which her son had bought her, it came with a "Windows Vista Capable" sticker - and it wasn't (in fact were any?). Just logging onto the bloody thing gave me palpitations and many times I wanted to fix it with a 2lb lump hammer. Eventually she got the idea I wasn't keen, got an Android tablet, and just left it to rot in a cupboard. When she has problems with the tablet she goes to my kids 'cos she knows at least they'll be happy to see her ;-)
That time Windows got blindsided by a ball of plasma, 150 million kilometres away
Back in the days of open reel tape drives I went into the computer room one evening to take some pictures, press shutter release - click - all TE16 tape drives go offline simultaneously. The flash had triggered both the EOT and BOT detectors and the drives didn't know what to do so went offline as a default failure mode.
Not the end of the world, but had to stop taking pictures and restart all the backups :-(
Never let something so flimsy as a locked door to the computer room stand in the way of an auditor on the warpath
confront the guards and ask "why they hadn't done anything"
Don't security guards etc in South Africa tend to be armed? It could have ended so differently, and no great loss to the world!
Eons ago when I worked at DEC I was in on a Saturday and needed to get console acces to a VAX, but there were no Operations staff in at the weekend, so lifted a floor tile outside the Computer room, wriggled under the partition wall, pushed up another tile, did my thing and left the way I came - nobody ever found out :-D
Staff sacked after security sees 'suspect surfer' script of shame
Back in the '90 I worked for a big computer co, so we had internet and USENET access, though the dodgy groups weren't available from the UK, but it didn't take much to figure out that if you proxied your USENET traffic via one of the German offices (where they were more enlightened ;-) ) then you could get everything. One of our devs did that and was amassing a lot of pr0n which he was then taking home on floppies!!!. Then one morning our manager took me to one side and said a complaint had been made by a staff member over something she "thought" she had seen on a screen, he told me he was going to instruct me to scan all the dept's workstations for pr0n the next morning and he didn't expect me (nudge nudge) to find anything unsuitable for the workplace; so after a friendly word in the dev's ear everything was wiped, I blocked access from our systems to the German proxies and told him in no uncertain terms that if he tried getting around it any other way I would find out and I would feed him to the boss, and everything blew over.
Boss visited the night shift and found a car in the data centre
Essex drone snapper dealt with by police for steamy train photos
Blade Runner 2049 review: Scott's vision versus Villeneuve's skill
Bombastic boss gave insane instructions to sensible sysadmin, with client on speakerphone
Once had a system rapidly running out of space on one of the disks which stored the word processing docs for the company, it wasn't really my job so I wondered into the Ops room for a chat and to make sure somebody was dealing with it.
Response I got was "yep we'll have it sorted out in a few minutes" which surprised me a little as I expected it to be more complicated than that so I asked what they were going to do "Eric (the boss) told us to delete everything on the disk more than 6 months old[1], if anybody needs any documents older than 6 months we will get them off backup".
Thankfully I managed to intervene and stop him before he hit <CR>, I then had a brief chat with Eric where I explained that the disk also had the word processor system executables, which was installed a couple of years earlier and so would have been deleted in the purge, and all the template docs that the various departments used which had also been around for a long time., The fallout would not have been pretty when all the word processing and telex (yes it was that long ago) stopped working.
[1] VMS, IIRC: $delete [*...]*.*;*/bef=-180-00:00:00
Here's how the missile-free Royal Navy can sink enemy ships after 2018
You say I mustn’t write down my password? Let me make a note of that
I remember watching a news item a few years ago where they were interviewing Inspector Knacker of The Yard over some incident, over his shoulder in the background you could see a terminal on a desk and written above it on the whiteboard in HUGE letters was :
Username: PCPlod
Password: hefelldownthestairsyourhonour
(names changed to protect the guilty)
Anyway in the interests of reporting accuracy I tried to track down the actual incident and googled "police password written on whiteboard" and there's fscking dozens of them !!!!!!
Who was downloading smut in the office while eating ice cream?
During the war....
When I first started work there was no Internet, and nobody in the company had anything that could be called a PC, they were just dumb VT100 terminals, so the only porn was a stash of jazz mags which we kept hidden under the floor in the computer room, in the same place we used to keep the beer (really!) being right by the underfloor blowers for the AC units kept it nice and cold ;-) .
Buggy? Angry? LET IT ALL OUT says Linus Torvalds
Disk areal density: Not a constant, consistent platter
A different type of "disk crash"
One night I saw a stack of those removable packs fall off the back of a disk drive while the "washing machine" was on "fast spin" and the vibration caused them to crawl off the back. They belonged to another company who shared our DC (but didn't run a night shift), I left the packs on their ops managers desk with a note taped to them warning him not to try and load them into a drive :-)
NSW Govt spends half a million dollars on XP support
Even 'Your computer has a virus' cold-call gits are migrating off XP

At least they're not scamming someone if they're wasting their time on you....
Had one call me once while I was watching a film on the telly. Told him my PC was upstairs and I was on a corded landline, so every time he asked me to do something I had to "go upstairs" and then come back and report the status (took 5-10 minutes each time). By the time I had "mystyped" t the URL 4 or 5 times he started reading it out phoenetically, I then had to report that typing "hoteltangotangopappacolonslashslashwhiskeywhiskeywhiskeydot....." into the browser didn't work.
By the end of the call the only time I'd got off the sofa was when I went to make a cup of tea, and had had him on the phone for 45 minutes - I could have kept him going longer but the film finished and I had other stuff to do, and before I hung up I made sure he realised he was a theiveing scammer !
Naked Aussie gets wedged in washing machine
On the matter of shooting down Amazon delivery drones with shotguns
Windows NT grandaddy OpenVMS taken out back, single gunshot heard
Another old lag here, from VMS 3.2 up to today, still the most reliable boxes in any of the organisations I've worked for.
What other OS would allow you to exit a script with:
$ exit %xB70
%SYSTEM-W-FISH, my hovercraft is full of eels
$ exit %xB72
%SYSTEM-E-FISH, my hovercraft is full of eels
$ exit %xB74
%SYSTEM-F-FISH, my hovercraft is full of eels
$ exit %xB76
%SYSTEM-?-FISH, my hovercraft is full of eels
$ exit %xB78
Very sad.
US Navy blasts drones with ship-mounted LASER CANNON
This week's BBC MELTDOWN: Savile puppet haunts kids' TV
Google denies smacking Botswanan ass
Patrick Moore: Lived with cats, accompanied Einstein on the piano

Farewell to the one time finance minister of the Monster Raving Loony party :-(
Seemed like a great bloke to me, and still brings a smile to my face thinking about some of his eccentricities, and the first Astronomy book I got from the library as a kid was one of his.
If he did hold some personal views which some people disagree with at least he kept them to himself and just did what he was good at. Unlike so many so called celebrities these days who think their opinions are so important that they have to be foisted on the rest of us.
Stick him into a photon torpedo tube and fire him off into the Milky Way !
Post-pub nosh deathmatch: Bauernfrühstück v bacon sarnie
IT angle time....
I once Emailed in to the Alex Leicester radio show for a comp and mentioned I worked at HP, he replied back t
AL: I used to work near there can you still smell the sauce from a mile away?
Me: It's HP the computer company not the Sauce company!
Anyway there was too much bacon in that pan, it lowers the temperature and doesn't fry properly.
And another vote for dipping the bread in the fat, rather than Butter/Marge/I can't belive it's not butter/Bugger me it's Marge
Sysadmins: Your best tale of woe wins a PRIZE
Remember Thin wire Ethernet?
Years ago in one of the I was working at offices every morning a whole section of the office would have a network failure, causing some of the clustered boxes to reboot and untold woe and a gnashing of teeth. Investigations on the network continued for several days but still each morning the network would go down at about the same time of day. Eventually they tracked the problem down to a segment of thinwire Ethernet (coax cable with a BNC connector on the end), they then discovered that someone in that area had recently got a new monitor, and when connecting it up had decided that every socket on the back should have SOMETHING plugged in so he'd plugged a thin wire Ethernet segment into the Video out connector on his monitor, every time he turned the monitor on it put s spike on the network and brought the segment down.
'Call Corporate Of Apple and tell them to stop there persuit!!'
North Korean GPS blocking sparks cyber war fears
Virgin Media cuts Pirate Bay access for millions of punters
Rogue IT employees - give us the down and dirty

Dumb extortionist.
One place I worked as looked after by a single BOFH, he got the bright idea that there was a better way to get rich than working for a living. For a whole month he made sure the backups didn't run, although he signed them off as successful.
Then one Saturday he came into work:-
Took a full backup to several tapes
Deleted all files on all disks (including the OS)
Took the tapes home in the back of his car.
He then made a call to his boss telling him what had happened and demanding a ransom for the backup tapes. The boss made 2 phone calls, one to the system vendor who sent a team onto site to 'undelete' all the data (If he'd been brighter he could have erased the data but as with many OS's simply deleting it left it on disk and you just had to find it to recreate the files) ; the second call was to the police. Then on the advice of the police he arranged the meet to hand over the ransom and this dimwit chose the car park of a Motorway services to do the deal, so it wasn't really difficult for the Police to cover all the exits!
The system was back up with minimal impact, although it cost a bit to pay the vendor for the consultancy.
He went down for extortion.
FBI track alleged Anon from unsanitised busty babe pic
IT urine bandit fired and charged
Re: At least it's am original idea!
Well I'm no Brad Pitt myself but having followed the link to the original story I can't see him having much joy myself.
As for the $4,500 bill, I would assume that also includes the cost of cleaning the carpets around the soggy chairs, and if I was sitting in those chairs I'd want the desk disinfected too.
Ten... digital adult toys
Councils tout £1.2bn for IT whizkid to grab their backend
How can family sysadmins make a safe internet playground for kids?
Page won't show his ring to prove Google+ 'engagement'
London 2012 team pulls swamped ticket resale site

bunch of useless b@stards
Have been trying to buy tickets all day, and every time it shows me the tickets that are availble, takes me to the confirmation screen, makes me wait up to 5 minutes before saying the tickets are not available, but they're still showing as available if I go back to the start.
Whoever designed this crap should be used for the targets in the rifle shooting finals.
Demon Currys iPad showered kids with HARD-CORE smut
Man sues boss for 'condemning him to eternal damnation'
Linehan turns IT Crowd off but NOT on again
Name that donkey: Barbarella battles Bathsheba
Fedora 15: More than just a pretty interface

XFCE4 is the way to go.
I upgraded my lappy from Fedora 13 to 15 the other evening and was appalled when I logged in and saw what GNOME3 was like.
A quick "yum groupinstall xfce" and I had a new desktop installed in a minute (try that on Windows AC @Wednesday 25th May 2011 09:47 GMT) and was happy once again, I might even have another look at KDE which I haven't used since Ubuntu was still in nappies.
I've still got GNOME3 on there and I will revisit it but I can't see me getting on with it TBH.
Doctor Who's Elisabeth Sladen dies at 63
Plane or train? Tape or disk? Reg readers speak
backup error condition./run out of tape... #
Once had a customer who loaded the tape and kicked off their backup every Friday night, when they came in on Monday morning the backup had 'failed' (their words) and uloaded the tape, so they reloaded it religiously and the backup completed in no time.
This was only discovered when they had a disk crash and couldn't restore to the new disk so I was sent onsite by DEC to help. Every tape they had only contained about the last %25 of their data.
We had to send the dead disk off to the factory to be stripped down to recover their data.
Naked Cowboy wrestles Naked Cowgirl
Boxer sets lawyers on Facebook
Tasered Oz man bursts into flames

@Adam 52
"including keeping a safe distance or using the not-a-source-of-ignition baton."
So keeping a safe distance, better hope you can run faster and further than him. And if you came at me with a stick while I was carying 2L of petrol and a lighter, I could chuck petrol all over you before you got your first whack in, then when you're close enough all I've got to do is click the lighter once before you beat me unconcious.
It's sad but if I was the cops I'd have done the same.
Beer, 'cos it's safer than petrol.
Mines the Asbestos one with the fire extinguisher in the pocket.