Do you come from a land Down Under? Where diesel's low and techies blunder We go Down Under for today's Who, Me? with a slightly NSFW tale of an incomplete checklist, a surprise outage, and an even more surprised gerbil. "Bruce", as the somewhat unimaginative Regomiser has dubbed our reader, is today's contributor and tells us of an event that occurred in the 1990s. He was working for a telecoms …
From a former company, overheard conversation between CEO and IT Manager:
CEO: You mean it's possible to check exactly what sites our employees are visiting from the firewall logs? Why don't you do this then?
ITMan: Because if I did, I'd have to tell you what I found. Do you really want me to? Really?
CEO: <thinks carefully for a bit...> Err, no. Best not.
This post has been deleted by its author
>> exactly what happened at my work when we first set up internet access
Yes.... the most depressing period in all my history with IT.
Once the rumors of internet abuse reached upper management, setting up the monitoring of internet activity was assigned to me. The results led directly to employee dismissals and demotions. Additional monitoring of phone logs led to one more dismissal. Either due to type of content or amount of time being wasted.
Employees learned quickly to be more careful with company resources.
Unfortunately I was given the task and nothing was ever done with the info. I had several users who should have been shown the door for doing f-all during work hours.
Instead the word leaked out I was the internet police and suddenly became quite a bit less popular.
And I'm not claiming Prom King popularity to start with.
Worked.for a company where at the dawn of the internet PR0N there were no written rules about net surfing. However the company had signs up saying that it was being logged but didn't mention it being checked at all. HR did do some checking and saw some naughty sites being visited by a fair few people.
HR decided to have a quiet word with everybody concerned. One staff member was looking at a BDSM site quite a lot after hours. She was invited to a pesonal meeting where this was brought up. She said the PR0N was lifestyle related as she was in a committed BDSM relationship with her Master. She was looking for inspiration about painful things he could do to her. HR lady said in the meeting that she didn't have to suffer like that, it wasn't normal and the company could get her help.
She explained she didn't need help they'd been together for 15yrs and she was perfectly happy. So happy she'd married him a few years previous. If she'd been watching lesbian PR0N and announced she was gay in the meeting would their reaction have still been it wasn't normal and they'd get her help? "Nope thought not' was what she told me she'd hissed back at them.
I had something similar, late '90s. We installed some internet-facing stuff at a customer site and a couple of weeks later, during a follow-up call, complaints about performance. I got sent in to troubleshoot and of course started with a traffic log analysis, conveniently available from their outgoing gateway (an AS/400, but that's a different story). Within ten minutes, it was clear what caused the performance issues, and the logs also gave with the IPs of the culprits' PCs where company time and resources were spent, well, "watching gerbils".
Wasn't the most fun chat with management to tell them what I found. And, indeed, I said that while I could give them the IPs of the offenders (I didn't tell them I already had the data), I strongly suggested that they would operate under the assumption that that info did not exist and just send a stern warning email around.
Which, luckily, they did.
Sigh....
Alright then, straight from the horse's errr donkey's mouth:
https://www.moroccoworldnews.com/2017/08/225797/15-teenagers-treated-for-rabies-after-engaging-in-bestiality-with-donkey
But indeed, apparently they did not contract rabies:
https://mobile.ledesk.ma/desintox/zoophilie-sur-une-anesse-ni-rage-ni-bestialite-mais-un-travestissement-de-putes-clics
I was a brand new PFY at a small ISP when I got assigned to the security side of the place. My boss wanted to know who was looking at what & when, so had me checking the aforementioned logs. My boss, a rather nice lady that inspired my hormonal young self to go the extra mile so as to hopefully cause her to smile, was definitely non-plussed when the person I found visiting the most disturbing porn sites & the most often turned out to be the CEO... Her husband.
I'm glad I was safely at home when she exploded on her hubby, but the aftermath was obvious when I returned to work on Monday. Her hubby no longer worked there, his office was empty, and I was taken aside by her to ask me most politely to not discuss the real reason for his departure with the rest of the staff. I agreed, she smiled, I floated on cloud 4.5 for a while (it's halfway to Cloud 9 because it's tinged with that icky feeling in the pit of your tummy that YOU may have done something that'll get you fired) and I tread *very* lightly from then on.
Nothing like the knowledge that your every site visit is recorded for eternity in the bowels of the corporate archives & that info can be dredged up to not just "bite you on the arse" but bite your head off, shite down the hole that used to be your throat, & get you fired so hard that even large calibre cannons get jealous.
*Nervous squeak*
Back in the dialup days I was asked help a friend of a member of my family. She had switched on the monitor and on one of the IE windows was a smutty site. Convinced this had to be her son she wanted proof of all the Desmondesque sites he'd visited. After a virus scan proved negative I looked at the logs rather than the totally deleted history. There were quite a few of them and she was hopping mad. I asked when her son went to bed and it was early say 8pm, the daughter half an hour later. All but one of the sites were accessed after 10pm so that ruled the kids out. The daughter confessed she'd gone to a previously normal domain (that had lapsed and been bought by a smut peddler) and saw some smut.
That accounted for the earlier one and I got the feeling her husband would be getting a right telling off that night. The son was obviously playing Wolfenstein obsessively given the game was installed and the maps stored under the keyboard etc.
Anonymous since I and the others still work at the same place...
Many years ago I was asked to watch the web logs for a specific PC as there had been complaints that one of the kitchen staff was browsing porn in the chef's office. Over several days I'd randomly check the logs and if there was activity going on, let HR know who would rush over and nonchalantly enter the chef's office on some pretext, and find it empty and PC off. After about a week of this I found that I was accidentally watching the wrong PC, and was actually watching the traffic for a different staff member with the same first name who, it turned out, was also browsing porn in an open plan office shared with 3 women without being spotted.
Became a bit of a dilemma for HR as to whether to take action or whether the monitoring was invalid but wasn't asked to do the same again...
1) On the quiet, install camera to find out who's siphoning diesel out of the storage tank.
2) Review footage.
3) Present footage to CEO, and wait for a sudden resignation. In our case, of the building services manager, who was earning a bloody sight more than I was, but still felt the need to steal a few quid's worth of diesel once a week.
Optional step 4: While officially declining to give a reason for manager's sudden departure, make sure that unofficially the entire company knows exactly why they went, and exactly how they were caught. This last step can dramatically reduce all sorts of thefts, not just of diesel.
Some years ago (about 2005 or so) I was working in the automotive third party business and we had a contract with one of the large waste disposal companies in the UK.
They were using a lot more diesel than they should be (big trucks but even so...) so a bright individual came up with an idea to fit a RFID tag at the fuel cap of each of their vehicles and put a reader on the fuel nozzle.
The reasoning was that the pump would only operate if it read a valid tag so we implemented the system for them such that a hash of the tag was available from what was basically a password hash file.
Diesel consumption went down by about 25% almost immediately.
No points for guessing who were most put out by the new improved fuel delivery system.
But that's brilliant. You see you dont actually need to set up the cameras! Just tell everyone that due to a series of thefts, you're having hidden cameras set up in "certain" areas, no need to mention specifically where, and boom thefts will stop overnight.
As all the guilty parties (even the ones you werent aware of) will stop assuming it was them you were after...
Whereas in this legislation, they are unannounced and hidden, because they don't serve the purpose of traffic safety, but filling government coffers.
Populistic drivel?
Well, let me give some additional evidence:
- When people started behaving better, i.e. generating less speeding tickets, the fines went up instead of down. Cashflow had to be maintained. In a normal country, fines go up when they DON'T work.
- No speed controls near schools.
- Average speed trap set up on a stretch of motorway that had only seen one lethal accident in the past five years, and that had happened on the parking lot of a gas station. But is was a very busy stretch of motorway, so lots of revenue.
When I was in Canada, I saw signs at road works that read "Fines doubled when workers present". Very sensible. Here they are doubled whether all workers are safely home in bed.
"find out who's siphoning diesel out of the storage tank."
I'd never thought of this aspect before but is generator diesel marked in the same way as agricultural "red diesel"? If so a quiet call to HMRC and a spot check on the offender's tank's content might save the need for involving the CEO.
Back in the day when I lived on a canal boat we had red for agricultural, blue for leisure vehicles (which included house boats for some reason). and standard. There was talk about removing the blue designation. Don't know. Boat sank for other reasons. There's no difference except the price as far as functionality is concerned.
Can confirm: the only difference is tax. (And colouring additive)
Hence the harsh penalties if caught with unauthorised red diesel: you are literally committing tax fraud.
Agricultural: purpose is: subsidy for Food. Ie, fruit and veggies (primarily) are cheaper. Target beneficiaries: low income earners.
Yes, you are permitted to use red diesel for generators. Whether a company goes to the trouble of doing so in order to save a few £ on something that will only be used occasionally is a different matter.
Personally, I filled our genny tank from diesel ferried in cans ferried from the nearest petrol garage, though I could have instead bought it from a more rural garage 20 miles away that sells red diesel - but that would have taken all day.
Here in Oz, as far as I'm aware, diesel is diesel.
Major users such as farmers and truckies can claim significant tax relief (or at least you used to be able to) but there was no difference between what went into the tractor or into my Merc Sprinter in either colour or performance.
Chopper doesn't use diesel, though ----------------------------------------------------^^^
Umm, these days you need a legal opinion before you can install cameras 'on the quiet' to 'spy' on staff.
In my previous employment, we had a presentation where a senior manager presented a story of someone who had been stealing laptops. Basically there was a laptop set up team who received literally hundreds of new shiny Lenovo laptops a week, built them to the company standard and shipped them out to new joiners, people who needed an upgrade, managers who had left theirs in the car overnight and discovered it 'missing' and a broken window in the morning* and others who just wanted a new toy to impress the clients.
Well, it seems that the company was buying a few more laptops than made it out of the build team to staff, and 'no one' knew what was happening to the others. So eventually, and with police authorisation, covert cameras were installed one weekend. The miscreant was caught on video, bold as brass, taking four laptops in boxes and one out of its box (his company car wouldn't hold five in boxes in the boot, poor chap). When his home was searched there was £60,000 (sixty thousand pounds) in banknotes wrapped up in clingfilm, some stolen laptops and various other dubious items.
He was adamant that the company expected a certain amount of loss of laptops and that he was entitled as part of his job to take them. I believe that he went to prison for several years.
The important thing about the police involvement and approval for the cameras is that you are also spying on potentially completely innocent employees too, which is currently against the law (at least for companies to do on their own in the UK).
*That was an interesting de-brief session. I even got to help the PR team to prepare a range of press releases depending on what level of breach might be revealed if things went seriously bad. But that is another story ...
He was adamant that the company expected a certain amount of loss of laptops and that he was entitled as part of his job to take them
Well that's certainly bold as criminal defense strategies go!
"Well your honor, hospitals expect a certain number of patients in the ICU will die, so I'm entitled as part of my job as an ICU nurse to kill my ex husband's new wife by injecting air into her"
DS999: "Well that's certainly bold as criminal defense strategies go!"
Well, it does remind me rather of a short scene in an episode of the northern detective series 'Vera'
Vera Stanhope was telling a teenage girl that there was once a little girl who tried to find and save a man from anti-social activity (euphemistically 'philandering'). The teenage girl asked; "That girl was you, wasn't it?"
Stanhope; pause "yes"
Girl: "And did you? Save him that is?"
Stanhope: pause "No, love, he didn't want saving. They never do."
Criminals generally believe that what they are doing is 'right' in some sense, until they actually meet the victims of their actions. Then they either have a sort or revelatory moment, or genuinely don't care. Stealing 'a few laptops' from a large multinational company just did not seem to be so bad to this guy.
At what was once described as Europe's Biggest Data centre . have no idea if this was true, but as a 23 year old, it was pretty damn big. Think a large aircraft hanger filled with mainframes, tapes, printers and drum storage. The local power builders managed to put a JCB through the heavy duty three phase power supply, which mist have brought tears to the operator.
We ran a motor-generator set, with a large fly wheel to keep the generator going until the diesel kicked in. Which it duly did, coughed spluttered and died. We had forgotten that diesel can wax, which ours, unused for years, duly had. So the injectors clogged and the thing refused to work. The embarrasing thing is that we were an oil company, and we were supposed to know about such things!
When I worked for a large facilities management company, they decided to do a disaster recover test at our site. The power company shut down the electricity supply, the UPS's told the genny to kick in, which it did, and management were all beaming smiles, showing the assembled clients how safe our site was. Beaming smiles right up until the genny ran out of diesel after about 10 minutes. Seems the blokes who drove the nightly backup tapes to the off site storage had been filling up the van from the genny tank.
"The embarrasing thing is that we were an oil company, and we were supposed to know about such things!"
Brilliant - I've only been at one company that did DR processes well, and it doesn't come cheap (but it does come with great reliability, which is correctly more important in certain sectors).
The closest I've come to the failed diesel generator is working next door to a company that, supposedly, had it happen.
Given the number of people here who have claimed to have had it actually happen to them... I am more inclined to believe the reports.
Have seen the effects and got the explanation from someone I knew (broadcast engineering stories are great!)
The location shall remain nameless, but a major market broadcast station suddenly went off the air. UPS kicked in, generator started but did not power the facility because the wiring was incorrect. 'Twas a brand new facility and the electricians had screwed up.
Electricians? Over the years I've come across an alarming number of them that have no idea what they are doing, both in industry and domestic environments. It seems they memorise just enough of the current regs. to get though their exams then forget everything.
Oh, including one who was muttering about the three phase motor he had to fit, as it was going in the wrong direction. When I said offhand "You've probably swapped two of the phases", his dismissive reply was "What difference does that make?"
Yet I'm not considered capable of adding a spur to a domestic ring main.
Mr Godfrey: "Electricians? Over the years I've come across an alarming number of them that have no idea what they are doing, both in industry and domestic environments. "
You seem surprised by this. They are obviously management trainees on their ways to higher and greater things.
Yes, I know what you mean. Some particularly resourceful chaps round my way managed to cause chaos with their wiring of a backup generator: utility power failed, UPS kicked in fine. Generator started, system attempted to move to generator, building went completely dead. Somehow, the cut out tried to send the generator power back into the grid, frying a decent sized area, which took most of the weekend to sort out. I think there were probably three or four follow up visits to the site once it had been fixed as it was tested, very thoroughly, afterwards. The owner of this particular building? Well, that would be the local electric supplier.
Helps to be able to turn a blind eye. It's a lesson I taught myself after helping to fix the computer of a sixty-something year old woman who lived by herself and had multiple things on her backdrop and bookmarks that would suggest that she really liked gay men dressed as what's-her-name from the Wizard of Oz.
From then on (and until a different expat made me refuse to ever do that sort of thing again) I just paid no attention whatever to anything not directly related to the job in hand.
I raise you the "back end computer" from a pizza shop.
Stopped going to that one because there was no way that I was eating pizza stretched on hands that created that browser history and folder structure.
Professionally repaired it, made no mention of the history/favorites, left the cache of pics, and never returned.
Shame, because it was pretty good pie for a reasonable price.
Mine is the one with Purell in the pocket back before everyone had a bottle.
I used to maintain the computers and network for a friend who had a chain of builders merchants locally.
The kind of people he had to employ (knew the business and the parts) had the filthiest minds imaginable. But one time he was having a particular recurring problem with a computer at one branch and he asked me to look into it for him. The computer was brand new, but it kept getting various malware on it that stopped it working for the sales and inventory purposes it was primarily intended for.
The Event Log was revealing enough (packed with pr0n), but it didn't show who was doing it. He wanted to know, so I installed a keylogger (he specifically requested it when I told him it was one possible way).
It identified the one person doing it after only a few days - log-in details, and search terms. This was over 20 years ago, but what he was looking for was stomach-churning,
"Wombat: eats roots and leaves"
Text from an old Australian children's encyclopaedia, which immediately entered Aussie legend.
(For the uninitiated, "root" is Aussie slang for --how should I put this? Faux-formal, methinks-- sexual intercourse, typically of a casual nature or lacking emotional involvement.)
Unknown author Liz Truss much later nicked the joke and mangled it for non-Aussie consumption to entitle her Punctuation book, "Eats shoots and leaves".
If there was an EMP, then what would there to be power once the new generator came up ?
.. and secondly ... I though it made no difference if things were on or off in regard to an EMP.. being that the pulse is induced into the wiring of everything, so this will be both sides of the magic power switch.
In the event of a major solar flare causing a global EMP it is, it seems, best to leave everything connected up so the pulse can be distributed across as much infrastructure as possible. If something is in the 'off' position, there is a spark gap for the induced current to leap across. Curiously one of the dangers of cave diving is lightning storms. Being the bearer of a metallic cylinder in a water-filled cave is apparently not a good idea if the ground above you is struck by lightning.
Oh, and try to live on sedimentary rather than igneous rock.
https://www.livescience.com/38848-emp-solar-storm-danger.html
https://curious-droid.com/1349/how-prepared-are-we-for-a-carrington-level-solar-storm/
At my last company we moved into a new building which had a huge flywheel generator thingy on the roof. It was frequently tested, so everyone was under the impression it would do the job if it came to it.
Obviously one day a JCB (or similar) went through some nearby power lines and it was time for emergency power... or it would have been had someone remembered to flip the switch from "testing mode" to "do something when the power goes out mode". Oops.
This didn't effect me as I wasn't in the nice shiny building, but the old building a lot of the company didn't know existed which housed the computers/phone switches etc where we toiled in the basement. Ironically it couldn't be made a machine room down there as "it might flood".
One (large) place where I worked had an automatic diesel generator set (ISO-container sized IIRC) set up to come on automatically on detecting loss of incoming mains, and off again once mains was restored.
I can't remember whether this was on a deliberate test, or a real mains failure, but the first time it detected loss of power and switched itself on... it immediately detected that the power was back, and switched itself off again.
... but a good one. Learnt this from a colleague who was present at the time.
A certain US hospital way back in the early 90's had installed a highly resilient power system, mostly because of certain issues when they had previously lost power. Basically they had a substantial UPS to handle critical power circuits (e.g. the ICU) for up to 30 minutes, plus a backup diesel generator system that could handle the entire hospital for as long as they kept the fuel tanks topped up. The fuel tanks where regularly checked and the diesel cleaned (no waxing here!); also the whole system was checked out every month by starting up the generators and making sure that they produced the correct power. The UPS only had to handle the load for about the 5 minutes it took the generators to come online (all automatic of course) and spool up to full capacity.
One thing about this setup, the fuel tanks where about 100 yards away from the generator for safety reasons (you do not want large fuel tanks catching fire right next to a hospital) so they hand a pretty impressive fuel pump to get the diesel from point A to point B. Again, this was checked every month.
One day, the inevitable happened - the mains power went off. UPS took over the load fine. The diesel generators powered on and started to handle the load as designed - for about 30 seconds, then they died and could not be restarted. Major panic started; fortunately power was restored before the UPS expired, but it was a close run thing after which many trousers need to be changed.
On investigation it was discovered that the fuel pumps (remember those) had been wired into the mains power feed only, and could not pick up power from the UPS or generators.
When I was in college during the 1965 blackout the associated hospital had emergency power (of course). The New Hampshire power was so unreliable that the system was used regularly. During the blackout the hospital was a blaze of lights surrounded by darkness. Sometimes these things work.
Worked at a company many years ago where the sysadmin had a *very* liberal attitude to content filtering ie there was none. He believed in treating people like adults and letting them have 'personal responsibility' for their web surfing. This worked very well until he had to have a talk to one of the guys:
[name redacted], you know I don't look at the firewall logs for your web browsing and, frankly, I don't care. But if you do print things out, for Christ's sake please collect them from the printer!
One place I worked we were given advanced notice by the electric board that they were cutting our power for a whole weekend.
This meant the regularly tested 1 hour UPS wasn't going to cut it, so we went up to the roof to check on the big Cummings diesel genny, which after 5 years of standing idle surprisingly kicked into life almost immediately, and then it died. We dipped the tank, empty.
So we ordered a barrel of diesel which was delivered that afternoon.
In the loading bay we had a pump to squirt the fuel up to the roof, but it wouldn't work, it had seized up because it was about 15 years since it had last been used.
So we rolled the barrel over to the freight elevator, which we then realised stopped one floor short of the roof, so 4 of us laboriously stepped the 55 gallon barrel up the emergency steps and onto the roof, where we siphoned the dirty diesel into thirsty generator's fuel tank, we fired it up and it worked just fine, delivering a reasonably steady 230v, which the online UPS would clean-up before it hit the servers.
Then back in the office we got a call from Electricity North West telling us they didn't need to cut our power anymore as they'd found a different way to fix the grid without cutting us off...
I was amazed once when visiting a Castrol oil processing place to see a fairly small guy spinning two 45 gal. imp/55 gal. US drums on their edge, each controlled by one hand as he walked along. A final flick and spin, and the drums spun off to stand upright in their proper positions. I felt like clapping.
I worked for a tyre fitting company for a short while, just about when the "Super Single" tyres were first being fitted to semi- trailer tankers. I was only quite small then, about 8 stone, but I was sent on a course on how to fit and demount them. My first callout was to a tanker in the Services at Watford Gap on the M1. A whole gang of truckers were sitting in the transport caff, watching me and expecting me to struggle with this tyre/wheel assembly that weighed almost as much as I did. Remove wheel from axle, keeping it upright, insert specially shaped tyre levers at 11 o'clock and two o'clock, then let the wheel fall towards your legs, snatching the ends of the levers upwards just as the tyre is about to hit you in the stomach. Tyre stands up again, and wheel shoots out between your legs. Nonchalantly roll dead tyre away to sound of astonished applause from truckers in transport caff, before dropping replacement tyre onto now horizontal wheel. Only stayed with that company for three months, until I landed a draughtsman's job at a local engineering works.
"Then back in the office we got a call from Electricity North West telling us they didn't need to cut our power anymore as they'd found a different way to fix the grid without cutting us off..."
Did anyone send the power company a thankyou note for teaching you the lesson that you need to check your fuel tank and fuel pump more often?
Part one..... when fitting a genny make sure that phase and neutral aren't swapped over behind the wiring panel ......
Part two ..... big 3-phase UPS ..... make sure that the incoming power sense leads are actually conected to all 3 phases....
Part 3 .... dont let a <redatcted> Manager demolish a wall in the warehouse using a JCB.....
Part 1 -- bang, bang ... bang went the PC power supplies
Part 2 -- the lights are on but why is the main frame not .... oh ....
Part 3 -- big blue flash followed by much darkness and a bit of silence ......
Our church is still "small" but a large enough building to require a 3-phase feed from the lines at the main road, fed to different parts of the building.
Last Easter (yes, 2020, during lockdowns) when the church was only doing Facebook Live (and the worship team still came in person to perform), one phase was dropping intermittently, and we had no way to cross-connect, probably to avoid overloading the others. Half the sanctuary lights were out as were the outlets on one side, which fed:
The main AV system, including main mixer, main speakers and front monitors, screens, HDMI camera, iMac, etc. were fine, as well as the Wi-Fi and cable modem elsewhere in the building.
The sounds and visuals as the phase went in and out were scary: lights flickering, HVAC ramping up then going quiet again. Finally a member who was an electrician -- who had brought his daughters that day to sing -- figured it out and shut the affected main panels down completely.
On a "normal" Sunday, we could just use the piano (baby grand) and acoustic guitar and carry on. But no, this year we HAD to stream the service, so we HAD to have working audio. Thankfully, saving our Easter merely required some extension cords across the sanctuary stage from still-working outlets. Those outlets were never used, so we had capacity to spare.
(The local utility had things back up and running the next day, and by His grace we didn't toast any equipment in the process.)
Years (well, decades) ago I went on a tour of the Mullard Radio Astronomy Observatory, and was told that the UPS for the 5km telescope was a motor-flywheel-generator set which stored enough energy to shut down the computer centre and park 8 90-ton dishes in a high wind. They reckoned that if the flywheel came off its bearings, it would roll about 25 miles, so it was aimed to have as clear a path between villages as they could arrange.
I hope the DR plan included a telephone number for emergency services in case the flywheel went walkies. It would make chasing a Double Gloucester down a hill look positively tedious.
https://theculturetrip.com/europe/united-kingdom/england/articles/a-quick-guide-to-cheese-rolling-englands-strangest-sport/
https://www.youtube.com/watch?v=cvuktushEhY
My father was an "His Majesty's Inspector of Armaments" at a large factory during WW2, where they produced camera lenses, gunsights, and bombing predicters. The factory was on the north side of Clapham Common, and there was a huge steam engine in the basement driving a generator to power the machines in the factory. One night in 1942, Jerry came over and bombed the shit out of the factory and surrounding area. The seismic shock snapped the 12" diameter crankshaft and the 10 foot diameter by 2 foot face width flywheel climbed out of its pit, demolished what was left of the basement wall, and trundled off across the common, eventually ending up spinning like a giant coin and settling down in a pond. They never rebuilt the factory, it was relocated in Barnet, on the northern outskirts of London, so Dad had to relocate as well, so I was born in Barnet General Hospital some five years later.
One building I heard about never tested their backup generator. Cue power failure. Generator kicked in. Yay!
No-one thought to check where the fresh air inlet for the HVAC system was. The tech support team (in the same building as the data centre) got gassed with the exhaust
During the 2004 eastern North American power failure, I was working at a generic drug manufacturer who kept a large amount of their output in very large refrigerators. The diesel generators kicked in but they only had 4 hours of fuel. The warehouse manager tried to order more, but all the diesel deliveries were now unavailable. Some one noticed that the gas station at the end of the road still had, for some unknown reason, power so they started running 55 gallon drug transfer barrels to the gas station on handcarts and bypassing the cars in the fuel queue to keep the generators running.
A major bank had a power outage. The operators opened the procedures book and started the fail over to the backup site - a well practised procedure.
A was senior executive was passing and came to assist, saying "We have these big generators in the car park for times like this. Restart the systems in place"
The operators said "That's not in the book - it has not been tested". Senior Executive "If you can restart in place it will be back in no time! - read my job title"
They did what they were told - only to find the generators did not provide enough power for the machine room, and they could only start half the kit.
The operators made the bold decision to wait for the power to come back. They said that they thought they could not successfully fail-over to the backup site once they had started the restart. Not the sort of thing you want to first try in production.
Back in the day, I was leading a team qualifying the server room for a government service responsible for medical supplies across the entire country. The project scope included building security, server room air handling, server room fire suppression, and emergency power.
The power test was to take down the main power to test the generator auto-start, then take down the generator to test the switch over to UPS, then bring the generator back on line, then switch back to main power. I had been told by the IT manager that the UPS was good for forty, maybe forty five minutes.
Take down main power feed: Reassuring rumble from the roof as the generator kicked in.
Take down generator: Everything still running as the UPS kicked in.
IT Support Guy: "You better get the generator back up. The UPS is overloaded; it's only good for about ten minutes" WHAT!!! I was told forty! IT Support Guy: "Who told you that?"
Frantic walkie-talkie call to guys on roof to fire up generator... NOW! Reply from guys on roof, "It needs a five minute cool down before we can restart".
The generator came back online with about two minutes to spare.
As a post-grad student at my first real employment, it is one my deepest embedded memories of a 1MW diesel-generator starting to fail after operating at 120% load for a few hours; it was OK, we knew what we were doing.... The alternator caught fire, evidenced by streams of smoke from the engine cell. The reaction was exactly like an episode of the Keystone Cops: guys in overalls running in every direction, across, into and out of the smoke, I thought for our entertainment.
The Chief turned the load off. Island application, so limited fall out.
It was running again a few hours late at a lighter load..... The 'Cops' were a bit shamefaced for evermore.
It was the first time I had experienced that 'electrical overload' odour that has proved useful more often than I care to admit.
I was told this tale by an IT support guy I worked with back in the day. He did a lot of shop and small office work and one of his customers was a large store which had a couple of back-office servers plus PCs for stock control etc. which were supported by a smallish UPS. If there was a power cut the store would be closed but the UPS would allow the office staff to shut down everything tidily before the UPS power ran out. One of his regular duties any time he visited the site was to check out the UPS under load, make sure it worked and that the batteries were swapped out every couple of years to keep them functional.
Day came, there was a power cut, the UPS ran for less time than expected and the servers went bonk! before they could be soft-landed. Oops. Investigation discovered someone had taken the microwave out of the staff break room and plugged it in to a UPS-fed socket over by a stock control PC to heat up their lunch while they waited for the mains power to come back on. Ding!
Worked for a Stockbroker in Perth on the Esplanade back in the 90s and the building we were in had a large genny set in the basement to provide essential power. During a major winter storm, the power supply to Perth was taken out for about a week. During that time, the building management had to bring in tankers of diesel to keep their generators running. I remember having to remind staff that the red outlets were for essential computers only and no you could not plug your mobile in to get it recharged.
When Auckland City had a major, weeks-long power outage [20 Jan – 27 Mar 1998 according to Uncle Google], lots of businesses found out two unexpected things about their backup generators.
The first was that, it being summer time, the generators tended to get all hot & bothered if they were required to run for hours & hours at a time, rather than the one hour every other Tuesday lunchtime that the testing regime required. A lot of sheet metal was bashed into airflow ducting and had fans installed to keep fresh coolish air blowing onto the generator.
The other was that the diesel tank went from full to empty in between sixty and seventy hours, so needed a refill every other day. Since the tank was down a twisty narrow central city service lane, a "mini tanker" was always used. The fuel companies ended up getting every mini-tanker in the country to Auckland to feed the generators - they even drove one from Invercargill to Auckland, including ferry across Cook Strait, using a relay of drivers.
Fun times ... not
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
