Techie was given strict instructions not to disrupt client. Then he touched one box and the lights went out

Welcome to another instalment of On Call, The Register's weekly reader-contributed column that tells tales of times when tech support turned troublesome. This week, we're again sharing a story from "Kent," who last week told us how a client held him hostage. This week, a client held him responsible for something he didn't do …

[Sorry that handle is already taken.]

"panic spreads faster than Wi-Fi"

How fast does Wi-Fi spread? I couldn't find it here.

And is it temperature dependent, like honey?

[Michael H.F. Wilkinson]

Re: "panic spreads faster than Wi-Fi"

I doubt it travels merely at the speed of a sheep in vacuum. Must be a lot faster

I'll get me coat

[Ken Shabby]

Re: "panic spreads faster than Wi-Fi"

Like “Snot off a stick”

[Yorick Hunt]

Re: "panic spreads faster than Wi-Fi"

An unladen African swallow.

[brodders]

Re: "panic spreads faster than Wi-Fi"

That fast?? I need an upgrade. Perhaps carrying only one coconut would help...

[Gavsky]

Re: "panic spreads faster than Wi-Fi"

That Einstein, right? Not as stuffy as some think! Obviously this was Futons, not WiFi - but the principle is the same...

[Anonymous Coward]

Re: "panic spreads faster than Wi-Fi"

From the nocat.net website, now relegated to the wayback machine:

Albert Einstein, when asked to describe radio, replied:

"You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat."

[Paul Herber]

Re: "panic spreads faster than Wi-Fi"

That'll be the hypothetical round sheep of zero mass.

[NXM]

Re: "panic spreads faster than Wi-Fi"

I read that as 'I'll get me goat'

[I'm not Welsh]

Re: "panic spreads faster than Wi-Fi"

Not as fast as the speed of darkness.

Hat tip to Tel. RIP

[Evil Scot]

Re: "panic spreads faster than Wi-Fi"

GNU not RIP

Always forward.

Do not record.

Return when end of network reached.

[Return To Sender]

Re: "panic spreads faster than Wi-Fi"

I'm forever indebted to Tel for "like prunes through a short grandma"

[jake]

Re: "panic spreads faster than Wi-Fi"

"How fast does Wi-Fi spread?"

Almost as fast as gossip through a small village.

"And is it temperature dependent, like honey?"

Probably not. Although gossip is faster in the winter than in the summer.

[Gavsky]

Re: "panic spreads faster than Wi-Fi"

Hmm, I feel a publicly-funded, research project coming on! ('coz Cancer, yada-yada). We need a small village, a bored population & some juicy gossip. Then, repeat using WiFi. Oh, and at +100c then -100c (both).

[Gavsky]

Re: "panic spreads faster than Wi-Fi"

First thoughts: yes - heat a room up & WiFi spreads more easily ('cos psychics, right?). Too hot & the binary runs all over, splitting from the bits it needs to be connected to...you don't want it seeping between floorboards etc.

[The Oncoming Scorn]

Re: "panic spreads faster than Wi-Fi"

Obligatory...

https://pbs.twimg.com/media/EMzPpSrXsAICxMK.jpg

[keith_w]

Re: "panic spreads faster than Wi-Fi"

That's none of your beeswax.

[Sunset]

Re: "panic spreads faster than Wi-Fi"

Honestly, that wording sounds vaguely LLM-y. They're fond of inserting flowery but nonsensical analogies.

[NetMage]

Re: "panic spreads faster than Wi-Fi"

You may need to rub the LLM out of your eyes.

[Patch Wombat]

Re: "panic spreads faster than Wi-Fi"

faster than norovirus through a cruise ship daycare center

[Anonymous Coward]

I once entered a small glass-walled air conditioned server room in the middle of a room filled with cubicles. Everyone curls see me from the waist up. I dropped a pencil and bent down to pick it up. I didn’t touch a keyboard or mouse or button.

When I stood up, everyone in the office thought I caused all their terminals to freeze. I laughed and they were ok because it was a regular client who knew me.

I’ll never forget it as a classic botched cause and effect error humans make.

[lglethal]

Admit you dropped the pencil across some open wires or ports causing a short... You were just lucky nobody could see... ;)

You said you didnt touch a keyboard, mouse or button... You didnt say anything about wires/ports/other electrical equipment, now did you...? We're on to you AC... :P

[Anonymous Coward]

We were doing an out of hours non essential equipment manged shutdown as a test in preparation for any power grid problems caused Millennium Bug. This was to test the generators and UPS so that if there was an issue we knew about it well in advance if the power did go out. We had people stationed in all areas and on all floors of the complex supporting and reporting that the bits of kit that were supposed to be still powered and running running. So 18:25pm rolls round and we are all doing final checks that desktops, printers and other equipment not involved in the test are switched off. A voice booms out from the tannoy that we have five minutes left, so I moved to the equipment room on my floor and my area hot drink in hand to wait.

At 18:30 exactly power was cut and then a minute later mains power was restored, not the 5 minute test it was supposed to be. Tannoy booms out again thanking everyone and saying that we would be doing another one in a few weeks. I caught up with the business resilience manager and the complex facilities manager who were sitting in a meeting room thumping the table in there. It turns out the UPS was fine but the generators did not fire up when the power was switched off. The work on this and a few other things was contracted out to a maintenance company. As we discussed the implications and how quickly the remedial work could be done. Then we turned our attention to the company paid to maintain all this and patently obviously hadn’t. We got the head of the legal team on the phone and he said that firstly we would sack them and the sue them to recover out money. They were in for a serious kicking as a result, and security revoked all the ID cards they had, that night.

Next day two of their staff turned up to work and were very surprised when they couldn’t get in, and were told to call their company. Of course then there were no problems with the power on December 31st/January 1sr but at least then with a new maintenance company and another test, we knew it all worked as expected.

[Anonymous Coward]

I have known some people who also could make a whole room freeze when they bent over to pick something up...

[I could be a dog really]

Going waaaay back, when you needed a copy of a drawing, you'd put a requisition in to the tracing department. They'd get the master out of storage, and a tracer would spread it out with tracing paper over the top and copy it - line by line, character by character. YOu'd then get the traced copy and the master would go back into the store. Many of these drawings were A0 in size (no idea what that is in ElReg units), or even A0+ (i.e. A0 width but longer). And to trace them, you had to use a large drawing table, which meant leaning over a lot - especially when working towards the top of the sheet.

Most of these tracers were young ladies, and then the 60s arrived with very short skirts being the fashion - so naturally the engineers (almost all men of various ages) would find all sorts of excuses to visit the tracing office :)

It wasn't long until the manager of the office made the tracers pin dusters to the back of their skirts :(

[Aleph0]

According to paper-size.com A0 is exactly one square metre (849 x 1189 mm) so 0.0481 nanoWales.

BTW A0+ is 914 x 1292 mm, converting to 1.181 m² and 0.568 nW.

[I could be a dog really]

And an interesting fact about the A series paper sizes that you may or may not be interested in.

They are geometric, and all the standard sizes have the same length-width ratio. So if you take an A0 sheet and fold it (across the shorter mid-line) in half, you get A1. Similarly, A2 is half an A1, A3 half an A2, A4 half an A3, and so on.

[Yes Me]

And so on until what? Is there a quantum mechanical limit somewhere beyond A42?

And do two A0s make an A-1?

[ArguablyShrugs]

> And so on until what? Is there a quantum mechanical limit somewhere beyond A42?

According to XKCD, you get a black hole around A190

https://xkcd.com/3033/

[Mr Army]

I think you need to check your maths.

There is not too much difference in the dimensions of A0 vs A0+ however A0+ appears to be more than 10 times larger in NanoWales that A0.

[Alan Brown]

Or cause the office gopher to walk into a pillar...

[Lazlo Woodbine]

I used to run a photography summer school course at the school where I work.

One day, I'd just set everything up, introduced myself to the group then as soon as I pressed F5 to start the powerpoint everything went dark in the room.

Looking outside showed the other parts of the school site were equally without power, indeed the whole village had slipped off the grid.

Luckily, we have a secondary site about a mile up the road, so we decamped to a spare classroom there, which turned out to make the course better, as we had a lot more interesting things to photograph in the afternoon, so I ran the course from that site for the rest of the summer...

[Paul Herber]

No negative outcome from the event then.

[phy445]

Downvote - really?

Trying to understand the downvote on this. Fat fingers or an El Reg reader too young to understand the allusion?

[Lazlo Woodbine]

Re: Downvote - really?

I've decided that some people are so joyless they just downvote everything...

[Terry 6]

Re: Downvote - really?

Yeah, it's an El Reg thing. Par for the the course a Commentard with 10+ upvotes for an indisputable comment will still get that one downvote.

[stiine]

Re: Downvote - really?

OK, I'll check back later.

[Gavsky]

Re: Downvote - really?

I like to think it's a Yin-Yang, Universe balance-type 'thang. A negative vote has more mass (obviously, look I don't make the rules). But, too many & the whole comments board will simply slide-off our screens...

[Stu J]

Re: Downvote - really?

You're welcome

[chivo243]

Re: Downvote - really?

every commentard has their downvoter.

And wait for it!

[M.V. Lipvig]

Re: Downvote - really?

SOME folks around here, commojly known as leftists, will be offended by something you posted on one story, and will automatically thumb you down every time they see your name.

Others, like myself, decide on a post by post basis with no care about your username.

[Anonymous Coward]

Re: Downvote - really?

Correct apart from the 'leftists' trope.

:)

[Anonymous Coward]

Re: Downvote - really?

You need to focus on the subject at hand, don't worry about the negatives at this stage. It'll all be over in a flash.

[wolfetone]

Re: Downvote - really?

Took you a while to develop that, didn't it?

[Korev]

Re: Downvote - really?

That's the cream of the crop

[Will Godfrey]

Re: Downvote - really?

You took your time exposing that comment.

[The Oncoming Scorn]

Re: Downvote - really?

Just f-stop with the puns will you.

[C R Mudgeon]

Re: Downvote - really?

We're all fixers here.

[jake]

Re: Downvote - really?

Probably doesn't know how to filter their RAW thoughts.

_{Joking aside, I assumed it was a kid who is probably too young to remember a world without the iphone.}

[KittenHuffer]

Re: Downvote - really?

Yeah, some of these young whippersnappers! I'm still running computers that are older than half of them!

And when I first got these machines they were state of the art! ..... Now they're considered to be state of the carpet!

------> The can is Red Bull .... to keep me going! (Other high-sugar energy drinks are available!)

[A.P. Veening]

Re: Downvote - really?

The correct expression is state of the ark.

[Pete 2]

Re: Downvote - really?

> Trying to understand the downvote on this

I wouldn't worry, down votes are just as good as upvotes¹. They both indicate someone has engaged - even if they didn't read or understand the comment in question.

[1] Says someone who has received 3971 down votes in the nineteen years (on Monday coming!) I have been commenting here.

[l8gravely]

Re: Downvote - really?

I never knew elreg kept these stats, awesome! Of course I've been here for years... but rarely post.... but when I do it's _quality_! :-)

your 197 public posts have been upvoted 1252 times and downvoted 88 times.

[Anonymous Coward]

Re: Downvote - really?

If you really want to rack up the downvotes, make a snide remark about some dead bloke on a roof in London.

[Terry 6]

Re: Downvote - really?

Yup

In total, your posts have been upvoted 38466 times and downvoted 2740 times.

Decent ratio IMHO

[Dave314159ggggdffsdds]

Re: Downvote - really?

There's (at least) one commenter here who has a script that scans all the new comments and automatically votes down anything posted by anyone he's ever had a disagreement with (which is basically every long-standing commenter).

That aside, I suspect some fairly large proportion of otherwise inexplicable up or down votes on posts across the internet come down to people scrolling on touchscreen devices and accidentally tapping the button.

[M.V. Lipvig]

Re: Downvote - really?

"That aside, I suspect some fairly large proportion of otherwise inexplicable up or down votes on posts across the internet come down to people scrolling on touchscreen devices and accidentally tapping the button."

That's caught me out a few times, as touching a second time does not erase. Fat thumbs, poor eyesight and a 10 minute edit window is why you also see misspelled words from me.

[Peter Gathercole]

Re: Downvote - really?

Hit the upvote on the comment you downvoted, and it cancels the downvote (and I presume vice-versa).

Has anybody else noticed that the vote counter highlights the up/down arrow on a comment when you've pressed one of them, so you can see whether you've already voted on it, and if you've voted one way, pressing the other arrow cancels and clears the highlight it if you pressed one by mistake.

[NetMage]

Re: Downvote - really?

But there is no way to return to no vote meant so Inalways just downvote.

[Peter Gathercole]

Re: Downvote - really?

No. I think that one click on the opposite arrow cancels the original. To reverse it, you have to then press a second time.

[Yes Me]

Re: Downvote - really?

Not so. I have just been happily toggling you between 1-1 and 2-0.

[thosrtanner]

Re: Downvote - really?

Sadly you can go from an upvote to a downvote and vice versa but you can't go from an upvote or downvote to no vote

[Excused Boots]

Re: Downvote - really?

"Hit the upvote on the comment you downvoted, and it cancels the downvote (and I presume vice-versa).

Has anybody else noticed that the vote counter highlights the up/down arrow on a comment when you've pressed one of them, so you can see whether you've already voted on it, and if you've voted one way, pressing the other arrow cancels and clears the highlight it if you pressed one by mistake."

And why on Earth did someone downvote this post? There is no way it can be construed as controversial, it's just explaining how the system works and how you can ‘fix’ a mistake.

And yes I have noted the highlighted arrows, always useful if I revisit a thread, re-read comments and think to myself ‘that’s good, I’ll upvote it - oh I already have’!

I’m sad to say that I am beginning to come around to the idea that some commutards on here do actually run scripts that scan for and automatically downvote any and all comments from someone that they have ever disagreed with - irrespective of what that post is talking about or it’s usefulness!

Which is a real shame.

[Aleph0]

Re: Downvote - really?

"And why on Earth did someone downvote this post? There is no way it can be construed as controversial, it's just explaining how the system works and how you can ‘fix’ a mistake."

Maybe because they see it as factually wrong? (Disclaimer, I didn't vote one way or the other) I don't know how it goes for other browsers, but on Firefox if I have upvoted a post and want to rescind that vote there's no way to do return the post status to unvoted:

* if I press again on the same up/down arrow it remains highlighted and the vote counter doesn't change;

* if I press on the opposite arrow the highlighting moves to that one, and both up/downvote counters change by one (one increases and the other decreases).

Icon is for El Reg programmers, if someone could get round to address this minor annoyance it would be quite helpful...

[The Travelling Dangleberries]

The story might still be developing as we have only seen a snapshot of the situation. Not the whole picture. It could still easily slide into being a complete disaster.

[Mishak]

Yes, could be a bit of a red-light moment.

[ModicumSuch]

The photography puns on this thread are all below standard and have been reported to ISO.

[Headley_Grange]

You should have pressed f5.6

[JulieM]

That's the problem. Should have pressed f/8!

[Paul Herber]

Who are you calling effete?

[ChrisC]

Does yours not go up to f/11 then?

[Paul Herber]

Stop!!!!

[Bonzo_red]

lfl

[NorthIowan]

To be clear

To be clear, you have to go to F64. Right Ansel?

[LateAgain]

Undo is usually control z.

Just saying you might want to remember that.

[Korev]

Sounds like some light entertainment

[lglethal]

He shouldnt have kept people in the dark...

[Anonymous Coward]

Stop light?

[Giles C]

Hmm

Somebody yesterday told me a very similar story was the source for this visiting Cambridge yesterday……

[Rivalroger]

power-off command

A distressingly long period of time ago I was working for a local authority. The facilities people had scheduled some power work in the Town Hall and we had to power off the computer room we had there. I, as the UNIX admin had cronned a shutdown on all my Solaris boxes and a few new-fangled Linux machines. To be sure, I had issued an init 0 to the Sun boxes and went in to enter power off at the OK prompt. I walked up to one of my test machines to do do the do. At the exact moment I entered return, someone flipped the switch for the whole computer room. I knew my Windows colleagues hadn't finished their carefullly timed series of power offs so I had a moment of (unfounded) clenched buttocks. The lead windoze guy laughed rather a lot a my expense when he saw the look on my face.

[Anonymous Coward]

I was once installing something in a school (I don't recall what now) - powered by USB, so I plugged a USB PSU into an extension lead under the reception desk.

As a great man once said: "I felt a great disturbance in the Force, as if millions of voices suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened"

There was a bang and suddenly an overwhelming sense of silence. It took 10 minutes looking for the distribution board and finding someone with the appropriate keys. Thankfully it had only tripped the circuit covering offices, not any of the classrooms. I went back the following day with a different PSU which I'd already tested off site!

[jake]

in the frame to wear the blame?

No, but I did cause the outage once ... and got away with it[0[. As I posted 4 days ago[1]:

When we were decommissioning the old Fabian Avenue telco Central Office (now home to the Charleston Village condominiums), I was given the task of making sure the electrical power to the site was off. Not just at the breaker down at the street pole, but the physical breaker at the Colorado Avenue Sub Station in Palo Alto was to be pulled, thus making certain all power was deactivated until we could make certain everything was isolated.

This would involve taking out the entire Charleston Gardens section of Palo Alto for an hour or so mid-morning, mid-week when it would cause as little disruption as possible. (Charleston Gardens is a mostly residential section of Palo Alto, bordered by Middlefield, San Antonio and Charleston roads, if you care.) The neighbors were notified the week before, both by snail-mail and people physically knocking on doors to explain and hand out the small paper notice explaining what and why.

Come the morning of the Great Event, I was selected to physically make sure the power was off at the sub-station. Cell phones being a fad of the future, I drove down the Frontage road (West Bayshore) and arrived at the the sub station promptly at 10AM. Conversation went something like this:

Me: I'm here to see that the power to the Fabian project is off.

Site Engineer: We're all ready for you, I'll get to it in a second ... or you can just throw that switch (points).

Me: OK (throws switch).

Engineer: NOT THAT SWITCH!

Most of South Palo Alto: WTF‽‽‽‽‽

Management swallowed the story that the main breaker tripping was an un-foreseen knock-on effect of the smaller section being taken down ... The Engineer and I made a bee-line for Fred's (well known dive bar on the Palo Alto/Mountain View border) as soon as he cleared up the problem, which took into the late afternoon. We're still friends.

_{[0] Statute of Limitations applies. I'm in the clear. I checked before I posted it the first time, years ago.}

_{[1] Timing, as they say, is everything ...}

[Anonymous Coward]

Headline writer...

"'Techie was given strict instructions not to disrupt client. Then he touched one box and the lights went out"

What you've essentially done is, like every social media post out there, given us the story in the headline

Please look back on previous el reg articles and see the dry wit used.

So much mileage could have been gained from MC Hammer for a story like this

[Anonymous Anti-ANC South African Coward]

Re: Headline writer...

HALT!

HAMMERZEIT!

[Uncle Slacky]

Lecture demo gone wrong?

Reminds me of an demonstration given by one of my lecturers at university (a true "mad/eccentric professor" type) - it was something to do with nodes and antinodes in AC circuits, and the moment he turned it on the entire city's power went out. I wonder to this day whether he mgiht have caused it...

[Giles C]

Happened to me

Working the offices we were doing some cabling work under the floor of the comms room with the digital telephone system, which had bundles of cables everywhere.

I needed to move a newly installed 63A socket which had been left on top of the cable bundle. So I picked it up there was a bang and the entire site went black, a few seconds later the emergency lights kicked in and the office was silent.

Oops…..

It turned out the electricians hadn’t screwed on of the terminals in the socket properly and when I picked it up the cable had fallen out and dropped onto one of the other connectors. The short circuit too at the board in the room we were in, 5e distribution board for the comms room, then the one for the building and finally tripped the generator breaker so no emergency power…

We got everyone on site and then restored all the functions and went home, this was on a Saturday night and I had the Monday off. By the time I walked In the office Tuesday the rumour mill had me either in hospital or dead due to an electrocution…..

[C R Mudgeon]

Re: Happened to me

I hope they were stimulated to learn something about cascade failures and how to prevent them.

[andy the pessimist]

It was a long time ago...

I had set up the ATE (credence LT1001) and wafer prober. The setup was good.

I hit return to run the program. The lights went out, i shouted. The whole site was affected.

Eveyone evacuated the test area. (No nasty chemicals).

The FAB people evacuated asap. No power means no air extract. Nasty chemicals so get out quick.

It wasn't raining so my clean gear and me didn't get wet.

[Alister]

I had a similar experience.

Working at a tech college, we had students who had a classrooms of machines on which they tested pushing out machine images to hosts from an AD domain.

The AD servers, DNS and switches, and a UPS were all in a mobile 12U rack which we used to move between classrooms as required.

We were tasked with closing down the rack and moving it to another classroom.

Just as my colleague unplugged the rack from the room's mains socket, the whole building went dark. His face, as he stood there with the plug in hand was a picture.

Again, it was just a coincidental grid outage, nothing to do with what we were doing.

[David Hicklin]

> Again, it was just a coincidental grid outage, nothing to do with what we were doing

There seems to be an awfully large number of coincidental grid outage's, maybe similar to the million to 1 chance happening 9 times out of 10?

[Sherrie Ludwig]

There seems to be an awfully large number of coincidental grid outage's, maybe similar to the million to 1 chance happening 9 times out of 10?

Since many people are doing lots of stuff, some of it electrical, at any given time, the chance that somebody is in a position that implies causation is quite high.

[KittenHuffer]

It's not just electrical failures!

My favourite perfect timing/just coincidence/did that really happen moment was about 3 decades ago. Walking through the pedestrian shopping precinct in the middle of Portsmouth with my ex-SWMBO. the problem being that the ex-SWMBO had a near phobia about animals (ALL animals) so avoiding the flying rats (eons of pigs!) was her prime concern. And trying to pull my arm out at the shoulder when one got too close.

For a bit of fun, I quietly said that all you needed to do was say the magic words, and when asked what they were I said "Pigeon Pie! Pigeon Pie! Pigeon Pie!". And just as I finished EVERY flying rat decided as one to leave as quickly as possible! The look on ex-SWMBOs face was priceless! (And no! There were no dogs running around, or stamping sprogs, or firecrackers going off. I really didn't see anyone or anything that set them off.)

The story was repeated by the ex-SWMBO to friends and family, and years later I would occasionally catch various relatives repeating the magic words hoping for a repeat performance.

[Anonymous Coward]

Re: It's not just electrical failures!

A friend at church told the story of how he happened to be going through Kearney Nebraska during the Sandhill crane migration. Massive flocks of big birds.

Well he just had to stop along the road to get out and look awesome sight. After he slammed the car door shut he realized that was a big mistake. The startled cranes flew over his car and he heard multiple hits as the cranes jettisoned their excess weight. I think he lucked out and he came out of it clean.

[Pete 2]

In a crisis ...

> he was prime suspect

Presence of mind is good. But absence of body is better.

[Martin an gof]

Sometimes it really is your own fault

Like the time I kept a few inches of Magnesium ribbon behind after we had cleared the chemistry experiments away, then tried to light it on the one-bar electric fire which was heating what passed for a computer room in that school. Probably good for me that I (assume I) had it touching the properly-grounded metal guard at the moment it also touched the glowing heating element and tripped the power to the whole wing, three classrooms, couple of offices, changing rooms. Yes, for some reason the lights also went off, I was only 14 or so and it was a long time ago so I've never worked out exactly what happened.

And sometimes it isn't. Some years ago we were sitting down to Christmas dinner. Mother-in-law was with us this year. In our house Christmas dinner is sometimes in the evening rather than at mid day, so by now it was dark outside. "Pop" and all the lights went out. Bleeping from the UPS in the computer cupboard. The children just carried on eating, conversation didn't even falter. One of them got up to fetch candles. Mother in law staring at me as if it was my fault. To be fair, I'd just done some major work on the electricals in the house, but a quick glance out of the window proved the whole village was off, and it stayed off for a good few hours until the electricity board managed to plug a generator in. Turns out a tree had brought down a large section of overhead power line.

Not at all an unusual occurrence where we are, hence the children's non-reaction. I think M-i-L blames me to this day though, or at least blames "us" for living in an area where power cuts are part of everyday life.

M.

[Brave Coward]

You get me standing in the dark, now.

I can't see why a country having produced Wuthering Heights shouldn't pop the occasional Mothering Lows.

[tweell]

It works for data too!

I was setting up banks of IP cameras at a remote site. For ease of control and because the network switches were already full up, I had installed and programmed a separate switch for the purpose. When I patched that switch to the main router, the outgoing data link for the facility dropped. Screams of rage, cries of panic and demands to fix what I had broken rained down. My claims of innocence were ignored. Fearing for my life, I got out of there as fast as I could.

A backhoe operator working a kilometer away had dug up the telecom fiber cable.

[DarkConvict]

Ran out of IPs

I was placement student, with 2 years acamedic experience in mid 2000's, and no knowledge of networking. I was tasked with setting up a private network at a defence contractor, so they could run a self contained test, on their own network which also required their own windows server, DNS, user management, database, and a few virtual computers etc. physically there was 1 switch and 4 real machines. I was prewarned to make sure not to connext the DNS to the main network to avoid IP conflicts, as whoever did it last year brought down the company network.

I did it, set it up and low and behold everyone starting to get connection issues due to a lack of IPs addresses. It turns out the computer had just brought online a load of new computers for hot desking and it was nothing to do with my 30+ virtual computers on its own network, or my DNS trying to issues IPs out to the main company intranet.

[David in NL Canada]

Same... same...

As a 33-year IT veteran now, I have has a couple of similar situations of horrible coincidence.

The most similar was one where not the customer, but I, was panicked thinking I was somehow the cause. Happened about 10 years ago at a large government client data centre - this was a DC that housed the majority of IT systems for an entire Canadian province, including significant number of centralized healthcare systems.... so a pretty serious environment. (unlike the typical back-hand comment of "its not connected to a heart machine" used to minimize the fear of criticality... these systems were, indeed, connected in some way to heart machines!

Anyway - I was doing a pretty simple job, installing some new server platforms inside the DC, dealing with 4 individual racks. Nothing major in a DC with maybe 200 or 300 racks of equipment. I had just completed by secure access process to enter the DC, documenting the individual racks I would be working with, etc... (everything but the rubber-glove security checks). I entered the DC, and this was the start of day 2 after doing physical installs the day before - and this morning was the system startup. These were relatively new IBM systems at the time, and the installation process happened during initial power-on.... so my process was to first connect the "crash-cart" console, and power them on.

I walked to the first 2 racks and powered on the devices and other than the very loud startup fans, all was normal. I walked to the second pair of racks to do the same... and on cue: PRESS POWER BUTTON.... *** everything in the DC goes dark and quiet... emergency lights blinked on along the outside walls for a dim, somewhat yellow lighting.

The DC was down. I had just powered on a large server chassis. It was me!!!!! I DID THIS!

I think my heart stopped for a few seconds before I finally got up and walked to the door. As I opened the door, the security guard was standing there, looking at me... and simple said (deadpan, without emotion) - WHAT DID YOU DO?

I was lost for words and as I was about to explain that what I did SHOULD not have been an issue... it was approved, part of the power design, etc... I hear a couple of the facilities folks rushing over saying something about a power issue.

In the end - it was a large bird nest that cause a weird power outage that coincided with a failure of a transfer switch in the DC. The grid had died, the UPS had cut in (unknown to me in the DC as the entire area was UPS powered), but when the generator automatically kicked in, transfer of power failed "half way" (I'm no electrical engineer if you cant tell) - caused the system to believe there was stable power from the generator (generator was running fine), and the UPS stopped providing power... but the transfer of the ACTUAL power was not completed.... so the UPS transferred to a "switched-off" incoming power service. (this is my recall and am sure there are some screwed up details).

Power was returned relatively quickly.... and I wasnt at fault. But was about the worst scare I ever had in my career thinking I didn't just make a mistake.... but royally screwed up!

[azander]

Not mine but my spouse's

While we were college, I was the student admin for the campus micro computers and PCs. My spouse was working in the computer center officde as support. It was a slow day, so was working on a program in pascal. This was back in the late 80's. We had 1 Novell server (4.x I believe) on a UPS, and 50 workstations in one building, and across campus another 20 or so PC's shared by the staff and educators.

My spouse hit compile and the lights went out. Came back on. Went out. Came back on. Went out. Finally it stayed out. I was in the building with the 50 PC workstations but in one of my classes. The timing was just right that we were about to leave class when this happened. I made a quick run across the building to the Novell Server. Too late. The UPS was fried (smoking!) We lost one of the 2 Raid Drives (early Raid 5) and the second drive was in bad shape.

Our boss, the Computer center Dean looked up and watches every system across campus go dark, EXCEPT his office and the 2 mainframes. They were on really good UPSs and power line conditioners. He looks over at my spouse and asked what they did. Talk about a nightmare. Luckily he was joking. My spouse didn't know that, at the time. That power outage took out a good portion of the Western US.

Spouse and I rebuild the Novell Server over that weekend. the Dean was rather surprised at the speed were were able to do it with. I just said "Backups are a good thing."

[Timo]

In one child's first few weeks at college found that the power was off in their dorm room. She immediately connected the dots to the orders for "no extension cords for refrigerators", which they were using in order to have it in the correct place in the room. She and roommate went to hardware store, bought heavy cord or something to match the stated requirements, and went back and plugged it in with no luck. Called me in a panic that they thought they had burned the world down.

At that point a google search uncovered that the power was out in half of the campus. They had skipped over any sort of checks, like were any other rooms without power, or floors, or buildings, or blocks?

[Gavsky]

Wax on, Switch OFF...

Ah, yes Sensei - my mantra is: is it plugged-in? (no, properly). Is the socket switch ON? (not in all parts of the world). Is the power lead seated? Look through the Blinds of Mystic Awakening - what of the neighbours & street lights?...

[GlenP]

Re: Wax on, Switch OFF...

is it plugged-in? (no, properly).

Doing telephone support many years ago I asked that, user replied, "Yes, I've checked it!" I then asked them to unplug it and plug it back in firmly, which they then said they'd done.

Their boss wasn't happy about a £120 bill from IBM support for the engineer visit to push the power cable back into the socket on the PC but I had warned them.

[Andy A]

Re: Wax on, Switch OFF...

One of my standard questions: "Can you check that the cables are properly in the back of the machine? Maybe the cleaners have dislodged something." Knowing that the last visit of a cleaner to that office was over a decade ago.

[Thereneverwasaplot]

Re: Wax on, Switch OFF...

My favourite. "sometimes there's a dust build up, could you please unplug it, blow on it and plug it back in again. Works particularly well for network cables too.

[Gavsky]

Ah, the old perfectly-timed, imperfectly-timed occurrence! The accusing looks of 'YOU did this!' from the less technically-minded & fact-aware are hard to take. You couldn't make them-up but that works in our favour, too.

Consider the times they all look at you in wonder, with new respect - because 'YOU fixed it!'. "Nobody else could do it after four days, but he just came in & did something - now it works! What did you do?! Our tech guys are morons!".

"Oh, it was erm - nothing really. Fresh pair of eyes; some useful IT experience, y'know?" I relish those times, from home & work!

[Niek Jongerius]

Post hoc ergo propter hoc.

[david 12]

Ah yes, the classical logical fallacy: "The reason I know Latin is because I went to grammar school"

[Anonymous Coward]

it doesn't make any sense in English either

[TooOldForThisSh*t]

Happened To Me Too

Just two days ago. Bad outlet in the kitchen and I was checking it with a voltmeter when the whole house went dark. Panic City! What did I do? Then I looked down the street to see the traffic light at the corner was off too. Quite a relief. Power was out for the whole town for 2 hours as a squirrel met his (or her) end at the nearby power substation :(

[chivo243]

Anyone worth their salt

Anyone worth their salt working in IT has a horror story about UPSs. And firmware updates gone sideways~

[A.P. Veening]

Re: Anyone worth their salt

As long as those firmware updates only go sideways, there is no real problem. The real problems start when those firmware updates go on a rollercoaster ride.

[Boris the Cockroach]

The power went

out where I was working as an apprentice, it was strange because the 3 of us working there became deaf too at the exact same time we threw a power switch and all the lights went out

Plus the big electrolytic capacitor vanished at the same time , it was a perplexing mystery to all present.

Apart from our instructor who learned the value of not leaving a room with 3 17 yr old apprentices , a power supply and a big fat electrolytic in it.

[_Absinthe_]

Dot Matrix Mayhem

About 37yrs-ish ago I was staying with my uncle for half of the school holidays, as I usually did (he was my idol haha). He was a self-employed software dev one-man-band; didn't want to work for anyone else, but also didn't want the responsibility of anyone working for him either. Very successful chap and very friendly & personable, just preferred to do his own thing workwise.

One day, he got a call to deal with some aspect of the payroll system at Tyco Toys, and took me with him. So, we pulled up in his TVR, a rare 420SE which he gave me in '09 a few months before he died. Top down, sun blazing, music going, arriving at a toy company with 10yr old me grinning like a frigging maniac.

After some troubleshooting he decided he needed to inspect their main dot matrix printer. Obviously this is being recalled through the eyes & ears of a 10yr old, but I swear that thing was a bloody behemoth. It was in its own room (with windows into the office area) and encased in a full-on soundproof enclosure. It was in the middle of a job but so that he could crack on as soon as the job ended, he removed the top panel and the noise was deafening. I remember people at their desks looking over at us and then a couple seconds later, lights out, all equipment dead; the sort of silence a necromancer would be proud of.

Needless to say, I shat myself, my uncle laughed and cursed about unfortunate coincidences. We left and had to go back the next day after they'd got the power sorted. Called at the local pub for tea on the way home, god I loved that guy :)

[C R Mudgeon]

Re: Dot Matrix Mayhem

Are you sure it was dot-matrix? From the description, it sounds more like an old-fashioned mainframe line printer -- positively huge by today's standards, noisy, and fast.

Here's a picture of a typical one, with a person in the image for scale.

I could well be wrong of course. You were there; I'm just speculating.

[NetMage]

Re: Dot Matrix Mayhem

Some line printers were also dot matrix printers - they just had a really long print head! (We had one in its own room in its own sound deadening cover - it had a dedicated fiber line running from the room next door where the server was.)

[C R Mudgeon]

Re: Dot Matrix Mayhem

I didn't know such devices existed. Thanks.

[Excused Boots]

Re: Dot Matrix Mayhem

"Needless to say, I shat myself, my uncle laughed and cursed about unfortunate coincidences.”

Well we all know how often million to one events happen, don’t we?

[DS999]

Somewhere

There is someone who did "something" electrical at the exact moment they were hit by the big Northeast US blackout in August 2003, and probably got blamed until it was determined how widespread it was. Even then there were probably still a few people who looked at that someone suspiciously from then on.

[David Hicklin]

Re: Somewhere

A case of the straw that broke the camels back ? Plugging that one extra appliance in causing a cascade failure all the way up the supply chain...

[DS999]

It was probably a squirrel

East of the Rockies in the US, squirrels cause more outages than all other causes combined.

[Stuart Castle]

One of our users in one of our rooms reported that he was given a shock everytime he touched an electrical appliance (even a switch or socket) in that room. Being concerned there was a problem and being unable to find it, having unplugged all the electrical equipment and him still having the problem, we got an electrician in. He checked all the sockets (over 30), and all the light switches and fittings. He tested each with every test he could, and they all passed. He was confused, until he realised the user was wearing polyester.

Yes, the user almost exclusively wore polyester and tended to shuffle while walking, so over the several hours he was in the room (he tended to walk), he was building a large static charge.

We asked him to change his clothes, and the problem went away.

[Andy A]

The place where I started work was built from metal partitions about 8' high, with windows in the top half. Someone coughed up cash to replace the ancient carpet tiles with nice new carpet. The static problems affected everyone, no matter what their clothing was made of. I took to carrying a pencil with which to touch each door before I suffered an inevitable spark as I reached for its handle.

[David Hicklin]

Ah yes, static carpet. I working in an office like that where you would invariably get a shock from the door handles on the way out, I think they sprayed the carpet with something periodically to keep it at bay.

[Tim99]

Perhaps I'm gifted?

I worked from home and managed to take down the electricity for our entire block with a Nokia 8200. I plugged it into my desktop UPS and there was an immediate very loud bang and everything including the PC on the UPS went dark. Not a major problem I thought, I'll disconnect the UPS, and reset the breaker. Still no power. The next door neighbour saw me checking the outside breaker, and said that their power was out too. Checking showed that all 26 units in the block were down. I called the electrician who did some checking. The main fuse to my unit was blown, and the main fuse to the block. I told him what had happened, he came inside and we both saw that the terminals to the phone charger were fused into the UPS socket. He then called the power company, who replaced the blocks' fuse... I was expecting trouble but the power company electrician said "it was unexpected", and it would be a good story to tell his mates. The phone and UPS were totalled though, the PC needed a new fuse.

[DanCoganwritesthrillers]

Burn the witch!

[Richard Pennington 1]

I was recently on a course where the instructor was explaining something using a fancy big display/touchscreen setup, when suddenly all the lights went out, including the display. A few seconds later, the power cut back in, but the display was no longer working. The class was treated to my observation "Have you tried turning it off and turning it on again?".

It turned out that it needed the instructor to log in again and to restart the sequence from the top.