Nobody does DR tests to survive lightning striking twice Lightning may never strike twice, but each and every Friday The Register runs another edition of On Call, our reader-contributed tales of tech support gigs that did not spark joy. This week, meet a reader we'll Regomize as "Karl" who once worked in the services limb of an enormous and venerable three-letter corporation's …
Not paying for the overtime? Wow, that is really shitty.
In a previous role I provided out-of-hours and on-call support and the official company policy was to book extra work as overtime in the first instance, or take time off in lieu (TOIL). After booking my first batch of out-of-hours overtime my manager said there wasn't a budget for this so I'd have to take it as TOIL. I checked for the company's policy document on this to see how much TOIL to take - the equivalent of the hours worked as overtime, or the equivalent of the hours worked at the overtime rate, which was at time and a half. There was nothing in the document to specify what TOIL to take, so I booked it as equivalent to the overtime rate.
Naturally my manager queried this, so I explained that if the company were unwilling to pay me, say, four and a half hours pay for three hours overtime at time and a half, then they would need to give me four and a half hours of time off in lieu. I referred him to the policy document and said it wasn't detailed in there so I'd interpreted it at face value, but that I was more than happy to go and have a chat with my union representative to get their take on it. My manager said it was fine and that's how I booked my TOIL for over five years.
I did the same thing and after a while I pointed out that because of the amount of TOIL I was getting (same as you, time and a half, double at weekends), it'd be worth my while doing some small bits and pieces on the side for the clients. My boss pointed out that I can't do that because there is a clause in my contract prohibiting me from performing any work for clients on the side...so I read the clause and the clause in the contract read "...within the scope of works as set out in the client contract...". Since the sales team and legal team were very keen to stay ahead of any edge case complaints, they made the scope of work for each client extremely specific and classified anything outside of the extremely specific scope as "best endeavors", as in, they will assist with it, but it isn't deemed as "in scope" therefore if anything goes wrong, no claim can be made against the company regarding poor service, negligence etc...but here is the nice bit, all the really valuable work (server side admin, hardware upgrades, Linux stuff etc etc...basically anything that wasn't desktop support or really easy), was outside of the scope...I pointed this out to my boss..."yeah, but the scope of work in the customer contracts is really specific, there's a ton of work I can take on during TOIL and I can make a killing without having to take money out of the business"...well he went white as a fucking sheet, the lawyers stepped in and insisted on giving me a 30% payrise and they cancelled my acces to TOIL in favour of paying me 2.5x for each overtime hour I did...and yeah, you bet your ass I took the piss...we had a three shift pattern, 7am til 2pm, 9am to 5pm, and 11am to 7pm...you can bet your ass I never took the 9 to 5 slot...I took the 7 til 2 slot and booked 3 hours of overtime a day, 5 days a week for years.
Yes, well I ultimately ended up in a senior position at one of their biggest clients which I was able to take based on other oddly specific clauses in my contract (and the client contract) and the wiggle room afforded in my change of status when In handed my notice in.
Essentially, because of bean counting and insurance bullshit, they wanted to scrimp on liability while someone was on notice. Essentially when on notice you were no longer deemed to be an employee. Which had the effect of technically cancelling out a load of clauses...including non-competes, working for clients etc etc.
To make it iron clad, I was never hired by the UK arm of this client. They hired me through the Swiss arm, which never had a contract with these clowns and is where all the employees of the client were placed.
The icing on the cake was I got to become the European Head of IT....so it was my job to cancel the contract with my former employer.
When they showed up for a meeting and realised I was the new head of IT. They hit the fucking roof. I think the penny finally dropped at that point because they didn't lawyer up and walked out looking absolutely defeated. One of the directors found it funny though and had the balls to shake my hand and say "well done, you're someone else's ballache now, don't fuck it up". They went bust a year later and the "hot headed twat" director ended up working at Rackspace as a third line tech. The other two (less unreasonable ones) started a new firm and they are still going I think.
These days I am freelance, yoinking contracts on a regular basis, fighting contractual injustice...like Batman, only nerdier.
You'd be amazed at how shitty the average MSP contract is...remember kids, if you spend too long trying to cover your ass you might forget to put a helmet on!
We had a weird problem with some kit on another continent and we had to send a development engineer out to fix it instead of one of the normal support team. We estimated a 3 day trip and he was gone over 2 weeks and left a very happy customer. When he put in his expenses his engineering manager wouldn't authorise a lot of the expenses cos they were outside policy. E.g. he'd only packed enough clothes for a couple of days so he'd had his clothes laundered a few time by the hotel and bought some other odds and sods he hadn't packed. The customer PM went ballistic and read the riot act to the engineering manager, who signed off the exes. The engineering boss, in a fit of contrition, sent some flowers to the engineer's wife to apologise for him being away so long. She sent them back to him with something pretty unpleasant written across the card.
Still grates on me... Y2K and HR says "All hands on deck January 1st, just in case. We will comp you an extra vacation day."
Nothing happened on Y2K, except we sat around the office thinking about missing the New Year's Day holiday.
Three months later the company downsized and none of the promised extra vacation days were paid out. That was early in my career, and lessons were learned.
I was lucky to have rather better management. Because of all the pre-Y2K prep work we had done -- about a year and a quarter of it -- we only had to have 6 people on deck for the Dec 31st 10PM - Jan 1st 6AM shift, and we had no significant problems, and we were properly paid.
If my management had been better than they were, we'd have started in 1998, but as it was, I made more money on Y2K-related overtime in 1999 than I made on my regular salary!
I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there) during a downpour, decommissioning a load of customer kit to relocate it to another DC...the roof caved in and there was water gushing into the datacentre everywhere...thankfully though, not on my racks, which were almost dead centre...I was there all day working on my own kit and I observed precisely zero people coming in to sort out the racks that got drenched...I had a wander over to some of the, by this time, dead racks...and I noted at least a few million quids worth of kit that was now soaked and ruined...most of which were storage arrays...so quite a few people, somewhere were having a really shitty day.
I really hated that datacentre...pretty much any datacentre in MK to be honest...it's far enough away from London to be relatively cheap, but also just far enough away for the trip to have a long boring journey...I've also never been in a nice datacentre in Milton Keynes...they're all miserable...several of them have pool tables but they all have at least 1 ball missing, cues with no tips...broken coffee machines...backed up bogs...a miserable twat on the front desk...walking around them is creepy, like walking around a derelict wreck from the 90s.
"To be honest that description of the DC extends to most of Milton Keynes in general"
Thanks, just what I needed to be reminded of while packing my bags to go and visit the family in MK!
Though Gallery MK hosts some good exhibitions and the National Museum of Computing offers many moments of nostalgia to older Commentards.
And this looks potentially interesting: https://miltonkeynesmuseum.org.uk/collections/view-our-collections/#Telephone
A good weekend to all Commentards -->
Oh fuck Croydon man...in fact that general area around Croydon as well. Sutton, Mitcham, Colliers Wood and Mordor (Morden).
Mate of mine got stabbed in the neck in Sutton for nothing (he's fine and still very much alive) and I've had the misfortune of a couple of nights out in Croydon...what a shit hole.
I can think of worse places to be (e.g. Reading)
What a lot of southern centred commentards we have here.
I see your Reading, and raise you ... Barrow in Furness. Turn off the M6 at J36 and Barrow is the end of a 40 mile cul de sac. Anon because, well there might be some Barrovians here and there's "a certain amount of risk" in criticising the town.
Reading is a fucking dump. It's a town full of commuting jobsworths...pretty much every major town along the M4 is shit. It's like Slough is ground zero and the general grubby wankiness spread up the M4...the further you get from Slough the less (bit still) shit it gets.
Now I really do not mind MKS. We have a DC there, staffed by a handful of great people.
The Stadium Hotel is great to stay at, there is an absolutely superb Turkish restaurant - Enfes - opposite (I really cannot stress how great this place is)
And lots of green cycling and running routes.
Always look forwards to a trip down south. But...
... we are moving to Slough.
Slough! Concrete boredom, the only place worse than Basingrad, as I used to refer to Basingstoke when with Global Crossing. For some reason people were not too happy with that label. No idea why :)
I used to work nor far from Slough. When the wind was in the right direction the smell from the Mars factory coupled with the stench of sewage would waft over our office like a well-brewed fart. To this day even the thought of a Mars bar makes me wretch.
Not the cliché MK is dull response surely?
I moved there for a couple of years 30 years ago, am still here. It’s great: drive at NSL speeds and get across town on the grid roads in < 15 minutes unlike the peak time crawl whenever I go back to what I still call home. As mentioned in other replies it’s got Bletchley Park, MK museum (fab), the theatre, The Gallery with its well worth a visit cafe too. Everywhere is within walking distance of huge green open spaces and linear parks, the same time in any direction is a great bike ride in open countryside. Pubs, nightlife, cafes, decent shopping centre, kart track, ye olde villages within and close by outside, as well as the old towns, and London 40 minutes away. Etc etc etc. Quality of life here is great.
But yeah - roundabouts and concrete cows. Yawn.
NSL and roundabouts are the best thing about MK's grid. Although getting out of MK onto the M1 in the afternoons (during rush hour) used to be the absolute pits.
At some point I gave up with the horridness that was the last 400 yards or so from the H5/H6 roundabout onto the M1 junction and then the crawling-slow merge onto the actual motorway, would zip past the John Lewis warehouse to the road towards Bedford, and then either merge onto the M1 there (if you were lucky with light timing that was an absolute breeze) or chance the construction work (t'was the time when the A421 was being dualled between the M1 and Bedford) to skip over to the A1(M).
These days to get to Cambs, it literally is just a case of going straight ahead through MK onto the A421, then over to the A1(M). It avoids the mess of Northampton/Wellingborough.
"But yeah - roundabouts and concrete cows."
As only an occasional visitor to MK I find the roundabouts a bit difficult to remember. Here in NL they tend to put pieces of public art on them. So you recognise them by the archeologically inspired sculpture, the giant trowel, mega-tulips, etc. Makes the drive easier and more fun. But admittedly in MK you would need a v substantial arts budget to plant something on every roundabout! Just as well we've got satnav now.
I was once at the TalkTalk colo datacentre in Milton Keynes (don't know if its still there)
TalkTalk and Milton Keynes are still with us, sad to say...
II've also never been in a nice datacentre in Milton Keynes.
Datacentre and Milton Keynes should set off very loud alarms warning you that there isn't going to be any niceness.
Donkeys years sinceupon, when 8 inch disks were still a thing, a work chum was despatched from our Luton office to take a raft of bits to a water treatment works in Diss, Norfolk, which had been on the fizzy end of a lightning strike.
I got involved when he rang in to say that he'd found the problem - an 8 inch floppy unit had been fried and he'd replaced it. Everything was then fine he said, until the second lightning strike fritzed the replacement.
Guess who got stuck with a panic run to Diss and back.
I think we did something about power arrangements but I've been asleep a few times since then and I'm buggered if I can remember what.
Sadly my old chum is no longer with us but I do remember him in the occasional lightning storms we get here.
My memory may be playing up, but from what is surfacing I think the old adage about lightning never striking the same place twice is quite wrong, when lightning strikes it ionises the air and actually makes it easier (thus more likely) that you'll get another hit in roughly the same area.
I'm fairly sure there is/has been research into how to utilise that to direct more lighting to where you want it hitting thus building a better lightning rod by effectively creating that ionised pathway through the air before the first natural strike.
Correct, but not the general reason why lightning hits the same place twice, since few strikes hit twice in the second or so that the ion channel exists. (Or more accurately, most lightning strikes actually consist of more than one discharge that happens in rapid succession while that channel exists)
The real reason lightning tends to hit the same place more than once is the same reason your finger and the doorknob get more static discharges than anywhere else in your house. Weather patterns exist, just like you shuffling from your bed to the kitchen, which tend to accumulate similar charges, and there are only so many physical locations that fall under that pattern. Some of them are simply closer and of more appropriate materials/shapes. Ceramic coffee cup? Insulators by definition aren't good electrical conductors. Brass doorknob? Excellent.
Similarly a flat grassy field has a much lower ability to concentrate a charge than a house with a chimney covered in sharp edge. (Sharp edges and points are better charge concentrators)
So, until there is some kind of new construction that changes becomes a more attractive lightning rod or the old lightning rod finally gives way to a couple of gigawatts of power or climate change alters the weather patterns, lightning will keep striking the same place.
Hence that one house in every town covered in lightning rods.
And the little detail that lightning rods tend to have a sharp point. The idea is that they'll form a locally very strong field at the point, and thus draw charge into the rod - so in the vicinity of the rod, the air is less charged, and so the chance - both of a lightning strike, and of it hitting your building if it does happen - is reduced.
What I can tell you from hard experience, is that serial (all RS232 back then) links across large buildings, or even worse between buildings, do not mix happily with the very high voltage gradients in the local "earth" if there's a nearby lightning strike. Icon approximates what it did to our equipment :-(
> the old adage about lightning never striking the same place twice is quite wrong
And not just for recent ionization. As others suggest, the reasons lightning struck once are often still there after the strike, unless the tree/rod/manager vaporized or fell.
Much the same reason you go to Pat's Pub every week. It is nearby, it is easy.
In my youth my house was in sight of a 1,000+ foot (many meters) broadcast tower. We could see it get hit multiple times per storm.
So no large shock that we have two tales of double-hitters. All you need is a storm-line long enough to give time for a first repair before a second strike. (If 2 strikes happen "at the same time", bang--bang, for billing purposes that's just one hit; though often the potential is dropped by the first hit so will be some time to recharge.)
Icon represents Zeus, Lightning-guy.
We had something similar happen to a facility on the South Coast of the uK.
When I went over for an unrelated issue, I walked past a couple of hundred DEC VT100 terminals stacked 5 high in the corridor! A couple of days earlier, a lightning strike hit the building and shorted every terminal in the place!
DEC VT100 being "advanced" technology that the US didn't want falling into "enemy" hands, they had to be officially disposed of and witnessed and certified by a UK government representative.
Luckily, they was an scrapyard next door and the terminals were just pushed on trollies to the compactor of the neighbour and stuck in there, then, with the witness there, the terminals were counted and the compactor turned on. At the end, we got a signed certificate that the terminals had been properly destroyed...
Another time, working at a Naval Dockyard, one of the suppliers had tried to take shortcuts and replaced the heavily shielded DEC VT220 terminals on the purchase order with standard terminals... A few weeks later, a US destroyer tested its radar, whilst still in port. Queue a couple of hundred dead terminals and a very red faced, and out of pocket, supplier.
DEC VTs were usually fairly robust.
I'd left a VT220 in the computer room when I went on holiday as I'd been doing some tasks that were better on a VDU than the console. Unfortunately the computer room burnt down while I was away (no connection of course - electrical fault in the workshop next door).
The two VAXen, an 11/750 and 11/780, were a total loss of course.
The terminal was pretty much melted but for a laugh we tried plugging it in from a safe distance - yes, it came up VT220 OK, albeit somewhat faintly.
At least the guy that made the server rooms didn't weld the wrist straps ground to the lightning rod's ground... like someone I know...
The friend of mine that noticed this averted a Benjamin Franklin experience en masse, and got in charge of the grounding of the building for his troubles.
"At least the guy that made the server rooms didn't weld the wrist straps ground to the lightning rod's ground..."
You actually want to be at the same potential as the lightning rod's ground. If you're not, bad things can happen. That's why communication wiring has an intersystem bonding termination that ties all the grounds to a common point. Granted, running a dedicated wrist strap ground wire back to the ground rod can create some other issues, so typically you grab the closest grounding conductor.
If you're grounded to a different "ground" and lightning hits the building, now your wrist strap could be at a potential that's hundreds or thousands of volts different than the ground on your equipment. That would lead to a very bad day. The problem is that the earth is a relatively poor conductor.
If you're in the US and want to dive down a rabbit hole, read up on NEC chapter 250 on grounding and bonding. That's one of the key areas that low voltage technicians need to know for their license.
Lightning is a strange beast. A lot of it is RF. If you ground a lightning rod with a wire that makes a sharp turn, the lightning will go straight. The wire used to ground a lightning rod is a coarsely braided conductor more than an inch in diameter. All of the building gutters should be connected to the ground wires with a 1 inch wide copper strip. The actual ground to earth should be a 12 foot copper rod driven into the ground, preferably at three places around the house. The ground system should be wired to the electrical ground (that, by code, is a 6 foot galvanized pipe).
Remember, lightning rods attract lightning so you'd better be ready when lightning strikes.
I worked for the Home Office in the late 80s and was sent of to Fire Service College in Moreton-in-March in the midlands for several months - I got to go home at weekends, but it was a twohour drive each way, so I got home late on Friday night and had to travel back down on Sunday afternoon. I could claim for the travel time, so no complaints there.
I was staying in a lovely little pub in the village - horse brasses, wood fires, good beer etc. Civil Service rules said that if you were working with somebody of a higher grade and having work discussions etc you could claim the expenses rate of the higher grade. This applied to me, as I was working with somebody two grades higher than me, so the difference was significant.
Several months later I put in my expenses claim and it was the highest one they'd ever seen from somebody on my grade and they baulked at paying it. They tried to say that I should have stayed in dormitory accommodation at the college, but I said that if I stayed on site I wouldeffectively be on duty 24 hours a day, andthere was no rule that stated that I had to stay on site, so they had to pay out.
A couple of months later they rolled out a new rule that if there was accommodation on site, you could either use that, or only be able to claim 50% of the usual rate if staying off site.
It can vary radically from organization to organization. In one organization I worked at, there were plenty of spares for most things (no spare mainframe, but we had a hot-site agreement with another org, and no spare SAN, but spare discs, spare channel controllers, terminals, network cards, PCs, monitors, non-mainframe-printers and such). In a Government-owned organization which we were physically- and networkly-linked to, I was troubleshooting a user's PC and was told by their Help Desk that they'd have to get management approval for a replacement CAT-5 patch cable, a thing which by all rights should be considered "popcorn".
If the datacenter is mostly mainframes "frying every ethernet port in the building" might not amount to more than a few blades worth on the switches. Customers with mainframes have a lot of money to spend, keeping a fully configured core switch plus spare blades doesn't seem too crazy to me.
There's an old adage that "Nobody ever got fired for buying IBM", and a lot of that was because when things did go wrong there was a seemingly bottomless pit of spare parts they could magic up at short notice. Don't know if that's true any more though.
Yes. Mid 1980s. I was working on contract at place and we had the production AS400 that was fried by lightening. It was replaced by IBM well within the Service Delivery contract time (the warehouse was literally at the other end of the street).
IBM loaded the OS and we were right to go with the restore, blank tape, last weeks backup, blank tape etc etc
Luckily I had been in on teh weekend and taken a full copy of the data as we needed to some performance testing on the software for the next stage we were releasing.
Come in around lunchtime & people are running around like headless chickens - I reach into my bottom drawer and grab my backup which is then loaded (I knew it worked as I had installed the data on the performance testing machine). I was taken out for a very long, free lunch.
It was a totally unimportant application - it held enrolments and marks for the year 12 exit exams (UK = A levels, USA think SAT)
IBM don't always have the parts to throw at a problem.
Early in my programming career I was involved in a project using an IBM 360/40. Our company made money by running the cheapest kit possible. At the insurance company across the road, the programmers had full use of a 370/158.
Our compile jobs were always pushed to the back of the queue. One week I submitted 30 compiles and only two got processed.
Usual operating shifts finished late on Friday evenings. So we decided that we programmers would come in on the Saturday to clear the backlog. Some had been operators earlier in their careers, so we all knew the drill.
We powered up the system (who remembers it being called IPL?) and started jobs.
20 minutes later the system stopped with a Parity error. Restart then.
10 minutes later the system stopped with a Parity error.
We reached the point where the system would not even IPL. Obviously something was warming up and then failing.
No engineering cover for weekends, so we rang the higher-ups, who could then get things moving for Monday.
IBM found a failed power supply. One was rushed in, and the mainframe was running by mid afternoon.
Of course we got NO jobs through that week, so we trailed in on Saturday again.
IPL - 20 minutes - Parity Check.
Ring higher-ups, head home.
This time we were informed that the replacement PSU for the ancient beast was the last one in the whole of Europe. System back up by Tuesday.
The operators were informed that under NO CIRCUMSTANCES were they to power down the system that Friday evening.
We cleared our backlog in time for a couple of pints. The mainframe was left idling.
About a month later the 360 model 40 was replaced - with a 360 model 50 !
... we worked for the same three-letter company, and I have a story which I can't decide if should be featured under "On Call" or "Who, me?".
The annual DR test was supposed to be held at the end of a long, hot Spanish summer. We were supposed to simulate a power loss at the main site, wait until the generators kicked in, and then test the swing to the secondary site. Preparations for te big date took the two previous weeks, every runbook/manual/contact was double checked, and on D(R) day, the mains switch was flipped, and power was cut to the Production data center.
We remained connected for about 10 minutes, under UPS power, more than enough time for the generators to kick in and take the load. Except it didn't happen. We lost connectivity from outside the data center, and decided to call the hands & eyes team. They picked up the mobile phone (the landline was as dead as the network links) and we asked them to go to the back of the building and check the generators. 5 minutes after, they called back, stating the generators were dead and closing with an urging "ahora vuelvo, voy a por gasóleo a la estación de servicio!" ("be right back, I'll go get some diesel at the gas station!").
We had our phone on speaker, and the moment the H&E guy hung the phone, we busted out laughing until tears came out. We couldn't believe it: the only thing missing from the check list was to verify the amount of diesel in the generators' tanks. A quick note was taken on the DR runbook, we asked another member of the H&E team to reconnect the external power, and called the DR exercise a failure, to be rescheduled a month later.
So the disaster can be recovered but only if two weeks have been spent preparing for it.
Seriously - you could only call the exercise a failure if you didn't learn to have regular checks made on the generator fuel. DR exercises are to be learned so that when the D happens for real so does the R.
> .. the only thing missing from the check list was to verify the amount of diesel in the generators' tanks ..
And check for water contamination collecting at the bottom of the tank. And do your DR test more than once a year.
Where sites did monthly generator tests, but neglected to check the fuel level. With the result that when a real disaster happened and the generator had to power everything it didn't last nearly as long as had been expected due to the monthly tests running down the fuel and no one checking/refilling it.
> ...when a real disaster happened and the generator had to power everything it didn't last nearly as long as had been expected due to the monthly tests running down the fuel...
My school did not make that mistake.
Instead they assumed the fuel trucks would be running. This was not a deep snow area--- at worst they might wait a day.
A once-a-century blizzard (the second one that decade) came through and overwhelmed the snowplows. Few even tried to defy Governor's Orders to stay home. The fuel trucks actually did run the best they could, following the few plows, because many homes were on fuel oil and you don't want old ladies found frozen in their homes.
The bulk of the school was also crying for heating oil before water-pipes froze.
The IT staffer snowed-in at the datacenter saw the generator start and run fine, then the oil tank dropped and dropped for 3 days before it quit.
The next year there was a MAJOR rebuilding. Bigger tank, a new pipeline and street-gas generator on the side, and do-over of the legacy power distribution. Cut-over to the new breakers and BLAMMM! Some forgotten low-bid contractor had done something dumb not noted in the new job's bidding. I think Chuck had to fly halfway across the country for more of the special breakers to be made.
I'm not an expert on hardware or power supplier or anything but: I would have thought that if you are tasked with designing a power conditioner, you would designe it to survive a near-by lighting strike or at minimum, self-destruct to prevent damage to the ethernet switches and sundry other hardware.
ps: Never ever do unpaid overtime.
No plan survives contact with reality.
Moved into new (to us) building in the autumn. Returned from Christmas hols to find a lightening strike had taken out the thyristors in the UPS. The main building power was OK and so was all the equipment. We ran without the benefit of the interrupted UPS for many months. It's not always DNS; sometimes it's the UPS.
Well, an UPS saved our bacon once. Mountaintop, rack with some very specialized and expensive hardware, with a small 2k VA UPS protecting it. And that UPS had a feature to run the network cable through it too, to protect it against surges, so we did ("why not?").
Long story short, some time later lightning struck the building, a surge apparently traveled along the Ethernet cable(s) doing quite some damage, but our rack was saved (at the expense of the UPS, which was fried). Very fond of this UPS, it saved us a prohibitive (for us) amount of money. RIP, little black box.
Fair enough, but the sheer amount of power dumped into a system by a lightening strike is enormous. Attempting to have a solution that can cope with that would be stupidly expensive and whatever you did, nature would probably at some point conspire to ‘think you have engineered against any contingency, well try this…..’!
And having sacrificial devices in the chain which ‘should’ lay down their lives to protect….., is fairly normal, and probably mostly works. Well until nature decides to throw a curve ball and…..!
It is absolutely impossible to 100% protect against any and all issues to do with lightning, lightning really doesn’t play fair! At best you try to mitigate against most issues, and the more you are prepared to spend (or waste, as the bean-counters see it as), the closer you get to 100% reliability on an asymptotic scale, so you can never actually get there.
On the other hand, your point about unpaid overtime, is really spot on.
When your backup needs another backup but your primary got fried in the first strike, you naturally move to the second and so on.
However, when the owner doesn't understand I.T and the need for any backups because he owns the building, it's a really tough uphill struggle to get anything done, let alone a backup schedule and regime.
So, the workaround?
An on site tape backup, a daily removable database backup (thats what made the money), an off site hdd that only came in on Thursdays, a random backup that was done twice monthya d went elsewhere. Plus I shoved a separate server elsewhere in the building, think it was in a loft space, inside a modified waterproof cabinet. That had a dedicated ups, various surge protectors, plus I could remote access over, a, wait for it, 56.6k modem.
Oh the heady days.....
The seller of lightning rods arrived just ahead of the storm. He came along the street of Green Town, Illinois, in the late cloudy October day, sneaking glances over his shoulder. Somewhere not so far back, vast lightnings stomped the earth. Somewhere, a storm like a great beast with terrible teeth could not be denied.
Thanks Ray Bradbury. Here's a pint for you, wherever you are
Had a customer long ago back in the NetWare days who got hit by lightning. Phone system was destroyed along with melted thin-net ethernet cables. Compaq server and desktops all survived but the cheap off-brand clones they bought elsewhere all had fried motherboards. Needless to say they bought replacement Compaq's from us :)
I recently had to replace 2 network switches and use the backup link for a site that experienced a near miss. It appears that the strike caused a surge in the existing link (ethernet cable) that blew up the network switches at each end. Fortunately, when we ran the link cable, we ran 2 in case there were issues with one.
'Nothing stops lightning, and no one wins but the lawyers'
The trick is to get the two to meet.
Venerable Clarkson? Venerated by whom? I concede he has a peculiar imagination when writing about your Royal Family which apparently hasn't been entirely to their liking.
In any case I will try to avoid driving in the Cotswolds as I don't imagine he has many careful driving awards.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
