Cleaner ignored 'do not use tap' sign, destroyed phone systems ... and the entire building Well, look at that: the clock says it's Friday morning so it must be time for a new installment of On Call, The Register's weekly column describing readers' triumphs over adversity, mendacity, and stupidity. This week, a couple of stories we collected during 2022 that describe nature's most terrible rivalry: the coming …
I had an issue with mainframe disk failures, my maintainer was saying it was environmental, I was adamant that nothing had changed but decided to get the floor and ceiling voids cleaned to stop the argument
The cleaners found out that somebody had accidentally drilled a hole in the DC floor from below, and never told us, there was still the remains of the pile of concrete dust that my aircon system had been circulating for several months. It took a we had a couple of final drive failures before returning to a normal level of failures.
At a company I used to work at many years ago now, a fire alarm test at a datacenter in a capital city somewhere in continental Europe went horribly wrong, accidentally triggering the release of whatever (presumably outdated) extinguishing agent they were using. All the equipment in the room was covered in a microscopically fine coating of dust. While some initial disaster recovery was done immediately (after scheduling the most emergency power outage I've ever witnessed!) they basically had to send every single device from that room off to professional cleaning services to have them vacuumed inside and out. Took them the better part of a year, running with half capacity, and it was really not that big of a room.
Those fire fighting chemicals are all pretty much all super corrosive and the kiss of death to most electronics. The type K's are probably the easiest of the powder types to clean up(essentially baking soda, and only good for certain kinds of fires) but I wouldn't want them hitting electronic gear either. That concrete dust is no joke either, and if you ground that or plaster up, it will light up a PH strip as well as abrading the crap out of everything.
For bonus fun, the Halon type extinguishers will release hydrofluoric acid if the gas gets too hot. Like from a fire. So even if you don't have a full room fire system, you should have some respirators if you plan to use one inside. As opposed to those totally normal OUTDOOR server racks right?.
CO2's may damage electronics as well, as they tend to turn the humidity in the air and deposit it on surfaces as frost. At least they are non-toxic, non-corrosive, and easy cleanup. Just don't fire off to many in a closed space or the fire may not be all that gets put out.
As an aside, how many of you have an electrical breaker to pull for the server room that ISN'T inside the room? Funny thing, me neither, even though it's a lot easier to put out an electrical fire when the power is off. I mean I can march across the campus and pull the main breaker, but wouldn't they rather I be able to just dump one room instead of the entire site?
We also don' t have an air vent. The AC just recirculates around in the room(at least the AC is a split system, though I have mopped up more than one idiot box solution were some genius rolled a portable AC into a literal closet and tried to beat thermodynamics). We ARE winning, but it's a pyrrhic victory as it would be cheaper to exaust heat than to re-chill it, and the air from the hall is usually in the 70's. Most of the year we wouldn't even need AC. Instead, we installed a second AC unit just like the first one, and the room will still overheat in about two hours if they go down, meaning I gotta drive in in the middle of the night and prop the door open. So we probably spend 3 watts to cool every watt in the rack.
I should probably move on from here at some point I think.
All these horror stories are making me amazed, considering how rarely I dust and hoover at home (with a housecat and carpeting!) that I've never yet had a drive fail; I'm running about thirteen spindles 24/7 and most of them have been going for 5–11 years.
Of course I'll no doubt be posting a reply to this comment in the next hour having squarely tempted fate by even mentioning it…
All these anecdotes about concrete dust causing drive failures, and I'm left thinking that every single drive I've ever handled has been a sealed unit. Is the dust somehow getting into the housings (possibly due to how microscopically fine it is), or is it just getting into fans and causing cooling failures? Or onto controller boards / motherboards and killing those?
I'd assume when it's fine enough and there's enough of it that it can enter the airhole…I don't remember whether drives have those anymore and don't fancy busting my PC or NAS open though I'd assume so. Controller boards used to be just bolted to the underside of drives too…though admittedly I'm pretty much talking about consumer-grade stuff here and it's been many many years since I did anything with server hardware.
Fair point about the air holes (presumably for equalising pressure, I wouldn't expect any appreciable airflow through them). I wonder what it is about concrete dust, rather than other things, such as general dust or pollen, that makes it more destructive? Is it particle size, chemical make up, Hygroscopicity (good word that), or something else? A quick google indicates that concrete dust + moisture can form a conductive layer that isn't going to play well with electronics, so I'm wondering whether drive failures are from a combination of vibration and dust in the controllers causing it to get right in to the electronics and making paths where there should be none.
I might be talking bollocks here, but I 'think' that concrete dust is 'sharper' than most dust. That is, the tiny little individual items have sharper edges than most dust.
A bit like lunar regolith dust, which (again, I believe) is reputed to be one of the finest destructive (to things like seals and polished mating surfaces) materials known to man.
The argument put forward for the lunar dust is that because it never really gets 'moved around' much (due to no atmosphere), all the sharp edges and corners are never rubbed flat.
I'd would imagine that freshly produced concrete dust is similarly sharp edged, and thus more effective against smooth surfaces.
Imagine sliding down a hillside composed of fresh sharp-edge flint, compared to sliding down a hill of smoothly polished river-stones. The second will leave you bruised, but the former will also have you cut to shreds.
What you're describing are not Winchesters, but probably more like CDC SMD devices. Winchesters were 'nearly' sealed, with fixed platters that you couldn't remove.
Various companies made similar removable disk packs (DEC's RP03 and RM03 were similar, made by Sperry or Memorex), and you're right that that there was more chance for exposure, but the drives themselves had pretty substantial filters that needed to be changed on a regular basis, and when out of the drive the disk packs were pretty well sealed by the bottom cover, so the only real time they were exposed was when they were being inserted ore removed from the drives.
There were disks in cartridges like the front loading DEC RK05s, and I always wondered how well sealed the cartridges were when out of the drives
I worked on a climate controlled room that received filtered air where those removable disk packs were used in minicomputers. Naturally there was a prominent no smoking sign on the door. I witnessed the site bosses cigar destroy one such drive in front of my eyes as he exhaled a couple of meters from the unit, 5 seconds later smoke and an acrid smell was coming out of the drive.
There were also larger sealed drives that were so delicate they had to be transported on air suspension. One shipping company ignored that directive and two thirds of a batch of drives - crazily expensive then - arrived from Germany broken, thus delaying a big project.
It's not an obvious gaping hole in the drive, but almost every drive you've ever handled almost certainly had a very tiny hole on it somewhere with a small foam-filter inside the drive to filter particles from the incoming air. The newer helium drives are obviously airtight/sealed, but not your run-of-the-mill spinning rust :-)
"The newer helium drives are obviously airtight/sealed, but not your run-of-the-mill spinning rust"
He is the most prone to leakage of all the gases. Since H likes to exist in pairs, it isn't as leaky as He. This makes me very suspicious about He filled drives and their longevity.
In the "dim time", disc drives were huge affairs, spinning multiple platters of 14 inch (?) diameter on a single spindle. Often these stacks were removable, using a specialized device, allowing "unlimited" storage.
While the drive units were "sealed" when in use, recirculating HEPA filtered air and utilized a "cleaning cycle", "head crashes" where common even in "clean" environments.
True, and I suppose the harmfulness of powdered 'crete depends on what epoch of computing we are talking about. In the days of "walking fridges" and blinkenlights I can well imagine that disks were fragile and temperamental beasties. I don't recall seeing air holes on any of the various disks I have in the last PC I built. I tend to re-use drives from one generation to the next, I think last time I did that I "put into storage" any drives smaller than half a terabyte, but I've had plenty going back to the days of full size IDE jobs,and they're all in a box somewhere (I've never experienced a drive failure despite not being particularly precious with them).
These days, a lot of storage is solid state, and I wonder how well that plays with particulates. I'm thinking it probably depends on particulate density and conductivity. Despite the air filters on the inlet fans in my gaming PC case, I still have to periodically open it up to remove the accumulation of dust, although this is more for cosmetic reasons, it has never caused a failure.
Speaking of walking appliances, I had a funny thing happen to me this week. We must have done something terrible to upset our clothes washer, for it decided to go on strike, and to prevent us from crossing its picket line, it walked across the laundry room, bringing its drain pan with it, and barricaded the closed laundry room door.
Twice.
Thankfully both my wife and I are strong enough to shove it back into position using the door. I think I'm going to have to bolt the drain pan to the baseboard or something. (Top edge of the drain pan, so as not to create a place for it to leak.)
Check the date of the anecdote - time was that removable disk packs were all the rage, and even if they had a case of some kind on them, they definitely weren't sealed against the fine concrete dust produced by a diamond drill.
And once in a while, you really need a new cable duct and have to call in the diamond drillers. Even if they take all possible precautions, they'll leave 0.0001% of the dust behind, and that's plenty.
(And those precautions include constant running water, which doesn't help matters if they're drilling above an equipment room.)
One side note regarding concrete dust: holes in concrete are usually drilled with a hammer drill. That generates lots of vibrations. Spinning rust hard drives don't like vibrations very much.
Of course, the impact would probably vary depending on if the holes were drilled in the floor vs. walls and ceiling.
Depends on the diameter of the hole. Bigger ones are made using a diamond drill which is technically not really a drill. It doesn't cut with it's perimeter but hones the concrete off with it's rotating front side. The result is very fine and very hard dust from ground concrete, steel and some diamond rsp corund particles. It it's not completely washed away by the water that should cool the drill it would make a perfect drive killer.
"Of course I'll no doubt be posting a reply to this comment in the next hour having squarely tempted fate by even mentioning it…"
Definitely!
Happened to me after praising the quality of some kit I've got on this forum. Though the good news is that shortly afterwards I could post an item about how well the supplier supports even elderly kit :)
It's that time of the week again -->
I'm a bit OCD on backups - all really important stuff lives in Nextcloud on a VPS somewhere, synced to my (or the relevant family members') PC/phone. The Nextcloud is backed up hourly to my home server and my brother's home server, some 400km away.
Regular stuff lives on my PC, with an hourly backup to my home server, which is in turn backed up to my brother's and vice versa. So everything exists in at least 3 copies in at least two locations. (The backup is never mounted as a drive, so your run-of-the-mill ransomware would not too easily get to it, and all files are versioned.)
Maybe overkill, but drives are cheap.
"Maybe overkill, but drives are cheap."
I have a big stack of drives that came from a rack mount drive case I purchased second hand. I think there were something like 16 320gb drives. The case had some cool blinkenlights so became a movie prop and I got to keep the drives after building some circuits to blink the LED's at random (for a limited amount of random. Ok ok, it wasn't random at all, sue me). The drives have been great archive units. I store my off-site copies at my mother's. If anything were to happen that affected us both, the need for the backups would be unlikely to arise given the distance. I used to have a friend with a small software company and he carved me out some space for backup use, but he sold that business. Cloud backups seem for my needs a big waste of money. Drives are cheap.
Given that stone dust is heavy and unlikely to remain airborne for any length of time, I suspect the association of the drive failures with the drilling of concrete is not due to dust ingress, but vibration.
It's likely that the drilling was carried out with an industrial hammer-action masonry drill (or similar) while the servers were live. If not fully suspension mounted, that could easily cause minor head contact with the platter surface , leading to longer term failures as the metal oxide released from the 1st crash caused cascading head crashes. It would also be a much more probable cause of any immediate failures, as mentioned in one of the posts.
There was a fire where I used to work and as the building was well-designed the flames only affected one room and pretty much burnt itself out before the fire brigade. The smoke; however, got to other rooms in the building and the computers inhaled whatever nastiness was floating around. For some reason they decided to have the laptops cleaned instead of just replacing them. Pretty much all of the affected computers' discs died a month or so later.
My worst was concrete moment was not in a server room directly though that didn't make much difference, picture a brand new almost completely fitted out facility, that fresh plastic smell of new cables and seats not sat upon, with a commissioned server room and running HVAC.
There are two simple ways to level a surface before putting down the corridor floor tiles, you can skim over the concrete with levelling compound or use a dirty great grinder (much faster) - guess which one was chosen on this particular Friday.
Yeah I did a weeks support for a high end interior designer many years ago. They were doing a minor piece of work on a massive house that was being renovated by another designer. On my one site visit I met the IT team installing the various systems and kit into the house. You could see who was at the front door (or any of the CCTV cameras) from any tv in the house or a panel on the wall.There was also an entertainment server in the basement, this was also connected to an expensive set of satellite receivers which (from memory) used a motorised dish but it might have been several dishes using DiSEqC. There was cabling from each room to the computer room, wifi and Ethernet points in each room. Cooling for the computer room was supplied by an AC unit outside the room and then filtered before being vented in. This setup would have supported a small business and then some quite easily in the house. The racks were also raised off the floor as well to help deal with if there was ever any water on the floor. The amount being spent on the IT was immense and I dread to think about the rest of it.
Talking to the lead IT bloke he said they’d had a nightmare scenario a few weeks earlier. The IT kit was in the computer room fully installed and cabling was being tested from everywhere. Well it seems some room in the basement (where this kit was located) also needed to be cabled up etc. The building contractor/designer/someone had missed it off the plans or there had been a change of use. The problem was they hadn’t drilled any holes for the cables and they couldn’t get into the computer room via any existing hole in the walls/duct. So a new hole had to be cut through a concrete wall then through the plasterboard and into the room. This was a nightmare for the IT people as the room was mercifully dust free, for a building site it was fairly spotless. To solve this they had a vacuum cleaner well outside the room connected with a very long hose into the computer room. They’d put plastic sheeting all around the door because to get the hose in meant leaving the door open. The end of the hose was connected into a small wooden housing the carpenter had made to enclose where the drill would penetrate the room. Then they’d covered the racks in plastic sheeting too for good measure. They’d also vacuumed the hole from the other side during drilling as well. This was all to minimise eliminate the dust near the IT kit and the pictures they’d taken of the precautions were extensive.
This was all because the owner of the house was known to be a control freak and if it wasn’t perfect or went wrong would threaten to sue anyone and everyone until it was fixed. As it was the drilling went fine and they had no issues with any dust from the drilling.
Had a system in a mill that had a train derailment and chlorine gas spill outside.
https://eu.greenvilleonline.com/story/news/local/2015/01/05/years-graniteville-train-wreck/21278089/
People died.
All sorts of machinery failed over the following years until they gave up and shut the plant. Very sad.
One large customer had servers dotted around the city. We used to do acceptance tests before placing fresh servers on contract - correct software.
We received a call asking for help on a server we had no previous knowledge of. Tape backups were failing.
I went to site once we had a fax stating that all work was chargeable.
I found a server room which was still under construction. Only about half the tiles for the false floor had been fitted. The server rack lacked all its outer panels.
There was cement dust EVERYWHERE.
The two tape drives, each costing around £1500, were scrap. They had probably never made a successful backup. I scrapped every tape cartridge which was not still factory sealed.
I vacuumed about two pounds of cement from the system board of the server to keep it running for the time being.
When the time came for its acceptance test, the server failed on several counts, including lack of a working backup solution.
The worst I've seen was volcano ash! Multiple fixed disks, hard-media removable drive cartridges, and floppy diskettes died despite filters in the drives and filters outside the computer rooms. We also had to replace some disk-drive heads.
Icon for volcanic eruption --->
I think that plaster dust is finer. At MIT on the MULTICS machine they had a "fire hose" drum for swapping. One day a plasterer totally clogged up the filters causing the drum to go off-line. "No problem," said the tech. "I'll just go get some new filters."
So he pulled out the clogged filters but, before he got back with the new filters, the drum was totally toast.
We once had a UPS fire, and it turns out that the smoke from a battery acid fire isn't good for drives, if not reliably fatal.
The more expensive kit in the room got sent to professional cleaners and was fine, but we had ~45 Pentium 4s in the room (old then, but working).
When I powered them up, most of them were fine, but one started taking multiple attempts to boot, and a two others just refused to start any more.
As a note, the smoke from a battery acid fire, even after it's out and had a day or two to vent, isn't great for your lungs either. I had a nasty cough for weeks...
"We once had a UPS fire, and it turns out that the smoke from a battery acid fire isn't good for drives, if not reliably fatal."
I find it odd that a UPS would be housed in the server room. I've been fiddling around with a design for a home I'd like to build and one of the components is a mechanical room that is separate from the house for things like a storage battery. I figured that fire and other disasters are often due to the sorts of systems that could be housed in a separate shed. I've been playing with some designs for a battery enclosure that could be flooded in case the battery caught fire as a way to cool it down. It would be a total loss from being immersed in water, but it would likely be a total loss if it caught fire as well.
Usually not a good combo! Gotta love the architects that scribble IT in as an after thought. At one place, the data cabinet was in the basement directly under the drainage system for the roof of the building, which was flaaaat! Bring on a once in 50 years rainstorm! Yay! Someone call the insurance company!! I couldn't wipe the "I told you so" look from my face for weeks...
Raincoat just because!
At a company where I worked, they had a Tandem non-stop computer in the basement. Due to a leakage, the system was knee-deep in water before someone noticed. Tandem computers advised us, that this was not within specs and did not fall under the non-stop guarantee. However, the system did not stop. Afterwards, some components had to be replaced, but even with the flood, up-time was 1000%.
I commissioned a new Mainframe for a customer in a beautiful new DC in a basement. they'd spent about £1.5 million on the DC and equipment, there was a strange looking door in one corner which I had to open, sure enough there was a 5 inch cast iron sewage pipe routed through the DC. Apparently it would have cost £50K to reroute the drains and would have meant that a pipe would have been attached the outside of the wall and this wasn't allowed. I did warn them that they would be up to their chins in sh*t one day
Built a studio complex in Paris some years ago, and discovered the hard way that a cast iron sewage pipe made a vertical to horizontal right angle in our ceiling space.
When the foot of sewage was being cleared out we discovered the undeclared asbestos that had also to be removed.
Messy...
I "discovered the hard way" that a plasticsink drain pipe made a vertical to horizontal right angle in our Kitchen ceiling space.". Elbow leaked. Plasterboard ceiling now has a 40cm by 30cm irregular artistic hole with vibrant coloured edges where water reacted with the paper glue of alongside paper dressing to roof.
I was involved as a support person in one of the Olympic games. The servers were in the basement. With the many thousands of spectators, the toilets could not cope, and there was a slow sewage leak. People had to use wellington boots to get to the machines. They reckoned that with the rate of ingress, they could survive till 24 hours after the competition finished, when the water would be mid thigh deep. The sequence was medals awarded - switch off the power. The machines may still be there!
"Apparently it would have cost £50K to reroute the drains"
And nobody will stop and think about how much money will be lost from one day of that DC being off-line. The cost of the downtime could be much more than the cost to replace all of the hardware too.
If the pipe couldn't be moved, somebody would have done well to spec in a pressure door like you'd find on a war ship and make sure that any loss of that pipe would have it discharging elsewhere.
Wouldn't matter. When the pipe fails, years after anyone forgets why the pressure door is there...
"Hey, what't that creaking noise?"
"Dunno, but it's coming from behind that wierd door."
"There it goes again! That's gonna drive me up the wall, let me go see what it is."
<click>
"AAAUUUGHHH! IT'S EVERYWHERE!!!"
Which you'll find out if the downpour is greater intensity that drainage system can cope with and the water decides it's time demonstrate Archimedes principle. This is why there are usually pretty strong rules about what can go in the basement and what kind of failsafes (at other levels) you need.
My old house in the UK, had noticed the sourgas (H2S) smell when leaving\returning to the dwelling, came the weekend, ex Mrs Scorn running the washing machine, I'm in the home office & notice the water suddenly spilling out of the cover at the rinse cycle.
I go to inspect under the cover & its the easiest term I can use to describe it is "slurry", slam the cover down quickly.
Go to Wickes get a drain clearing rod set, tell wife to take her mum & the kids shopping.
Don't you want me to stay & help?
You gonna do what I'm going to do?
No!
Then fuck off as I asked, I don't need or want any helpful comments about how gross it is from you or the kids falling in.
I got it cleared, root cause women's sanitary products (Much retching & trying not to make it worse personally), patio area cleaned out & now bathed, refreshed relaxed... The Ex & co returned from the shopping.
She wants to look & so I open the cover to show her a nice clean open sewer & at that moment Mother In Law chose that moment to provide working proof of it working normally.
I did have to unblock it about two more times (Once at night which did make it more tolerable), but never so bad as that first time.
Icon was well deserved.
I used to work for a farmer who'd retired to a cottage he'd bought.
One day he noticed that the back yard drain was slow clearing water away - so assuming it was partially blocked with stones from the fairly recently dashed wall, he stuck his hand in to fish them out. He didn't find a load of hard stone, instead he found something soft ... and brown.
So he grabbed the plunger, then noticed that when he plunged down, the cover of the manhole in the back street lifted slightly. So he lifted the cover out to find it blocked to the top - and no it wasn't like slurry. "A number" of houses in the village would have had their sewage go down that route.
Now, what he should have done is call the local water company (utility responsible for the drainage for our US friends), but being a public spirited person he figured he'd just rod out the obstruction and it would flow away. But after using a bucket to empty the manhole (putting it over the wall into the paddock as fertiliser), and trying hard with the rods, he couldn't budge anything. So he gave up and called the water company - who arrived with one of their drain clearing lorries and jetted it clean in seconds, adding a "why didn't you call us earlier" comment.
For good measure, we heard from his wife that even after scrubbing his hands multiple times, he wouldn't touch his food and made his wife feed him at tea time. Given what a farmer has to handle and deal with as part of working life, that's quite something.
And for years afterwards, we always joked about "Bill's bangers"
Yep - been there!
During a building move we discovered that the proposed server room location (formerly the gents toilets) had the main potable water tank located right above it, and that they were proposing to remove most of the wall supporting one side of said tank! The tank was, reluctantly, decommissioned and replaced elsewhere but what they didn't tell us about was the open rainwater gutter that also ran above the false ceiling. Fortunately it overflowed during office hours, we noticed and were able to mitigate the problem (plastic tarp and buckets) before the AS/400 got a good soaking.
Well it isn’t that it related but when doing some work on a building the contractors removed some heavy duty partition walls, and then heard creaking above them, someone had installed a water tank on the top of the partitions….. and it was left hanging by the pipework.
A more it related one was the network cabinet in a small room on the ground floor which was under the toilet block on the first floor - I don’t need to say any more do I.
I looked after the support of on place, some decades ago.
The IT manager was only the IT manager because no one else wanted to be. She was lovely but relatively clueless. However she was also very willing to learn and was a genuine pleasure to work with.
She knew enough to do basic checks (are the coax cables connected and properly terminated, for example) so generally when she called it wasn't something silly.
This one morning I took the call. Nothing was connecting to the server. Of course she'd done the usual checks so it was a case of "can you tell me what the server is saying on the screen?"
At this point it's worth noting that the server lived in the basement so it was a "yep, let me go check and I'll call you back."
Five minutes later the call came in - don't worry, the basement is flooded...
The UK Academic network, JANET's POP in the east of England was in a room in Cambridge beneath a roof garden which had the inevitable consequences... The way around this was to build an internal shed in the machine room! Sadly the photos seem to have vanished from the Internet :-(
Where I used to go to Uni, we had a tower block, which, during a refit in the mid 80s, they added a server room in the basement.
This room also contained the electrical intake for the building and the outside phone/network links.
The building was (and still is, although the uni moved out years back and it’s now “affordable” housing) on the bank of the Thames.
That server room flooded regularly, and when it did, the electricity and phone lines died as well. Especially a pain in the arse when (as happened to me a couple of times) you were in the lift when the electricity went. While the uni did have people on staff who were trained in how to winch the lift to the nearest floor and open the doors, they were based in another building, and Mobile phones were an expensive luxury then.
Especially a pain in the arse when (as happened to me a couple of times) you were in the lift when the electricity went.
Having worked in a building with some of the least reliable lifts I've ever met - and being one of those responsible for any manual winching required - I got into the habit of only using the lift if absolutely necessary, a habit I continue to this day. The lifts where I currently work are much more reliable and can't be manually winched, but do sometimes need a "reboot" so I'll even put a trolley of equipment in and then race the thing up (or down) the stairs, rather than accompanying the trolley, particularly if it so happens that I'm the only technical type on site that day. It does help that this building only has a ground and a first floor, so I'm not racing up half a dozen storeys :-)
M.
At a previous employer, we periodically had to transport small (103L contents, 1.5L physical volume) gas cylinders. Site regulations required that if a cylinder was transported by elevator, no human was permitted to be in the elevator with it. So we, too, raced the elevators. The most common gas cylinder to be transported was 20.9% O2, balance N2... in other words, air.
To be fair to the regs, I'd not want to be in a confined space with any gas cylinder if it suddenly decided to let go, be that air, O2, N2 CO2 or anything else, no matter how small the cylinder. The worry isn't about small leaks (although these would be annoying) but total failure.
Safety sign foe obvious reasons.
Our uni computer center had a main floor and a basement. The machine room, I/O room, keypunch room, terminal room, restroom, and a few bureaucrats' offices were on the main floor. A tech room, machinery room (chiller equipment, motor-generator unit, etc.), and all the programmers' offices were in the basement. Since Halon 1211 is heavier-than-air, any Halon deployment would risk the lives of the programmers and techies before those of the bureaucrats ...
Simon might have some thoughts on that.
I worked in a Marine Lab for 20 years, the lab does lots of remote sensing and modelling resulting in lots of kit, racks of disks and compute. It had outgrown its old server room built to house a 1970's era mini computer so required a new DC to be built somewhere in the building. Us in IT favored using the old server rooms "terminal room" which at the time housed a load of open access PC's but was plenty big enough for the proposed 24 Cab DC and as it was next to the original server room had near by access to the necessary service ducts for easy routing to the roof for HVAC fan units and to the basement for connection to the electrical switch gear as a new higher rated mains cable would need to be routed from there.
The total halfwit head of operations of the lab had different ideas as he want this prime location to be turned in to meeting rooms! So his possible locations started in the roof space, no lift access and semi open to the elements so would cost a fortune to convert. The utter fool then moved on to a area out the back of the lab that had been built as an extension, half of it was used mainly for storage whilst the other half was a refrigerated mesocossum lab holding several large tanks each one holding THOUSANDS of liters of sea water!!!! And the whole area had a salty atmosphere, what could POSSIBLY go wrong! Anyway the total idiot was adamant about this so when we went out to tender to about half a dozen DC builders he had that included as a possible location along with our preferred choice. Lucky for us 4 of the DC builders refused point blank to tender for that location whilst 2 other did but said it REALLY wasn't suitable. But hey guess what the utter c u next tuesday went against all the tender scoring and decided the location would be his choice! Thankfully there was a happy ending, one of the reasons the DC was being built was to house a new HPC and the deadline was very tight to get the DC built and the HPC delivered and operational within the funding window. Due to the location and design of the extension the fan units for the AC would have had to be sited outside the building which was surrounded by residential housing and flats and that would require proper noise monitoring to be set up etc to enable planning permission, this of course would take ages to do and we would miss the funding window, oh dear what a shame ;o) So our location got the go ahead in the end.
What kind of total pr*ck who could barley work a light switch let alone have a clue about IT goes against the IT department and DC advice ignores the scoring criteria for the tenders and presses ahead with their own bonkers plans!
My father used to work in maintenance at a UK university and told me a story along the lines of...
Something went catastrophically wrong, leading to a flood of water threatening to swamp the room with The Computer in it.
This was beyond sandbagging, so a bright spark decided to commandeer a pump to get rid of the water...but where to get rid of the water to?
As the pump was fairly high-power, someone had the idea to hook the pump up to a very long hose and take the water up to a large tank with spare capacity on the roof on an adjacent building.
This was duly done, but even after a lot of pumping, the flood still seemed in danger of engulfing the computer room. In fact, it only seemed to be getting worse.
Yes - the initial flood was caused by a burst pipe linked to the tank in the next building that the workers were currently pumping water back into.
(The flooding eventually spilled out onto a grassed area outside the building. The foreman was in danger of having a sense-of-humour failure when someone ran off to the nearby park and stole a rowing boat to put in the newly-formed lake)
We had a Data Centre on the 3rd floor and the sprinklers went off on the 14th floor over the weekend. By the time the water got to the DC, it had accumulated a lot of carpet fibers and plaster from the ceiling tiles and found its way through a crack in the concrete floor and straight over the top of the main computer.
Needless to say, that machine did not survive and within a few weeks there was waterproofing urgently applied to the ceiling of the DC.
> The story has at least one happy ending: Clarence was paid for the job even though his work washed away.
... and hopefully got another commission to reinstall the replacement system when the museum was reinstated (if it was rebuilt?). Maybe, this time with flood alarms added.
In the dim & distant, I I worked for a mid sized finance house, where all the important stuff ran on IBM Kit. As part of a major system upgrade, the company acquired a swanky new tape robot backup system.
This consisted of an eight sided cylinder, about 10 foot tall lined on the inside with hundreds of tape racks and a forgotten number of drive units. Mounted in the middle of the cylinder was a robotic arm that moved the tapes around, with impressive speed & dexterity. I was part of the WinTel team and had nothing to do with the mainframe stuff, ever. But management decreed that everyone who ever needed to set foot in the computer room HAD to attend mandatory safety training on the new tape library. And very entertaining it was too.
The silo included a half height access door, which could be used to enter the library for maintenance. We were all told about the procedures required to access the insides of the beast, which included removing your tie (finance house), in case it got snagged on anything. But the really interesting topic explained what to do in the theoretically impossible situation of the library waking up, while someone was inside the machine. This basically amounted to standing upright and keeping still, overriding the natural instinct to duck out the way as the arm started moving. The reason, we were told, is that the arm did a slow speed, 360 degree rotation, in both directions, while changing heights, checking for obstructions. If it touched anything during these safety sweeps, power would immediately be cut. The description of the potential consequences of avoiding the arm during its obstruction check have stayed with me for over 25 years - "no one wants the job of cleaning out operator purée. Oh and you'll invalidate your warranty too.."
I was shown the massive tape silo library in a bank once and was told a similar thing - stay still and let the robot arm find you, otherwise you'd be hit by it when it got up to its 30mph speed. Also, if you closed the door to the silo without checking first if someone was inside, it was a sackable offence.
A company that I worked for had such a thing, and it had previously demonstrated its lethality. Whenever a new starter was getting an orientation tour of the building, the person in charge almost seemed to take pride in pointing it out and announcing that it had once killed man.
You'd never get away with that now.
We have the robots in cages, you cant open the cage without pressing the big red button, and standing orders are if you have to work in the cage, your padlock goes through the breaker lever (mines a nice purple colour... I did offer the pink padlock to the PFY when she started..... she said she'd rather have the orange one... not exact words but we're in polite society here )
If you're found in the cage with the power still on.... all sorts of bad things will happen to you....... and then you'll be given a final warning/sacked
Beer because I've escaped to the pub
When I was in university way back in the 80's, the CompSci department and the associated computer/server room was relocated from the edge of campus to two swanky, recently refurbished floors of the university's main building. This building was (apparently) the largest completely brick-built building in Europe, and included several large shafts that ran from the ground to the top floor that was originally used for heating (that heating system was long gone by my time). The CompSci department had decided to install a relatively new technology called "Ethernet" throughout the university's main building, and they used one of the air shafts as a conduit for the main backbone cable. Not being complete idiots, they used a cable that ran through an armoured steel shell just in case of unforeseen circumstances.
Meanwhile the next couple of floors above the new CompSci department had started to be refurbished (this was an on-going task at the time that lasted several years). The builders had discovered the air shafts and had decided to use them as an easy way to get the demolition rubble down from where they were working to ground floor, at which point they would cart it into the skips. Unfortunately everyone quickly discovered that the armoured Ethernet cable, while cable of surviving anything an drunk undergraduate (or postgraduate for that matter) could swing at it, was not cable of handling 50 tons of building rubble dropped from a great height.
I'll leave it to yo to image the conversation between the head of CompSci, the Vice Chancellor and the building contractor management.
On the register, if you write something about something that someone isn't supposed to do, you won't even be able to click "Submit" before someone has posted the quote about " 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.” - T Pratchett (Thief of Time)"
I had a similar problem with a cleaner who would helpfully turn off any plug socket that they found still switched on while they did their evening clean up. A handwritten sign in English asking him or her not to do it to a particular socket went unheeded, until someone mentioned our cleaners were all Brazilian for some reason. A new sign in Portuguese did the trick.
a cleaner who would helpfully turn off any plug socket that they found still switched on
Back when I worked in the MoD all our labs and offices had emergency shut off buttons outside in the corridor. The MoD police who patrolled at night would walk down the corridors pushing all the buttons as they went, which was a PITA if you had an overnight run going(*). Simple signs didn't work (MoD Plod being Plod but very much more so), so in the end we resorted to fixing a metal plate over the switch and putting a sign saying "Power MUST remain on, by Order of the Director" over the plate.
(*) Or worse, a vacuum pump maintaining a chamber that took 2 days to pump down.
"A handwritten sign in English asking him or her not to do it to a particular socket went unheeded, until someone mentioned our cleaners were all Brazilian for some reason. A new sign in Portuguese did the trick."
The next week they'll be a new set of cleaners as the contract got awarded to some other company and the new staff only speak Spanish, or Latvian or something else. You would also have to assume that the cleaners can read in the first place in any language. It's not a job where literacy is much of a priority so those sorts of jobs will often be done by immigrants coming from places where education was not that important. Cleaning is a job they can do where the amount of education they received is not a big factor.
I've always found it a bit of an oddity that in the UK all of the power outlets have switches. Most other places the power switch is on the device as the plug might not be very accessible. It could make sense to hardwire the switches out of the circuit if devices that will be plugged in need to remain on.
Most ranges of UK sockets are available switched or unswitched - but often the unswitched ones aren't stocked by the retailers (you either have to go elsewhere, or have them ordered in).
I recall, back in the 80s, when I was a junior engineer in a large engineering company. We were getting some engineering workstations in (anyone remember Apollo Domain ?) which was quite something for us as we still didn't have PCs on desks (the systems dept. would actively block attempts to sidetrack needing their mainframe). Of course, the sockets were switched, and on the other side of the partition from where our workstations were - in a corridor actually. I queried the wisdom of this - don't forget that this was back in the days of manually parking heads before power off for some drives, not to mention the issues of yanking the power on these Unix systems (it was also before the days of journalled filesystems). I was told that company policy prevented unswitched sockets being used - but thankfully we never actually had a problem with them being turned off.
At Xerox they hired cleaning staff that didn't speak English as a security measure (resulting in several signs in Russian or Polish saying things like "please do not lock this door."). Then they enrolled the cleaning staff in English as a second language.
We really had very little trouble after we managed to the cleaning staff in their language.
We had a meeting to discuss where to put the new sever and associated equipment in the 'new' Unit we had taken over which was adjacent to our existing premises. The small conference room we were using was ideally placed underneath the offices enabling convenient access without disruption. Modestly sized with false-wall panelling, the room had an outside wall suitable for the air-conditioning, and next to the electrical supply distribution panels.
The room had three full-size doors: one internal to the atrium, one to the workshop and another in the wall-panelling which went nowhere except into a 30-40 cm void.... so we thought. Brooms were kept there and old cardboard boxes. In my curiosity, I cleared the boxes to have a closer look and found the gas supply, meter and valves together with that slight 'whiff'...... Gas and connected voids into the new upstairs offices do not make a safe place of work.... never mind putting it next to electrical switches... This installation was illegal and contravened building regulations so had been hidden by the landlords who had 'furnished' the new building..... The previous users had not noticed or pretended not to, after all, Head Office was in another city...
> ....found the gas supply, meter and valves together with that slight 'whiff'...... ... This installation was illegal and contravened building regulations so had been hidden
New York City had that. Several times in recent years. One building blew-up/out. They found rubber hose(!) tapping into the adjacent building's gas supply. Nobody knew nothing about it. (And yet you'd think the tappee would want reimbursement from the tapper; and IIRC there was a private meter in the rubble.)
This led to a city-wide sweep of gas inspections, and not just to collect "tips", but seriously finding illegal hookups.
At a university I worked at, a routine electrical upgrade on a server room found that the main power feed had been tapped BEFORE the main breaker and was unfused by anything short of the campus substation. The load on that circuit had crept up over the years and it was sheer luck that they discovered this before it started a fire, but after most of the insulation had melted off the cabling.
School with cadet unit here built a proper mil-standard armory (for storing arms) when the government demanded it (in an effort to close all school cadet units).
The armory was eventually re-purposed as a chemistry lab, but the high-current redundant power connections were not repurposed: when some student shorted a power point, it took out all the power back to the substation
I was once involved with an issue with a large Optical Drive based Jukebox. Entire platters gradually stopped working one by one. The Optical drives were replaced but they still could not read the platters. It turns out that there were printers in the same room as the jukebox and someone dropped an open carton of toner on the floor. They cleaned up the mess but did not tell anyone. The fans on the jukebox obviously sucked up some of the toner "dust" and gradually covered all the platters. An engineer went in and carefully cleaned them all and everything else they could clean and it all magically started working again.
The same customer also had some building work needed in the computer room over a few weekends so an engineer was sent in on a Friday to shut the sysytems down, cover them up with large pink plastic anti-static "condoms" and then come back on the Monday to unpack and restart it all having cleaned up under the floor as well. They were therefore aware that computers dont like dust! I think the printers got moved elsewhere after the toner accident.
As a fresh young engineer I was our on site at Liverpool University. Back in those days Myrinet switches exhausted to the side.. so we cut holes in the side of the rack.
In this case I remember having to use a hacksaw to cut a large bolt to length... in the machine room. I was spotted doing this and dragged out of the room by the machine room manager....
Dont cut metal in a machine room.... lovely swarf and filings don't do the servers any good.
In my first job, at a local council, I witnessed two events in the year I was there.
The first was that the machine room containing a Sperry Univac 90/30 was in a wooden extension to the town hall, and there was a water leak in the ceiling that meant that the system was operating there for the remainder of the time in that room under plastic sheets. The roof was not fixed because it was mere weeks before they moved it into a custom built machine room in the new council offices.
But once there, the actual maintenance engineers from Sperry did the unthinkable of using an angle grinder to cut a floor tile, actually in the machine room with the system running. They set off the fire alarm, which resulted in an emergency shutdown and a building evacuation.
The one thing they could not do was to refuse to fix any problems as a result of their shoddy work.
> Dont cut metal in a machine room.... lovely swarf and filings don't do the servers any good.
School's Piano Technician had a (286!) PC in his workroom. Not a lot of metal-cutting done there. But he was a handy guy. Also other workers snuck-in to do little projects on his drills.
The PC was acting goofy. It was still days of bad RAM, so we replaced the lot (2 whole Megs!). Still every few days something goofy.
On a hunch, I took the PC out of the locked cabinet, removed drives and cards, turned it over on the workbench, and thumped it. Thought I found some tiny flakes of metal. Put it all together and it ran flawlessly.
Many years ago, I was shown the server room of the central govt department I worked for (at a satellite office): lovely, huge racks, great aircon, big halon fire system (it did say it was many years ago, mid 90s to be closer) and a back wall of toughened glass.
Not toughened enough, apparently, to stop a Transit at 20 mph, into which was loaded a few tons of govt kit...
I spent a some time working in a data centre in Milton Keynes for a commercial bank.
I'll swear that the glass in the windows was 6-8 inches thick, there were huge reinforced concrete buttresses, and the non-powered doors into the data centre proper were so heavy that you could barely move them to open them.
The data centre was only one story tall, and surrounded by a 10 foot earth bank (which looking at the gate, was probably also reinforced).
There were no identifying signs what the company was, which was really confusing the first time I went there, but I understand the security because it housed the servers that ran their London trading rooms.
I heard someone say that it was built to survive a plane landing on the building.
I used to work for a bank whose main data centre was so close to Manchester Airport, you could almost see the whites of the crew's eyes as they came into land.
(This data centre was surrounded by glass-topped 10 foot high brick walls, and the local urchins still managed to scale the walls and steal all the fish in the ponds.)
Similar story but different bank that had their main data centre under the landing path of East Midlands airport. After one incident in the 1980s (January 1989 - I lived a few miles away and could see it from the motorway bridge) when a plane made an impromptu heavy landing on the M1, they decided to build a backup data centre about as far away from any flight paths as they could in Staffordshire. (I found this bit out several years later, when working for said bank)
[small private US college in the '90s]
Had a late Friday page after a cleaner plugged their hoover into an orange "Do Not Use" labeled outlet outside the server room. [silence decends]
A new admin also plugged the old IBM chain printer (producer of much loved green and light green fanfold print outs) into an orange outlet and shut down the server room at 8:00 AM one day.
Reading signs is for chumps.
Any idiot knows that if you put stuff in an area that's under construction, it is going to covered in drywall/concrete/etc. dust, paint, and potential impacts from various forms of debris. And also be exposed to various noxious chemicals in the air. He should have either had the shipment held, stored the tape robot in a storage area/facility, or at least covered it in plastic with all seams taped and some additional protection on top. What he should not have done is try to "save time" by unboxing, installing and configuring it and expect it to work perfectly or that tradesman will call him if they're going to be doing work in the area. Or would even know WHO to call.
When you have construction done it is your responsibility to protect stuff in the area from that. If you are having your kitchen ceiling de-popcorned for example it is going to get dust everywhere, so either you cover everything yourself, you get a quote that includes that protection (with some sort of guarantees against dust getting inside your cabinets or whatever) or you plan to deep clean your kitchen and its contents from floor to ceiling, including nearby areas where the dust will inevitably go (or the whole house if you leave your furnace/AC turned on while they work)
Honestly he's lucky the tape drives lasted a whole year before conking out. I wonder if he tried to have them replaced under warranty, and what the vendor would do if they found construction dust inside them?
"Any idiot knows that if you put stuff in an area that's under construction, it is going to covered in drywall/concrete/etc. dust, paint, and potential impacts from various forms of debris. "
There's a reason why there is a sequence to doing things and why it's a bad idea to not follow that planning. At the point where electronics are being installed, construction needs to be complete and cleaning has been done and checked. Access control systems need to be in place to exclude people that shouldn't be in the space. If the contractor says they're done and uses the words "only" or "just", they aren't done. If they swear up and down that all of their work is finished, it should be checked against the project flowchart and their access removed.
AND, what's up with putting sensitive and mission critical stuff in a basement? Use the basement for stationary and cleaning stores since those are easily replaced and the company doesn't come to a grinding halt if there isn't bin liners on the premises. I recall a nuclear power plant that had their back up generators and gigantic UPS below grade, on the coast, where tsunamis happen. That didn't end well for them.
The best "construction/IT" conflict I can come up with was driving 150 miles to site, by appointment, to replace some networking kit and on arrival find it was in the attic, accessed via a hatch that the "handyman" had secured with some sort of home made lock and key mechanism[*] "for security" and he wasn't on site that day. It took 3 hours to track him down and gain access and I then found the only way to swap out the kit was to work on the ladder as there was barely room for the kit in the attic, never mind me as well. The job itself took about 20 minutes once the access was sorted. And yes, the job specification stated that access would be available.
* yes, tried all the tricks, but it was some sort or largish, sort of hexagonal almost but not quite Allen key, significantly larger than anything I had. Not even a large flat bladed screwdriver with packing jammed in would shift it. Even with the correct "key", it was hard to turn and unlock.
...builders.
I used to work at a very high profile art auction house in London (roughly 12 years ago).
I'd just finshed cabling the gallery up with approx 50 runs and it all came to a single truck at one point and ran up the back inside a plasterboard cavity.
Went home on a Friday and came back in on a Monday to find that the buildes had gone through it with a mini chainsaw.
They had been instructed to open up a hole in the wall in front of my cabling to recess a piece of "art". Being the lazy fucker that he was, he just ripped in with a chainsaw, not bothering to check for live wires, pipework or even worse...my cables. I was able to salvage 10 or so runs, but the remaining runs I had to completely redo...by this time the walls had all been sealed up so the job was at least 100x harder.
Back in the NetWare/ Compaq Deskpro as a server days, we had one particularly odious customer. Business was a small office with little room for computers and such so the server was on a HIGH shelf above a desk. The keyboard and monitor were on the desk below so when installing updates & software I had to get up and stand to insert and remove any disks from the server drive. Time consuming and a real pain in the back when working there. Coworker went on-site to do some work and was too short to reach the server even on tip-toe. He asked for a step ladder and was met by a string of expletives from the owner and was told to stand on the desk. Desk could not hold him resulting in him, desk and server crashing to the floor. Luckily he only had a few bruises but was greeted by the owner with another round of expletives. Coworker put everything back together and verified system was UP. From that day forward none of the support staff would work at this site and the support manager had to take care of any issues himself.
We had an issue with a construction crew starting to saw, sand and paint inside a computer room and had a serious chat with them. Next day we found all the server cabinets on the verge overheating because each was encased in plastic wrap to keep the dust out. Lucky we found the wrapped equipment fairly quickly. The computer room was operating in an ongoing building construction site but was supposedly cordoned off from ongoing works without special arrangements.
We had an issue with a construction crew starting to saw, sand and paint inside a computer room and had a serious chat with them. Next day we found all the server cabinets on the verge of overheating because each was encased in plastic wrap to keep the dust out. Lucky we found the wrapped equipment fairly quickly. The computer room was operating in an ongoing building construction site but was supposedly cordoned off from ongoing works without special arrangements.
It was my company who installed the telephone system and alarm in the new Apartheid Museum in 2004
The "Do not touch or tamper" sigh was written in English
The argument given by the cleaner who caused £15m worth of damage was that he was a recent immigrant from Mozambique and could not read English
I'm not Clarence, but that does not matter
This post has been deleted by its author
Somebody left a take library in what was still a construction site and then blames the conduction workers when dust gets into the tape library?
And that same person decided to try to clean out the tale library themselves rather than getting a professional maintenance technician in to do it?
Sounds like somebody is desperate to blame anybody else for their own errors.
Working. I send my four permanent field-hands down to visit their family in the Yucatan for a couple of weeks around solstice/new years every year. While they are gone, the Wife and I take care of feeding and cleaning up after the livestock, turning them out, along with everything else we do around here. This year, we gave 'em an extra two weeks off because they haven't been down there in a couple years (Covid).
Compound that with the weather, and ElReg has (mostly) been on the back burner.
Weather? In California? What's that? ... Well, today is only the third day without rain for the last three weeks. We've had at least half an inch of rain every other day. Several times we've had over three inches in a day, twice over four. In total, we've had well over half our average yearly rainfall since Solstice. Not a lot compared to some parts of the world, perhaps, but for California it's been wet.
I'll be picking up the hands at the airport tomorrow around noon, just in time for another storm to roll in. Hopefully we've managed to keep the place up to their standards, or they'll be giving me shit about it for weeks.
In all seriousness. I wonder if the cleaner in the SA story was illiterate?
This sort of issue can be a problem even in places like Europe and the USA (As I recall, there was an old episode of Quincy ME that centred around an industrial accident resulting from a worker not being able to read and understand a warning sign)
Illiterate people are not necessarily stupid and often develop very sophisticated coping mechanisms to hide their illiteracy from those around them so it is not always obvious that they cannot read. So in cases like this it is something that might well only come to light after the fact.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
