Vibrating walls shafted servers at a time the SUN couldn't shine If it's Friday – and absent some weird time/space slippage we're pretty sure that's the case – that means it is time for another instalment of On‑Call, The Register's Friday column in which readers recount their stories of the ups and downs of doing tech support. This week, meet “Rick”, who told us he spent the late 1990s “ …
Unlike the twin power modules, AC sequencer and extra power supply in those SUN boxen!
I am surprised this passed the applicable USA and EC tests. Though, on a second thought, Ultrasparc 2 - that is from the days before the the universal 100-240V power supplies when US vendors fitted a "Eu PS" which should have been called "Eu PoS" in their gear to sell over here. There were some Eu spec models which had 5%+ failure rate reaching 20% for the worst cases (some of the turn of the century dot bomb darlings who were unfortunate to get the early Flextronics PS for Eu).
I seem to remember something in ISO 9000 (1987 variation?) saying that 2.5% was an acceptable "normal" field failure rate, and 5% was acceptable after sourcing a new supplier. As long as you had it all documented properly, of course.
I always hated ISO 9000 ... chasing paperwork in the name of mediocrity.
Though, on a second thought, Ultrasparc 2 - that is from the days before the the universal 100-240V power supplies when US vendors fitted a "Eu PS" which should have been called "Eu PoS" in their gear to sell over here.
The VAX9000, power-hungry beast that it was, had a rather, ahem, unique power system layout. The power supplies inside the system cabinets themselves ran off a 280VDC bus, supplied from the Power Front End, a double-wide 19" cabinet, containing a transformer, rectifiers, control thingamajigs and several banks of electrolytic caps. Those were to provide a few seconds (!) of buffering in case of power glitches, such as when switching from utility power to a local generator.
For Europe they just added another transformer, in a cabinet the size of a VAX 11/750, turning 220V three-phase into 110V (the documentation states the power front end came in 110V and 220V models, but I've only seen the 110V model with that auxiliary transformer added).
the days before the the universal 100-240V power supplies when US vendors fitted a "Eu PS" which should have been called "Eu PoS" in their gear to sell over here
I worked in radio in the 1990s, and we used cartridge machines for playing jingles, adverts etc.
Apparently, just before I started, they had auditioned some machines from a US supplier who, instead of fitting them with 240V motors and 240V transformers to power the electronics simply wired the 110V motors in series with the 110V transformer and hoped for the best.
It sort of worked, until the current spike caused when the pinch roller hit the capstan and started pulling tape past the heads, at which point all sorts of nastiness ensued.
During my time there we used much better machines by Sonifex (YT clip), which had proper transformers (with mu-metal shields, to be vaguely relevant for a moment), proper power conditioning circuitry and low voltage motors.
I have a stack in the garage - anyone want one? Big stack of carts in the attic too?
M.
I remember back at Cheshire county council there was a line of sight network link to a school that was endlessly troublesome for no apparent reason.
It was eventually worked out that when the wind was blowing hard in a particular direction a rather large poplar was getting blown across the connection. Being the council they just did the necessary paperwork and removed the tree.
I seem to remember we had a similar issue with pigeons at a certain spread betting company, not quite as funny as the tech who kept going up to the roof to "inspect" the conenction during the summer without realising that there was CCTV up there and we all knew he was strpping down to his budgie smugglers and sun bathing.
An ex-colleague of mine was in charge of installing two communications towers with microwave links in Cairo for the Egyptian army to connect a couple of bases. All worked well for several years until his company sent him back at the request of the Egyptian army as communications between the two bases had suddenly stopped.
When he got there he went up the first tower to check line-of-sight to the second tower and found that someone had inconveniently built a new building in the way!
An ex colleague told me that at a previous job they'd worked for a multinational with HQ in the US. The firm had a line of site link between two rural premises and one day it stopped working. My colleague was on secondment there at the time and was dispatched with another worker to see what the problem was. The equipment was working fine but they soon found that a farmer who's land was in between the two sites had moved a largish tree to put up a metal sided building. The tree hadn't been a problem before but it had grown very quickly that year causing the line of site issues when windy. They couldn't tell him to move it back or cut it, it wasn't their land. They did go and talk to him and asked him if he or they could trim the tree. He said no to that but said if they moved the link away from the tree he wouldn't put anything else in the path.
Back in the late eighties and early nineties a friend of mine was living out in the sticks. The nearest neighbour was about 150m away over a small valley and looked like there was a small antenna farm on the land. He had a very large satellite dish and a couple of smaller ones these were all motorised from memory. There was also various radio aerials and a larger than normal tv one all making the place look unusual. I'm staying for a few days and they mentioned that they had issues with TV reception currently. Of the four terrestrial channels none of them came through clearly. It was alleged that this had only happened since the sat dishes went up although it might have occurred late at night as well. The suggestion round the dinner table was that he was a spy and maybe they should turn him in.
They'd never met him though as they'd only moved in 6 months earlier and frankly just hadn't bothered. So whilst I was there everyone went round to introduce themselves (and me) to him. He was very nice and obviously not short of a few bob as there were large televisions in each room. He had several satellite receivers in the living room and each one had various boxes attached. He showed that each tv in each room could receive signals from the satellite set top boxes. He explained he'd acheived this by hooking a(n illegal) (were they called rabbits?) transmitter to each box and tuning in that channel on the tv. He'd done this because it eliminated the need for rewiring the house.
My mates dad asked him if he could perhaps change the UHF channel numbers that he was using as they across the stream had no watchable tv channels at all. This was readily agreed to I suspect so he wasn't shopped to the authorities for using unlicensed transmitters. There was an interesting side note to this my mate discovered. If the set top aerial was pointing at the neighbours portable tv in the bedroom he could receive the satellite channels being watched. He found that he could receive Filmnet on one channel and watched some "very interesting" late night films.
My first job after uni, 1 building had an intermittent issue with people in some offices and floors complaining of slow downs and lost connectivity. Long story short, turned out to be interference from the lift motors. Re cabling avoiding the lift shaft permanently sorted that one. The previous guys spent over a year not figuering that out and I earned myself some browny points after sorting it in a few months.
If only all issues where as strait forward as that to prove, lift wasn’t used often and just needed someone to use it whilst observing users pc’s exhibit the behaviour also running a constant ping to see real time drops.
New industrial control unit sited within inches of the main three phase bus-bars for the entire factory (hidden behind a thin partition). Our first clue was when we heard people's phones didn't work in that area and an engineers laptop kept freezing, but was fine everywhere else.
Had a great one this morning, the electrical apprentice was supposed to turn off power on the electrical shop floor, instead he turned off the power to the whole premises!
At least we proved the UPS in the server room works!
I had an AS/400 once, the IT manager tested the UPS every week, it reported A-OK... Then we had a power outage and the UPS held for all of 5 seconds, even though it reported the batteries were okay.
Upon restarting, the bearings on the AS/400's DASD (IBM euphemism for a hard drive) had seized.
I have also been there, done that.
As an apprentice, I was asked to pull the fuses from a gantry crane; so I pulled the isolator, opened the box and unbolted the HUGE fuses with a spanner and pulled them out.
When the guy bossing me came to look he went white as a sheet; the isolator had been wired the wrong way round, and I had pulled those fuses while everything was live.
They had to shut down the sub station feeding the factory and re-wire the isolator to make it safe.
600V 300Amps I seem to remember.
When at Uni back in the dark ages I had a day where during one lecture the power was flicking on and off for about a second at a time every minute or so. Nothing was said about this until the power went out completely and we were instructed to leave the building in the same manner as if there was a fire. Very sensible and we all congregated in the parking lot. The power did not seem to be coming back on but neither did there appear to be a raging inferno consuming the building. So after a few minutes people started heading to the Student Union Bar or the pub. After a short while a bloke in a boiler suit appeared looking sheepish and wanting to speak to someone in charge. The head of facilities (having arrived a minute earlier) identified himself and was taken off to one side. Turns out the Electricity Board had disconnected the building by mistake they were supposed to have cut off a different one. They power had been reconnected but they had to go in and throw some seriously large switches (in the correct order) to get everything working again. One area still didn't have power and that left head scratching all round because there should have been juice everywhere. It took a tracing of the cables to find a hidden breaker in a cupboard in that area. No-one knew it was there - it wasn't on the plans and no one knew who had put it in.
It took a tracing of the cables to find a hidden breaker in a cupboard in that area. No-one knew it was there - it wasn't on the plans and no one knew who had put it in.
A few years ago I attended a mate's 50th birthday celebration, which was held in a certain large hotel in a certain large English city. They were having electrical problems from the moment we all arrived. It turned out the hotel had been built in four stages, from the pre-lekky Victorian era through to just this last decade. No-one had any clue about the power circuits, and for some unknown reason they were getting intermittent outage problems. The lights stayed on in the newly-built function room all night (and kitchen and bar, most fortunately) but not, variously, in the adjacent toilets, or in the reception area, or in the upstairs Victorian rooms.
They had the sparkies in all night trying to sort it out. At one point I went to my room just to be able to take a dump with some acknowledgeable degree of accuracy, only for the lights to go out mid-wipe. Thank arse-wiping-god for EU regulations on emergency evacuation lights!
Damn all those years I worked for Sun I was getting it wrong... and certainly by the time of the story it would be Sun. The original (well second really) logo was sun (lower case). I can't ever recall it being referred to as S.U.N. and I started at Sun in 1985. Most people don't even realise it's an acronym.
As for the node bit, I did say 'thought'.
Worked for a financial services company in the 9os. They installed a tape robot fed by three racks of tape storage carousels, a huge setup, and fearsome to watch, as the robot "hand" flung around, then ever so gently shoved the tape into the slot.
One day, the thing went wild. Tapes were all over the shop, the robot deciding to miss the slot like a comedian failing to eat an ice cream.
It turned out that they had failed to take into account the weight of all that kit on the fifth floor of the building, and xyz axes no longer held true.
"Strong enough field could easily induce currents that would basically scrap any data in RAM"
Though the better question is, would a field strong enough to garble the RAM not also have been strong enough to wipe the significantly-more-susceptible-then-RAM local HDDs in the server in the process? Presumably destroying the OS and preventing the reboot form happening.
For the electronics, it's not just a question of the field strength. It's more a question of the rate of change of the field.
For the hard disk, the write head is **very** close to the magnetic material. The magnetic filed falls off with the cube of the distance. So I guess it would have to be a fairly enormous MRI-style superconductor magnet that would be required.
I took apart some HDD, and was rather surprised to see that there was a very powerful magnet inside - right next to the platters, part of the head moving mechanism.
If that is the case, you must need a hell of a magnetic field to wipe a HDD.
On the other hand, I now have some bloody strong fridge magnets!
"Walls vibrating? Middle of the night? Was I the only one expecting something else about where the sun didn't shine? ;)"
Well, I was. Until I realised I'm the only one with plans to a diesel powered vibrator and that Dragon's Den weren't keen on it. Meaning the diesel powered vibrator remains a schematic in my cupboard, waiting for the day to be made a reality.
Icon because I'd imagine she may need it one day.
" I've suggested this so often - it almost designs itself..."Not just designed — built! I've used one in my misspent yoof!
Back in the early eighties, I worked on communications links for radio repeaters for public utilities and emergency services radios.
We had a rural repeater site on top of a hill, which had a microwave link to another repeater site on another hill about ten miles away. in between the two were a number of other ranges of hills, but all just low enough that line-of-sight was maintained.
This microwave link worked fine for a number of years, and then suddenly began to fail intermittently but always around the same time on a Thursday afternoon. It didn't happen every week, but say every two weeks.
After a lot of investigation, and a compete replacement of equipment at both ends, it was discovered that on one of the intervening hills was a small country lane which went over the brow of the hill more or less in line with the line-of-site link.
Off that lane was a sort of lay-by or turning circle, which was used by the local bus service to turn round at the end of one of their routes.
The buses would drop off at their last stop lower down the valley, and drive up and turn round, and then wait for half an hour before starting the next run.
Turns out that most of the time, they used a single decker bus on that route, but Thursdays were market day, so they used to put a double-decker on the route on that day. If it happened to park at a certain spot in the lay-by, it used to neatly break the line of site between the two repeater towers.
The only reason we found out about this was that we set up a temporary intermediate link (a van with a couple of dishes on it) in that lay-by whilst we were testing, and the bloke saw the bus come and park up whilst he was there, otherwise we could still have been looking!
The microwave antenna on both of the masts was at the very top of the mast, so couldn't be raised, but we managed to build an extension for the antenna on the other mast, and re-align the path just enough that it would be above the possible obstruction.
...but computers don't.
In the mid-90s I was hired as the new-fangled accountant for a family-owned factory that had never had an in-house accountant, thus no management accounts and no computerised data beyond the most basic of Sage ledgers. I set about installing networked PCs in each department's office to collect basic production data. This included some Thin Ethernet cabling. Once up and running I had endless problems with the PCs freezing, sometimes individually, sometimes en masse. Mid-afternoon was a particularly bad time. Turns out, it wasn't the cabling that was the problem, but I discovered by chance that around 3pm every day the large roof extraction fans in one (fume-filled) department were turned off, and I realised that this was one of the causes of the PC lock-ups, and thus I discovered the concept of "dirty power"... UPSs all round solved the problem.
Epilogue
My data analysis and investigations proved what the external accountant who had suggested hiring me had suspected - some of the company's activities were very profitable but others were making big losses, and the sales director was both incompetent and corrupt, and a very disruptive influence on the factory's efficiency. In short, they would literally have been better off without him! Unfortunately he was the MD's brother-in-law. My solution to that problem was to design a bespoke item-level sales order processing and allocated WIP stock control system that predicted and highlighted the knock-on delaying effects of any proposed changes to orders, and kept track of the actual effects of any changes. i.e. Name and shame, but subtly...
disk drives in one of the computer rooms would experience brief bursts of read errors during the day. Simultaneously. But not all of them, some appeared to be utterly unaffected. The read errors were diagnosed to be track misalignment errors, and when one of the field engineers noticed the building going *bump* being what triggered a slew of errors the problem was fairly quickly pinpointed to be trucks backing up into the loading docks below. The affected drives had linear head positioners aligned with the direction of the impact; those that had them at right angles didn't care.
Going back maybe 7 or 8 years, a friend of mine had a rack half-full of SAN boxes. 3.5" hard discs throughout for bulk storage of research data, so you're looking at maybe 120 spindles.
Friday evening, he gets an alert to his phone - drive failure. No problem, as the hot spare will pick it up and he can arrange the replacement on Monday.
On Monday, no servers are up, and the alarms are kicking off on the SAN everywhere. With no servers responding, of course, it can't alert him again, so he's gone in totally oblivious to the carnage that awaits.
It turns out that the bearings on a drive have failed, and it's just sat there vibrating. All weekend. And vibrating so badly that dozens of the other drives can get the heads to settle enough to read or write. RAID sets with part-commits. Loads of corrupted data. Basically, a single failed spindle took out all of the data on 100+ drives.
He had good backups, but had to beg / borrow drives from anywhere he could get them to get stuff restored.
.. and that, ladies and gents, is and example of why RAID is not a particularly useful way to maintain reliability. I've seen failed drives lock up the RAID controller, so that the array stops responding. Plus, generally you buy identical drives from the same manufacturer, and run them at identical temperatures with identical load patterns. Unsurprisingly, they all fail in quick succession, especially as the extra work caused by the array working round the dead drive hastens the end of the remaining ones.
"RAID is not a particularly useful way to maintain reliability"
RAID is a magnificently useful way to maintain reliability. It's just not a perfect way to maintain reliability.
Actually RAID may be a good way to ensure that when something does go wrong, it's been so long since anyone looked at the hardware that nobody can remember why they installed it in the first place. And that's if there's anybody there who has the vaguest idea what hardware is.
"With no servers responding, of course, it can't alert him again,"
And that is why you should have an external system monitoring and alerting, not rely on kit on just one site.
Working at BT we were responsible for monitoring all the systems connected to the NHS Messaging Service. As part of this we ran three datacentres that monitored each other plus other third party supplier systems.
When Buncefield oil refinery blew up, our systems alerted that Northgate's Newborn Hearing Screening service had gone offline. As the on-call engineer, I contacted Northgate to find out what had happened. They did have an RFC in place for server upgrades that day but their engineers had been delayed getting to site. This was fortunate for them because their servers were found 3 floors below where they had been.
"And that is why you should have an external system monitoring and alerting, not rely on kit on just one site."
This is where, back in the day, using external systems to monitor serial line console output was a good idea, aka "Out of band" monitoring.
The network stack could throw a complete barf, but the messages coming out of a bog standard serial line would still get to their destination.
@JamesPond
"And that is why you should have an external system monitoring and alerting, not rely on kit on just one site."
And that is assuming there is the budget and the equipment to do this. Most company IT systems don't have that luxury. And a big part of the problem, for a long time, was BT - let's be honest. Given the cost of leased lines back in the day, it was cripplingly expensive to monitor client equipment.
In 2000 we had a guy come in to demo Unicenter TNG to us. Nice South African chap. He tried to push it that we could use this to monitor a remote system over a simple leased line. Then we explained to him how much leased lines cost in the UK...
Somewhat different these days, but it still assumes that you have multiple sites to play with.
Scarborough, early 1980s. A big publicity campaign for BT's brand-new remote monitoring system which would trigger alarms if your house caught fire. A few weeks later there was a catastrophic fire at Scarborough's main telephone exchange - there were no staff on duty over-night and the on-site alarms didn't work...
As another poster said, RAID is an excellent, but not 100% perfect, way to deal with drive failures. Good RAID implementations that I have worked on do the following to try to address this issue:
1. Choose drives in different chasses (e.g. JBODs) for a RAID set. A mechanically failing drive's vibrations are far more likely to impact other drives in the same chassis than in the entire cabinet. That way even if all the drives in the chassis fail, no single RAID set is unrecoverable.
2. Power down failed drives.
3. Drives rarely (never say never when it comes to the ways in which drives can fail :-) fail mechanically without some forewarning. Mechanical failures often start with drive access time increasing (takes longer for the heads to settle) and software can detect this and take corrective action before things get really bad.
In the main article, I'm surprised the memory failures weren't more obvious, I would have thought the system would be warning of memory ECC errors (assuming by "RAM" they meant DRAM in main memory), some correctable errors and then an uncorrectable that brings down the system.
Where I currently work, we have an industrial facility that has a large central yard which has Wifi coverage thanks to four Cisco external APs installed high up on the surrounding buildings. These operate in Mesh mode (no data cabling). As a result, these things require pretty much uninterrupted sight to each other in order to function properly.
Unfortunately, this particular service yard has a number of large cranes operating in it at all times to move heavy equipment around.
Almost every day, we got alerts from our network team that one or more of the APs had temporarily dropped off the network due to the cranes disrupting the signal.
The solution? Silence all alarms on these four APs and accept that Wifi coverage will often be patchy. The company doesn't want to spend the money running cables up to them and patching them in properly.
Unsurprisingly, the network engineer who specced up this ridiculous configuration and didn't bother getting the APs properly cabled up when they were initially installed is no longer with us...
> Unsurprisingly, the network engineer who specced up this ridiculous configuration and didn't bother getting the APs properly cabled up when they were initially installed is no longer with us...
To be fair, if the company won't spend the money to cable up properly now, they probably weren't prepared to spend the money to cable up properly in the first place.
Would have been easy and of negligible cost if done from the off - cherry pickers were there already to install the APs, and power cabling had to be run to the APs as well. Would have been a minimal expense to run data cables at the same time.
Now, the cost is a lot higher as it means hiring new cherry pickers and getting cabling contractors back in for the job. That's why the APs have been left in their current configuration.
I worked at a dockyard years ago and they changed from some ancient UNIX system to a VAX/VMS system. As part of the rollout, they needed a hundred or so DEC VT220 terminals. These were supposed to be shielded...
It looks like the supplier tried to save a few quid and delivered the standard, non-shielded variants.
A couple of weeks later, a US warship in the harbour tested its radar and pfffzzzt! Dozens of dead terminals...
It took a heck of a lot to kill those DEC terminals.
We recovered a VT240 out of a burnt out computer room (electrical fault in the machine shop next door), case was a bit melted but for a laugh we plugged it in from a safe distance.
It came up with the standard VT240 OK message after running its self tests, albeit it was a bit dim.
At another client, we had a lightning strik and they lost a couple of hundred VT100s.
I went there a week later and they were all stacked along the wall of a corridor, waiting for disposal. They fell under the legislation that stopped hi-tech equipment being exported to the Soviet Block, so they had to be destroyed and certified destroyed.
They were taken to a near-by scrap yard and put in a compressor and crushed, with a government official looking on and he signed a certificate to say that they had been compliantly destroyed.
At another client, we had a lightning strik and they lost a couple of hundred VT100s.
Large warehouse facility. Probably close to a hundred VT100's. Serial cabling running to a Gandalf port switch near the systems. Lightning strike, right on the roof.
That was the end of the Gandalf and of about half the VT100's, most of which had spiderwebs of circuit traces holding pieces of charred circuit board inside.
At another client, we had a lightning strik and they lost a couple of hundred VT100s.
I had two defunct SuperBrain computers in the boot of my old Ford Cortina Mk3 when it got nicked at Arnos Grove tube station (this was back in the late 1980's - I'd been given the Superbrain computers by someone who knew that "I did computers" in the hope that I could get them working. With some work I got them to boot to the stage where they were looking for the OS on one of the floppy drives. Given that I didn't have said OS (and neither did the person who gave them to me - they had been chucked away) I decided that it was futile working on them and put them in the boot in order to take them to the dump).
Someone nicked my Cortina from the tube station in order to get home (it was raining heavily that weekend and, according to the police, things like that happened fairly frequently). No damage to the car - it was already pre-disastered so a few scratches round the you-could-open-it-with-a-screwdriver lock didn't matter. The ignition lock was equally as worn and you could pretty much turn that with a penknife or screwdriver too. So the only effect other than a day or so of wondering whether to get another car was that I no longer had to worry about going to the dump.
I'd love to have seen the face of the thief when they realised how utterly valueless the Superbrains were. They probably thought that they could get rich off a couple of computers!
Back in the 1960s there was a KDN2? industrial control computer that had been salvaged after a fire at the customer's site. It sat outside rusting away for a while. Finally it was brought indoors and powered up. Whereupon it printed out the date and time of the failure and merrily continued running.
Those machines had a very large Ni-Cd battery in the cabinet. The terminals were large threaded studding with nuts to clamp the cable ends. It was said someone once accidentally bridged the terminals with a spanner - welding it to the terminals.
Ironic how your tools can be the most dangerous thing on you some days. I worry about a stray "rm", but some people have to worry about wrenches blowing up half the state.
http://www.encyclopediaofarkansas.net/encyclopedia/entry-detail.aspx?entryID=2543
Puts it in perspective.
"It took a heck of a lot to kill those DEC terminals."
And maybe other DEC kit. Back in the '80s of Big Bang London there was the story of DEC being asked by a non-customer if they could supply a copy of the OS. They asked what it was for. Someone had found a presumably surplus Micro-VAX tossed into a skip.
"Back in the '80s of Big Bang London there was the story of DEC being asked by a non-customer if they could supply a copy of the OS. They asked what it was for. Someone had found a presumably surplus Micro-VAX tossed into a skip."
It was around that time that second hand MicroVAXes could be had for as little as £10K, and my thoughts turned to setting myself up as an independent developer.
It turned out that these were extremely hard to come across, because when the initial purchaser upgraded to something else, the MicroVAXes would get handed to another department within the same company, rather than released onto the second hand market.
The site intruder alarm would randomly go off 4AM.
Happened a few times and I was getting tired with the freezing early hours motorbike trip and we were at risk of losing police response due to the false alarms.
Long story short there was a bad joint on one of the alarm cables but it needed two things to trip
1. being bitterly cold, (maybe with dense air).
2. the guys in the food delivery company next door slamming big van doors as they left at 4AM for the first rounds.
We had a similar problem at an office I worked in. The intruder alarm would go off on cold mornings shortly after 05:30. We also hit the issue of police refusing the call outs until someone noticed that the large world map hanging in the sales office was right over a radiator.
In certain conditions when the radiator warmed up the map would waft enough hot air to trigger the movement detectors.
Electric macerators in the ladies toilets; electric kettles in the operators rest room; bridged dirty/clean earth on the false floor supports; vibration from Harrier jump jet landing/taking off nearby.
All diagnosed by chance observations or casual comments.
An interesting one was the remote network node that crashed intermittently - then eventually restarted itself successfully before the engineers could get there. Always about the same time of day. It would go weeks without the problem - then do it on several days in a week.
The node cabinet was about the size of a desk - in a small room with external windows on one side. It took a year to diagnose - the coincidence of factors was as follows:
1) the engineers occasionally leave the cabinet door detached after maintenance.
2) for a short period the passing sun can shine through the window and illuminate the cabinet's opening.
3) a board in the cabinet has a cracked DIL chip that distorts in the sun's heat - temporarily breaking a pin contact until it cools down again.
4) It is in Manchester, England.
The KDF9 mainframe at a university was being put through its acceptance trials. The tests kept failing at about the same point every time. Finally someone heard a slight "crack" sound from the papertape reader which was currently running through a large reel of tape.
The large tape collection bin was metal - and it hadn't been earthed. The static build up finally reached the critical sparking point at about the same point in the test every time. The discharge glitch crashed the mainframe.
Talking of trials, we had a mainframe vendor deliver a machine for test. We were looking at swapping to it from a cluster of VAX minis.
The salesman was full of bravura and confidently gave us a tape with some FORTRAN source code on it, which we should load onto the mainframe and one of our VAX computers to run some benchmarks.
He then told us to compile it normally on the mainframe and "use all optimization that the VAX compiler can manage." He then got up to leave and told us to call him in about a weeks time, when the mainframe was finished, the VAX would need about a month.
Half an hour later, there was a note waiting for him in his office to call us, the VAX was finished, the mainframe merrily chugging away.
It turns out, the FORTRAN compiler on the VAX was a little smarter than the one on the mainframe. The source code had no input, created a huge multi-dimensional array and filled it up with numbers, the source code had no output. The VAX compiler therefore optimized the FORTRAN code down to a signle NOP instruction and linked to an empty executable, which finished in 0 seconds.
"The VAX compiler therefore optimized the FORTRAN code down to a signle NOP instruction and linked to an empty executable, which finished in 0 seconds."
The VAX-COBOL compiler wasn't anywhere near VAX-FORTRAN in terms of optimising code, but it could still surprise you.
On one occasion I was trying to do something similar with large arrays, but the COBOL simply decided that anything not referenced wasn't required. Passing the appropriate variable(s) to an external routine brought in at link time was the only way to fool the compiler.
The Motorola 6800 microprocessor evaluation kit in 1976 came in a large box. The actual DIY circuit board was quite small - and the chips were shrink wrapped on a piece of cardboard. The manual was a large format tome at least two inches (50mm) thick. It was a comprehensive introduction to using a microprocessor to implement all its own peripherals with the minimum of external electronics.
Who remembers the "Blue/Orange/Grey Wall" ? Your little MicroVAX would be delivered on one pallet, the VMS documentation kit would turn up in another.
V6 was the beginning of the end, when they started shipping the "White Shelf" paperback manuals. Nowadays, you have to check the box in case you accidentally threw out the CD.
Your little MicroVAX would be delivered on one pallet, the VMS documentation kit would turn up in another.
An educational institution which shall not be named had, back in the days, several tens of mVAX workstations, as well as a fair few big'uns. At some point they ordered OS upgrade kits for the lot. Licensing required you to order one upgrade per system.
The order number they used was for the kit with full VMS documentation.
@Chris King my thoughts exactly. Walls full of manuals. Then there were the software manuals. We had an in-house oil survey system (DISCO), whose programming manual was about 3M thick.
I vividly remember the orange manuals, the grey ones as well, but I think the orange colour makes them stick out more in my memory. But we had "real" VAXes back then, 11/750, /760 and /780s, reel-to-real tapes drives (about half a dozen for each VAX ) and removable hard drives (you rotated out the platters in a sealed, clear plastic container).
My first experience of instant messaging was the VAX phone utility, sitting in rural England and chatting to operators sitting in a computer center in Houston, Texas in 1980. Forget WhatsApp and Slack, VAX Phone was where it was at!
I remember forgetting to log off my terminal when I went to the toilet, when I got back an OP had left a note on my screen saying "repeat 1000 times, 'I will not leave my terminal session unattended.'" I used the Phone utility and called him up and then piped a file containing that sentence 1000 times to his terminal! :-D
"First job one of my duties was to insert the updates in both the Vax/VMS manual set and the PDP-11/45 set [...]"
Had a similar chore as a junior programmer on an English Electric System 4 when the range was first introduced. We had a cupboard with a full set of manuals - mostly covering software which our system test environment didn't use. Every week there would be a sheaf of updates. Most of them pointed you to a specific page and wanted you to hand correct something with a pen. Occasionally there was a replacement page.
We soon realised that if we did the updates about once every six months - there would be a completely new issue of a whole manual's contents somewhere near the top of the pile..
Nowadays, you have to check the box in case you accidentally threw out the CD.
Around a decade ago I was setting up a device for a customer. I cannot recall whether it was a network firewall or some USB device, this particular detail has long escaped me and is not important. What is important, however, is the instructions came on a CD.
I kid you not, the only file on the CD was a shortcut to the manufacturer's support website.
in the print production department. It had worked OK for years, cover on or off, but it suddenly started failing and freezing up whenever the cover was lifted to refill the hopper.
Turned out that the tube light overhead was so old the phosphor had worn off in several places, and the diffuser had long ago disappeared. Just enough high energy photons were hitting the surface of one or more of the control chips to induce a photovoltaic effect whenever the cover was open for a long enough time. Changed the light in the room and all was well again. I feel sorry for the operator, though, and their unintentional exposure.
Years ago I worked for an SME that sold accounting systems to legal firms. One particular client in Aberdeen had a Netware server on a co-ax network (I said it was a while ago) that would go offline at "random" times during the year. The only trend that had been identified was that the comms problems roughly followed the sunset. The head of the our particular I.T. support team actually told the client it was sunspots causing the issue.
The customer wasn't particularly happy but asked about shielding options so I was sent to evaluate the office for a shielded cabinet to store the server in.
I arrived about 4pm, found the server was sitting on the floor next to a window. I had a look out of the window just as the pub below switched on their neon advertising sign. Hey presto, the Netware server briefly lost all connectivity as the sign flickered..
30 minutes to down the server and move it to another TAP and the problem was solved. Sales weren't happy...
The strangest one I've seen was a network router in our office which rebooted itself in accordance to the published times of dawn and dusk then got stuck trying to reconnect to the ISP.... it took our head techie a number of weeks to work out that these times matched the switch on/off times of the street lights outside the building.
Apparently the electrical surge in the street light power cables which ran under the basement where the router was installed had a bit of a kick
Worked at a place some time ago where the main file server was Novell and had constantly failing drives, replacing at least 1 a month with severe issues and cost waiting for engineers, rebuilding raid etc.
No-one could figure out why until we put a few disparate pieces of info together.
The office had flats all around it and a big aircon unit on the roof, turned out the locals were complaining about the noise overnight so the security guard was switching it off each night and back on again in the mornings before anyone turned up for work.
The drives heating up and cooling down were failing due to thermal stress.
Said guard had a stern talking to and the locals had to suck it up until the server room was relocated sometime after.
I think "Tivoli" means a fun-fair / "amusement park" but not in English?
Tivoli for most people in Scandinavia is rarely the regular fixed-in-place amusement park.
It is mostly a mobile variant, where the various rides are designed around large trailers, and everything is set up for a short time on a vacant lot, a football pitch etc. It's usually powered by a huge generator.
Most cities and town in Scandinavia are too small to keep a full-time amusement park constantly in business, so this has become a business niche. There's only a handful of these tivolis around.
worked at airline and handled it stuff (wasn't my primary job but eds was few hundred miles away so...) and kept having issues with the server shutting down.
due to being old hangar server cabinet (locked) and hangar mtx guy shaed same room.
when he left at night he turned lights out.
eds was supposed to plug cabinet/ups power into dedicated non switched outlet.
course they plugged into one easiest for them to reach which was switched off light switch.
didn't take long to figure out, walked in one night found lights off and ups alarm screaming. turned lights on screaming stopped.
In the late 1990's working for BT, we were developing various systems for the NHS Messaging Service. We had two DEC Alpha servers in a back room 'server and failover centre' using portable air-con units with tubes stuck out the windows to try and keep the servers cool. We put in several requests to the management to install proper air-con while we developed the solution, but as the building was rented they refused as it would be 'too costly'. After 2 years of development, the servers were shipped to our proper data centre and the system installed. After about 1 year of operation, both DEC Alpha servers started randomly shutting down and/or rebooting. We called Compaq out under the service contract and they reported that the motherboards were warped along with the RAM. They further informed us that the log files showed the servers had regularly exceeded 40 deg C over their first 2 years of life and therefore neither server was covered by the service contract. Ended up costing around £100k to replace the two servers. Cost of air-con would have been £15k.
Whilst working in IT Support at a local hospital, I got a call from a secretary at one of the other sites. She was trying to install Office 1.0 but the installation kept getting to disc 21 and crashing.
So I picked up my rucksack of discs and drove across to the other site. I started the install process running and started swapping floppy discs. All was going well with my set of discs and the secretary asked me if I'd like a tea or coffee. "Coffee please", she switched on the kettle, the PC blipped and the installation failed at disc 21 again with "unable to read disc" error.
This is a dedicated machine built around a Nova computer ( i'm talking mid 80's) used for IC layout.
This machine has a couple of diablo disk drives and is hooked up to specialistic terminals (Calma)
At a certain point in time we are baffled by the following : every thursday morning the machine is found to have gone through a full reset. Service is called. They are baffled too. These machines do not a have a watchdog , nor a realtime clock and there is no known software that can cause a timed reboot. One wednesday evening the machine is booted from a clean diskpack and left in a steady state with nothing running on it . And then the wait begins. Sure enough at a certain point : reboot.
Every possible avenue is tried : back-up UPS investigated, computer investigated , peripherals investigated.
Ultimately the diskpack is sent off to the manucfacturer where they insert it in a brand new machine, wait for wednesday/thursday turnover and nothing happens over there.
suspecting a serious hardware problem in the machine they ship an entire new machine. Machine gets installed, wired in booted and then the wait begins. Come wednesday evening ... nothing happens. All is well. Next wednesday .. all is well. Looks like the machine was faulty. Then the week after : it starts again.... So : someone camps out in the computer room ( the fishbowl we called it. a room surrounded in glass panels , with an adjacent room where the line printer and the large format flatbed plotter ( we are talking A0 sheets here ) is. To get into the computer room you need to go through the printer room and only select people have access to the computer room.
come wednesday evening late at night : the cleaning crew comes in. One lady opens the printer room door, plugs in the vacuum cleaner in the socket over there and proceeds to vacuum. The Nova reboots.
How can this be ? the fishbowl has its own supply , on a double redundant UPS, isolated from the rest of the building. That socket is not in use for anything else in the computer room. We even trace the wiring and the fuse feeding that socket is not shared with anything else. It is just a utility socket for just such purposes : provide 'dirty power' if any work is to be done.
So we lift the floortiles ( raised computer room floor ). Lo and behold: there is the cable connecting the plotter to the computer. 1 inch underneath the power socket.
What happened : switching on the vacuum cleaner caused a spike on the comms cable between plotter and computer causing the computer to freeze and reboot. We had changed cleaning crews a while ago. They were given explicit instructions not to plug anything into the orange sockets around the computer room ( clean power on ups ) , and this was a white socket. It turned out the trouble started after the new cleaning company took over. Then why did the new machine work fine for 2 weeks ? the vacuum lady was sick and her replacement simply used an extention cord to reach that area.
A customer's mainframe had started to crash intermittently. It was decided to investigate properly when the engineers had their Saturday long maintenance slot.
Even though we were software support we had a suspicion that the culprit would be electrical noise from either the card reader or a tape deck. The customer had an exceptionally long row of tape decks - with new ones recently added.
We went along the row - opening and closing the door on each one to automatically load a tape. Sure enough one of them reliably crashed the mainframe after a few iterations.
When the senior engineer was free of his task of installing yet another new tape deck he came to see what we had found. Open/close tape deck door - no crash. It was clean - no problem.
Much scratching of our heads - and the engineer went back to his task. We tried again - crash!
This time we showed the engineer - and asked what he had done in the intervening time. All he had done was put a false floor tile back in place - which he had lifted earlier to facilitate threading the new tape deck's cable.
The tile was at the start of the row of tape decks. At that point there was a crowded run of thick tape deck cables sitting on top of each other. In each of the cables was a large metal junction box - and one of these from the previous week's tape deck addition was sitting on the pile of cables. It was so high that when the tile was in place it pressed on it slightly.
The false floor was an unusual design. Instead of "mushroom" pillars at the corners - there was a lattice of unpainted steel bars supporting the length of each tile's edges. The undersides of the tiles were reinforced with an unpainted steel sheet. The lattice was connected to the building "dirty" earth. Thus there was a connection between the lattice, the steel sheet, and the junction box - bridging "dirty" and "clean" earths.
I once designed & built a rather substantial radio station - a long time ago. One of the more artistic directors demanded remote controlled dimmers in the studios, so I obliged. All was well for about 18 months and then the rumours of a ghost started. The building was a couple of restored Georgian townhouses c.1740s and the stories started to gain a little traction. Turns out the rumours centred around the behaviour of the lighting in one of the studios, specifically the circuit with the remote dimmers.
The lights would fade up an down mysteriously at certain apparently random times and some folks had mischievously associated this with the topics being discussed on air. Took me a little while to catch it in the act, but eventually I tracked it down. The remote controls were infra red. The studio faced West and just at that point in late autumn the sun would set between two buildings across the street. If it was a clear sky then the low evening sun would trigger the dimmers to ramp up and down.
It hadn't happened the year before because the autumn sun had been blocked by a large factory building that had recently been demolished. I was pleased to find the answer, but it was a shame to spoil a good rumour.
Years ago I was a radio tech and had a call about a station in an office that would lose connection to the system every 4th day. Took a bit of time to realize the problem was RF interference on one of the four control channels (trunked radio system). Sweeping the building for the offending signal was quite impossible as the signal appeared to be coming from everywhere, even with a directional antenna. After a couple of days, I finally figured out the signal was coming from the building computer network and was being amplified by every printer's NIC in the building. Disconnecting printers one-by-one finally revealed a bad JetDirect card in a plotter. Once I let the customer know what the issue was, they remarked that they were getting rid of that particular plotter in a few days. Had I not found the cause of the issue, it would have magically disappeared and left me boggled.
I used to work as a flight simulator engineer and the company had a new 3-axis flight sim installed. Commissioning of the whole setup had gone well and we started to get pilots coming in for training. Then after a while we would keep getting alarms that the flight sim had crashed and the only thing that could be done was to reboot the real time mainframe (yes it was that long ago). This would take about 15 minutes as we would do a thorough check that all was OK onboard the flight sim before handing it back to the crew.
The random crashes would occur at various times and try as we could we were unable to work out why this was happening until one day someone was updating something on the mainframe whilst the flight sim was being tested during the maintenance window and opened the rack door. Suddenly the flight sim dropped down to its rest position. Some further testing traced the fault to faulty cable connectors from the IO board to the mainframe. Needless to say, a new cable loom was ordered and fitted with all wiring tie wrapped to the rack posts and the problem went away.
Can't speak for the OP, but as far as I know, due to elevator designers not being total numpties, elevators typically get counter-balanced with half of their maximum load, so either full load or zero load are both worst cases vs. the typical half-load which might barely exercise the motor. Not to say that was the case - maybe any elevator motion triggered the problem but they only used it at night for freight...
"I don't get how elevators shunting a dozen people at a time are easier to deal with than elevators shunting a few garbage bags full of coffee cups and snot rags."
Well, first, it was a "service elevator", which is a large lift used for moving large items. Usually not used for moving passengers, usually not located conveniently for passengers, and many companies forbid casual use for passengers.
And if people ARE using it casually, it's not convenient to the offices, so it's probably one or two guys in the shop not wanting to take the stairs or walk all the way to reception and back.
So it probably doesn't get used much. At most it gets used a lot for items that won't fit easily into a passenger lift but aren't particularly heavy.
OTOH a large building could need a large cleaning crew, each member having his or her own cart full of equipment, and possibly some heavy polishing equipment. It adds up.
The legend at Brutish Rail when I worked there was of the UPS's at King's Cross station that had been (accidentally?) wired to the same feed as the trains used. So every two minutes there was a massive line drop and then an equally massive surge as another train braked coming in. Needless to say batteries in said UPS's lasted a matter of weeks.
Experienced it in a smaller way when I supported a client right next door to a steel works that had voltage going anywhere from 150 to 260 during the course of an hour as steel was poured. Not helped by the client also running some pretty high instant draw machinery themselves.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
