Fix five days of server failure with this one weird trick Welcome to another in The Register's series of confessions from readers who were either possessed by the pager or all too happy to fire off a demand for On Call support. Our story, from a reader Regomised as "Will", takes us back a quarter of century to the back office of a well-known UK bank. You know the one – it had a …
This reminds me of a time in my early days tinkering with electronics and pcs, i was literally a little more than a toddler, and my dad had a small pc selling and repairing business.
One customer brought his pc in for support, because it was "randomly freezing and had to be powered off". It was a big full tower with 5 5inch bays and 2 or 3 floppy bays in the front. My dad was trying to investigate a software issue, after all these were the days of the freshly shipped Windows 95, but it seemed nothing was out of place, all the drivers were looking fine and a fresh re-install of windows didn't solve the issue.
Meanwhile, being a curious creature, i crawled under the desk to look inside that chungus while the case was open (no shiny glass side panels in those, the two sides and the top came out as one piece)..... and EUREKA, the little cpu fan (think Pentium I 133MHz) was not spinning.
A quick replacement, et voilà, the machine got back to full performance.
I even had to make the call to the customer telling what had been wrong with his pc.
I miss those days, now i'm a sysadmin in a medium-to-large company and i have responsibilities... oh well, TGIF, cheers!
We used to discharge large capacitors on the rail of my high school's inner balcony, during the recess between classes. Made for a nice soccer style "wave" once you waited long enough to have about 80 other people laying on it to watch the playground below.
Ah, the memories...
When I was a young man, I worked at a as station as an auto mechanic in the days of the old points and condenser ignitions. We would use a spark plug tester to charge the condener and leave it on the work bench. Fun days whenever we had a new hire to retrieve said part.
I worked with a guy who went to electronics school back when an electronics lab was full of tubes (or valves, if you live on the same side of the Atlantic as The Register's home office).
Their favorite prank was to suddenly spin around on their lab stool and toss a tube to a classmate.
If said classmate had poor hand-eye coordination, they'd have broken shards of glass to clean up at their workstation.
If the classmate had *good* hand-eye coordination, you'd repeat the exercise sometime later, but world replace the tube with a charged capacitor.
My coworker wasn't sure how much he learned about electronics that term, but he did improve both his hand-eye coordination and his target identification skills.
In my case fingers and pain => don't try to catch a dropped soldering iron.
Or pick one up by the hot end, as you distinctly (& correctly) remember unplugging it 3 hours earlier while working in your hotel bedroom, but failed to account for housekeeping plugging it back in when you left for lunch (To help the still extremely hungover\sobering up process) instead of the mini-bar.
I have spent this afternoon in the school prep lab testing Physics’ class set of soldering irons, 3 sorts: ordinary plug into socket, plug into power supply with helpful two indicator lights or comes plugged into a dumb brick and you have the slide a plastic slide over the button to keep the thing on.
They got colour coded: Red: can’t even melt lead free solder, Blue: can sufficiently melt lead free solder but not proper solder. Green: can sufficiently melt proper solder AND of course lead free.
So I had a mix of irons heating and irons cooling down. I was not burned once because I was careful. The rubber feet of the useful power bricks were perishing though so my bench and my hands got covered in sticky black gunk. I levered all the damn things off and chucked ‘em.
Icon mine is the lab coat and that is me taking it off at the end of my day. I took a bus into town, did some shopping and still got home before 17:00.
They got colour coded: Red: can’t even melt lead free solder, Blue: can sufficiently melt lead free solder but not proper solder. Green: can sufficiently melt proper solder AND of course lead free.
All the lead free solder I have ever encountered had a significantly higher melting point than good old 60/40. Are you sure you got those concepts the right way round?
In my experience, lead free solder doesn't really melt, it goes a bit squidgy. I don't have to worry about that kind of thing any more, being a senior design engineer. Soldering is a technician's job. I was never that good at soldering anyway, even when the stuff melted properly. It is I think permissible to use the old 60/40 tin-lead solder in prototype workshops. I think we have still got some at work. But I read that you should not mix the old type of solder with the new, because of the formation of intermetallic gubbins, that can cause brittle joints. Being a senior design engineer, I research these things. One of the tricks to living a long life as a chemist is to delegate the practical work to technicians, who preferably work somewhere far away from you, such as another country.
I once had a particulary nasty iron with a hook and no other means of putting it down. One day while working, the phone rang so I absentmindedly hung it on the open chassis containing the pcb. On finishing the call, I leaned forwards to inspect the last leg of the chip I had soldered in - and promptly stabbed myself in the middle of the forehead on the hot tip of the iron. It only took 5 years for the scar to eventually fade away. Needlrss to say it was a wake up call and I bought a new iron at the earliest opportinity.
Ouch!!!
I knew a girl at college (a billion years ago) who would solder something, then tuck it behind her ear, the power cable dangling precariously.
She must have been into witchcraft because all the times I saw her doing that, the heart-stopping scream never happened. Not even singed hair. I have no idea how...
That sounds like Weird Liz, at work. A woman of phenomenal practical abilities, but rather odd taste in clothes and hairstyle. Liz is from Austria, and her accent is uncannily close to that of of Arnold Schwarzenegger. But she does not look anything like him, being of quite slim build. She went to work in Norway, but it did not work out. Maybe the Norwegians thought she was trying to invade, or something.
You just had to bring back that memory. The cord was tugged little brother 3-4yrs old. The iron was destined for the kitchen's new linoleum floor. I reacted with catlike reflexes. I caught it, by the hot end. I realized quickly enough to direct the pain reflex to send the iron towards the slate floor, but not abort the catch or retarget for the handle.
Arms and pain is why I still remember that CRT's retain charge even after the device they are in has been turned on for about 2 seconds & then unplugged.
That and picking myself up from the other side of my bedroom.
Thanks for writing most of that for me - Have a pint of liquid painkiller.
One of my early attempts at fixing electronics was on a valve amplifier, which a guitarist at school used to make music that was almost nothing like Eric Clapton. I thought I had done all the right things, like unplug the amp from the mains, and not just switch it off, before removing the lid. There was a fuse mounted on the chassis, so I thought I would pull it and have a look. AARGH! It still had hundreds of volts of DC on it, from the HT capacitors.
Had something similar when I was younger. I would have been around 14 years old and during one school holiday, my dad took me into work with him for the day. Most the kit there was SGI Indigo2 workstations (lovely machines back in the mid 90s), but they also had a Win95 PC in the corner of the room for running some applications not available under IRIX.
The Win95 PC (a Pentium 200 if I recall) was universally hated amongst the staff as it was very unreliable compared to the SGI workstations. I used it a bit and agreed. In fact, it was unreliable to the point that something had to be wrong with it. On a whim I popped off the case and pointed out the dead CPU fan that was causing the thing to repeatedly overheat and crash.
We shut it down and popped off the old heatsink and you could actually see the CPU's model number on the scorched heat-sink it had been getting that hot. With a replacement cooler fitted, it ran fine after that. I was actually quite amazed that the CPU hadn't been damaged by the prolonged heat if I'm honest.
I once had a client's PC showing typical overheating behaviour (working fine first thing, but crashed around 10am)
Took the cover off (honestly, isn't that what you do with ANY problem?) and could see that the fan had seized, but then the heatsink had got so hot that it had melted the plastic clips that held that fan in place. The fan and plastic frame were hanging off the heatsink to the point that even if the fan had spun it wouldn't have cooled much.
I didn't have appropriate spares with me so I replaced the fan and cable-tied it in place as a 'temporary' fix. That lasted about another 3-4 years until the PC was replaced.
"That must have been a very old CPU...I have a Duron machine, and when I first tested it the thermal protection was tripping all the time. Just took a little bit of new thermal paste of course."
I remember this video: it was comparing a Pentium 4 "Presshot" with whatever AMD was making these days. An Athlon XP?
It was a Tom’s Hardware Guide video. I still have a copy saved somewhere. Upon removal of the heatsink, the Pentium MMX soon locked up (but was fine after heatsink replacement and reboot); the Pentium 4 slowed down but kept running; and the Athlon locked up and released the smoke.
At the fledgling Online Media department of a large publisher we had a pair of SGI Indy workstations - one as the staging web server and one as the production one. This was 1995, so we were talking Netscape's web server package and pretty early adoption.
This was before HTTP 1.1 was introduced (and hence no host headers), so if you wanted to have multiple web sites on a server you needed multiple IP addresses. And at the time you could do virtual IPs on the Indy OS and persuade Netscape to run multiple sites, but it was proper messy. We made it work, but feared that the slightest wrong move would suddenly make it decide not to.
It was at a University and they had a virtual reality centre, so they were mainly used for 3D modelling - although the sysadmin had one as well. They also had a few labs of Indys and O2s for teaching the students.
Incidentally, I also collect them and have an O2 I acquired from the university (when they were being replaced with newer Linux PCs), plus an Indigo2 and a Fuel. I used to have a few Indys from the uni as well, but those were sold a few years back.
We've got some old (very much out of warranty but still going) Precision workstations that have once or twice in the past couple of years booted with a cpu fan error. The fans are magnetic bearing (they're big!) and spin freely, but occasionally seem not to start. Power down, open up, a little nudge, power on, everything is good again. (I suspect a brush issue, but not really worth replacing at this point, especially as ebay isn't exactly an approved supplier.)
I had a machine that I'd built myself, and often tinkered with. One day, I decided to fire up the new Unreal Tournament, but as soon as the game reached the menu screen the whole computer would crash. I tried with a few other games, and they all had the same problem; as soon as they started the game, the PC would reset.
Eventually, I realised that this sounded a lot like a thermal problem, so I opened the case to see what was going on. Turned out, in my last exploration I'd accidentally left a wire loose, and it had somehow got sucked into the CPU fan, jamming it. The heatsink (a fancy Zalman one) was big enough to passively cool the CPU when it was idling in Windows, but as soon as a game forced it to run at 100%, it would overheat, and thermal shutdown.
I'm always careful to check that all the fans spin freely before I close a case up these days.
God you make me feel old....
When I was little more than a toddler PC didn't exist, the first Commodore PET was still years away and TVs (my dad was a TV repair engineer/technician) were all valves (vacuum tubes for those on the other side of the Atlantic). Not even discrete transistors, never mind ICs
Colour TV was something few could afford to buy and most were rented...If they could afford that!
Never seen a server with a separate "power brick", but I'm surprised they didn't think to swap it out sooner when they had swapped out everything else. Old school SCSI gear was surprisingly finicky about power.
Though I have to admit I was expecting the tale to end with everything having been replaced except the SCSI cable, and one pin was going to be ever so slightly shorter than the rest. Having a pin that gets "pushed in" just a bit, rendering a cable as working but not 100% of the time, was something I ran into more than once back in the day. Though IIRC with Ultra Wide SCSI cables that used more / smaller pins not the original brand of SCSI with the stout pins where it would be easier to notice when something like this occurred.
SCSI cables always seemed to be as pernickety as a Siamese cat. Unless they were seated Just So they would sit there sulking until they were.
Centronics cables, on the other hand were made of sterner stuff. I say someone perform the impossible task of putting one in upside down - it required considerable force and a complete lack of mechanical sympathy, but they managed it. After realising their error they pulled it out, fitted it in the correct orientation and carried on printing.
I received an Adaptec SCSI controller and 3 Quantum Fireball SCSI drives (20MB, 20MB and 40MB, ISTR). A friend had had them and had tried "everything" and it was "just broken" and he was throwing them out. I grabbed them and plugged them into my PC and had a little look.
He had missed the terminator on the Adaptec controller for the second SCSI bus. I plugged in a spare terminator I found at work and it worked fine. A real stonker of a set-up!
You can actually tall peoples age by how hard they laughed at that story.
I managed a tech support team in the 90's my back ground was mainframes but within a few months even I was reminding techs with scsi problems to check the right terminator was fitted to the unused channel. I seem to remember that there were variations between suppliers in the early days and whilst the 'wrong' terminator would fit it could cause some very strange behavior, And don't get me started about APC UPS interface cables
Was there something about single-ended and double-ended terminators? I wasn't really a sysadmin but was involved in a large (at the time) migration of an Oracle db from old raid-5 disks to new super-dooper smart raid arrays on a "massive" HP T500. We had a whole night to do it but the new raid arrays wouldn't connect. Finally fixed the problem just before the takeaway closed, so we could enjoy our Chinese while the DAT tape slowly restored the database.
When I was working at a publishing company, I got friendly with the IT Manager (I later became his Deputy, but that was then still a long way in the future). There were several old 386 laptops on his bench, and one of them had "BER" written across its lid. Being a bit nosy, I asked him what that meant, and he said "Beyond Economic Repair", and said that only one half of the keyboard worked. I asked if I could try to repair it, and he said, go ahead, if you can fix it you can have it. I took it home and opened the case, only three of the four keyboard printed ribbon cables were connected, the fourth had snapped off flush with the push-in connector on the motherboard. I teased the broken bit out of the socket, stripped back the insulating varnish on the ribbon cable, and shoved it back into connection. That computer is still working today, 18 years later.
I worked testing big iron Sperry 1100/80s. The card were close to 8" x 11" and had 4 - 60 pin connectors that plugged into the back panel full of wire wrap pins.
So I was helping some new hires debug a problem and they had found a bent pin on the back panel. So I told them if the ever had a problem putting in a card, they may have just bent a pin.
So one of them grabbed the card puller and pulled out a different card and shined his flashlight in the slot. And he says "yes, your right".
From self builds I'm always suspicious of PSUs. Have a few times had odd stability issues that were either caused by an overloaded supply or one which had just gone bad. (On the other hand the current one is a cheap model that was bought because I needed a quick replacement and has lasted years, so maybe they're getting better.)
My rule for PSUs is never buy the cheapest one.
A PSU is literally the most important component of the computer. Its job is to power all the other elements and do so consistently and with the correct amount of power.
So I calculate the TDP of all the components of the computer, and I buy a PSU that can provide at least 40% more than that. Because PSUs "rated for" 400W are actually capable of 350W. If you push them to 380W, you're asking for trouble.
The reason this thing is 650W! Needed a new one quick, think it was from Maplin (I know, but used to be handy if you needed something that day) and there wasn't a great range in terms of quality, so I made sure to go up. Brand is CiT, which I see people have issues with, but it's probably pushing less than half its rated power.
My rule for PSUs is never buy the cheapest one.
Back around the turn of the century we decided that the best route for reliable new PCs was to buy them from the university computing department.
Until the day when one unit failed with a shower of sparks blasting out the back of the obviously cheap-ass power supply.
I'm talking Roman candle type sparks, the like of which I've never seen again
"Have a few times had odd stability issues that were either caused by an overloaded supply or one which had just gone bad."
The young'uns of today get to learn about weird symptoms caused by underpowered or flaky PSUs thanks to the Raspberry Pi. Education is the aim of the Pi of course, so that's alright :-)
Yes...it's amazing what sort crap "power supplies" people will try to use with Pis. Hence nearly the first response to most Pi problems on the RPF forums is, "What are you using for a PSU? Get the official one." When the cable isn't a fixed part of the PSU, that is also a Usual Suspect.
The place where I work has frequent emergency power supply and generator tests. These tests must involve some heavy breakers somewhere being pulled, because when they go BACK on, there's a surge which seems to run back up the neutral - this is NOT disconnected in your standard UK 3 pin outlet. This surge then proceeds to blister some component in switch-mode power supplies. I'm guessing it's a reference current source, because after a few of these surges the PSUs usually give up the ghost and shut down during boot-ups. You can eek a few more months out of them by setting the BIOS to stagger the device spin ups and other assorted tricks that smooth the heavy currents seen during boot up.
I used to support these in their Convergent Technologies guise, who were the original equipment manufacturer for the badge engineered Burroughs/Unisys version.
The 'power brick' was connected by a rather stiff flat cable with, from memory, RJ-10 connectors on both ends - one of the wider RJ variants anyway. Power bricks were a known cause of trouble as far as we were concerned - "Try a different power brick" was our equivalent of "Have you turned it off and on again?". The customer should have got Convergent hardware and support from us instead of the same thing with a Burroughs/Unisys badge on the front. :-)
As to "server" - not really. These were desktop machines. Convergent did make 'proper' server versions of these, but this is obviously a case of "The bank that likes to say 'Computer says no'" cheaping out and using desktop hardware in lieu of the, significantly more expensive, server variant. That or the appropriate hardware wasn't available from Unisys but only direct from Convergent - the Burroughs/Unisys versions of these used to lag several years behind the Convergent machines.
The Convergent NGEN, as it was properly known, was actually quite a cool machine - way ahead of PCs of the time for ease of configurability. Add-ons came as 'slices', boxes in the same form factor as the base CPU unit. To add a hard disk, a fancy graphics card, a tape drive, or somesuch to the machine you just turned it off, took a clip-on blanking plate off the side of the machine, pushed the self contained 'slice' onto the side, flipped a lever that locked them together, put the blanking palte back on and powered it back up. Total time 30 seconds plus boot time. The worst case was if you'd exceeded the power budget, in which case you needed to add an additional power brick to bring it back within spec. Also come with builtin local area network as standard (a multidrop serial affair running at a few Mb/s) at a time when an Ethernet card or any networking in a PC was a rarity.
Add-ons came as 'slices', boxes in the same form factor as the base CPU unit.
A concept resurrected by Acorn some years later in the Risc PC series (link is for a second version but has better pictures), where additional slices could be added almost ad infinitum, though I think they officially topped out at four - ISTR the extended backplane was only available up to eight slots (two per "slice") and case bolts longer than four slices weren't available either - though if used in "upright" mode, the slices actually fit together quite well without bolts.
Each slice had space for two expansion cards, one 5¼" bay and one 3½" bay and room for another power supply, for when you ran out of power for all those hard drives.
The ultimately expanded machine was a 10-slice monster created by Acorn themselves, containing a plethora of hard drives, floppy drives, removable drives, docking stations, TV tuner cards and even a pizza oven (apparently not actually functional) and a small sink (apparently functional).
The image seems to have disappeared from the above link, the Wayback machine doesn't have it and all I've turned up in a quick search is low resolution image in a Wayback capture of another now-dead page.
M.
That evokes painful memories of SCSI adaptors whose pins seemed to be made out of a metal with a stiffness slightly less than that of runny custard. And of SCSI chains which didn't work when correctly terminated at both ends, but which were perfectly happy once you removed a terminator and basically configured it so it couldn't possibly work.
Yep the only time you wanted to set the jumpers on the drive to enable termination was if it was an internal drive (and last in the chain, obviously) If you did that in an external drive, and didn't clearly mark that fact on the exterior of the case, you are asking for problems in the future when someone (or you, if you've forgotten) wants to daisy chain another SCSI device off it.
The B series used 40v power bricks with 6ft cords that fed dc to dc converters internally. They used a pass through x-bus (roughly 50x2 0,1 pins) that went out both sides and you used a lever to lock them together. These things were horrible, they looked a lot like toasters, you shoved expansion blocks (more toasters) into a long chain on the desktop. I had to run parts to the guy who serviced them, I did the PC's.
It was a server in that it supported client devices, but these machine were not very big. See https://www.blackmoreit.com/b38-mev-unisys-b38-mev-386287-cpu-module.html.
Disks and so on were latched to the CPU. These machines were configured in clusters with a server supporting several diskless clients, botting from the master.
Timely.
I have just bought and installed a little gadget that allows me to open and close my garage doors from my iPhone, as well as to get alerts when they open or close, and show their current status. Quite useful if you go away for a week or so, and as I'm retired, I do that a lot more than I used to.
A few weeks ago, the iPhone app indicated the gadget was "offline". I follwed the instructions to reset it, and...nothing happened. Pulling the power lead did, however, solve the issue. One off, I thought...until it happened again a couple of days later. Another reset, and we're good. For a week, then it happened again.
Enough is enough, I thought and gave the wall-wart power adapter, which I had plugged into the same garage ceiling outlet as one of the openers. Heat, rising from the parked cars underneath? Transients from the door opener motor plugged into the same outlet? So I ran an extension cord to a different outlet, moved the power cube towards the back of the garage, and it has now been several weeks without a failure.
The only problem is that it's a 5V output, so not much room for the wall of the garage, then run 18 feet to the gadget, which needs to be where it has a clear view of both doors and my wifi hub, so...on the ceiling, between the two cars. It doesn't seem to mind the heat at all (but it's on a little platform, hanging down a couple of feet from the actual ceiling.
Lesson learned. No matter how good your design is, you can be tripped up by a "commodity" wall-wart from China.
for the Unisys B-Series, you had a bus where you could plug as many peripherals as needed; you'd just have to be careful to provide enough power, plugin as many power bricks as needed.
So you had nice cheat sheets to help you with the task, unless you had enough money to add a brick to each peripheral.
Had two accounts with Lloyds before the merger and when they split I'm automatically with TSB, as with a lot of people I guess.
And immediately Lloyds ran a series of adverts saying they'd "been by your side for 250 years".
Not cool, Lloyds. You left me and my overdraft high and dry.
Back in the days before their Lloyds merger, I worked in the TSB offices in the City of London (St. Marys Axe? St. Mary At Hill? - some Mary anyway). In those far off days they handled Actual Money, so security was quite tight.
That was the place I found out that brick dust from drilling holes in a wall could set off their newly installed smoke alarms, causing much amusement to the security staff.
In a former life I received a call from a shop floor: funky harddrive. It made some spinning and scratching noise but otherwise didn't work. I changed it for a brand new one. And a few days later the same symptom appeared again. I changed it again and gave them stern warning not to physically abuse computers. Given this was a shop floor and knowingsuspecting that they'd rather take a hammer to solve problems, was not a totally unreasonable assumption. Anyway, the harddrive crashed again shortly after. I admit, I did not believe their affirmation that they didn't hit or even touch the machine but I agreed to take it to the lab to have a closer look. Turns out the hdd controller developed a habit of crashing drives. And I not to jump to conclusions. Nah, just kidding - I still do.
Back in the 1990s I had a fleet of Mac Plus machines with external SCSI hard drives. Some of the drives would refuse to start up, and on removing the lids they all hard the same make and model of 40MB (yes, really) hard drives. The ones that worked didn't have that make.
Turned out it was "stiction" - the drive lubricant got viscous when cold and prevented the drive from spinning up. Research showed that whacking it with a screwdriver handle fixed the issue, but after a few attempts the drive would die completely. So we introduced a standard approach: whenever a user reported the problem we'd take a spare hard drive, connect it in series with the failed one, boot the machine from a floppy, whack the failed drive into operation, clone the faulty drive to the spare, and then replace the former with the latter. Must have done 30 of those in just a few months.
I used to use that B2x B3x kit when I worked for Unisys.
It was actually badge engineered Convergent Technology kit sold by Burroughs/Unisys and a few other big names IIRC.
It had an unusual design in that all the peripherals came in uniform grey box design about the size of a thick book; you stacked them beside each other then operated a plastic lever that cranked them together and connected the buses. It took a few seconds to do; it was really easy.
The thing is, it wasn't really something that needed to be easy; saving a few minutes of hardware install time with that mechanism wasn't really cost effective. What it did do, however, was encourage hardware sharing by penny pinching managers. Things like tape drives would wander round the office daily. What ought to have been a one time operation on initial install ended up happening daily.
Aside from that it was good, if expensive, kit. And the OS was way ahead of its time. But when the PC came along, it couldn't complete.
Now that bus clip together design already wasn't quite robust enough; it would flex along with whatever it was sitting on. Repeated operation of the connection mechanism swapping kit around just made things worse. The things ended up becoming really sensitive to the evenness of the surface and how much it flexed. It got to the point where someone sitting on the edge of your desk could cause errors.
I was sure that it would turn out to be that damn bus connector.
Ah, yes, CTOS, Convergent Technology Operating System. When sold by Burroughs, BTOS, Burroughs someThing Operating System.
What was the T supposed to stand for? No idea. The supposition always was that Burroughs wasn't confident enough in their customisation skills to change the length of the strings.
I think this kit was also badged as DRS300 by ICL. I was working on a customer site when a 'small systems' engineer arrived to swap out the faulty DRS system which controlled a significant part of the manufacturing plant (I was there performing a mainframe upgrade)
ICL had recently been taken over by Fujitsu so we ended up with a large number of level 1 engineers from their retail organisation. I had to physically stop the guy from doing a 'whole unit swap out' which would have left them with a new server with nothing but the OS on as that's what he did in a supermarket.
If he'd been successful I'm not sure the customer would ever have recovered the data and they would definitely have been unable to operate for several days.
I remember going into the Burroughs office in Montreal with my dad, and being shown the BTOS machine he had.
I thought it was pretty cool.
I also learned how to fix PCs on a Unisys 286 he brought home when I was much younger.
I eventually followed in his footsteps and became a break-fix tech at Unisys in Montreal back in '99.
Fun times.
Otherwise someone will fish it out of the bin and use it on another box that starts to mysteriously fail.
Anyone remember the horror that was active SCSI termination? Where the drive at the end of the chain had to be powered on (so it would have termination) before the server or it would all go to shit?
I remember trying to explain that "this drive needs to be turned on first" and being told I was complete rubbish.
I have twice had that issue with PCs. In one case, the motherboard was on a slide out tray. Only when removing the tray fully, something not normally required even when swapping a motherboard, did it become obvious there was a slightly rounded dent in it such that it almost, but quite, touched the motherboard underside. Once it all warmed up however...
The second, we never did find exactly what the issue was, but moving all the gubbins into a new case fixed the intermittent crashing issues. Personally, I suspected the front panel switches, but couldn't narrow it down in the time I had. The problem only ever occurred one or twice per week at the customers site and there was no option to swap out the whole machine for $reasons unknown to me.
Earthing is a big issue in a lot of remaining situations.
One of my clients had sporadic weird problems with their pc's and other hardware. I think it must have been a telephone engineer who came in - first thing they do is to plug in an earth-checker - and found there were problems that needed fixing before he could do any work. That caused the light-bulb to go on in my mind as to the root cause of those other problems.
===
London Underground's Signal Department had a belt and braces approach to equipment out on the track. Track Circuit Relays* were AC (33.3Hz or 125Hz). Not only did the coil need to be powered from the absence of a train on the track to enable a green signal to be displayed, the phase and frequency of the supply had to be the same as a separate supply coming directly from the same source, and fed into the same relay. This was to guard against spurious power finding its way through alternative sources, and was the reason why frequencies not harmonically related to UK's 50Hz supply were used.
*Double Element Vane Relays.
I'm always impressed by the ingenuity, genius and attention to detail of LU engineering.
Well, the old engineering anyway, but that might just be a survivor bias.
It certainly sets off something in me when I think of how these old systems were designed taking into account every conceivable failure mode possible, no matter how improbable, and that we seem to have learned nothing at all when today we still have monstrous fuckups of Boeing-sized proportions.
to work experience with Unisys in the early 1990s.
As a 15 year old who had only ever used BBC computers and dot matrix printers the world of Unisys servers was quite novel. I remember them being strange modular beasts with the separate power supplies as per the article. Most of the time with the Unisys field team seemed to consist of swapping them or one or more of the circuit boards in the machines or trying to sort troublesome cables.
It also introduced me to how cheque clearing/processing worked (and why it look 3 days) seeing the processing machines at one of the main sorting centres for a different well know UK bank to one in the story (who have since changed their name), a mix of human operators who added the data at the bottom of the cheque for the value and the account it was to be paid into and the large scary machines that then read that data and sorted the cheques to be sent off the the relevant banks, a mix of OCR and magnetic reading (the details at the bottom were printed in magnetic ink in case someone wrote below the line!). If they went wrong the mess was amazing!
I also recall being shown a very big laser printer that could eat reams of fanfold paper per minute, you didn't want to be near that when it was running.
"I also recall being shown a very big laser printer that could eat reams of fanfold paper per minute"
The machines that printed the cheques were like at as well. And the MICR line was placed WRT the bottom LH corner, just for fun when the positioning of the print engine worked from the RH edge, requiring much wielding of a cheque gauge and micro-positioning.
Twenty odd years ago I was quite happy to help friends or family with building or upgrading a PC. This led to a certain amount of involvement with the 'local' PC building crowd. And I always remember certain 'members' of that crowd who would boast about spending on the newest most powerful CPU, or hard drive, or motherboard ..... who would then boast about how cheap their PSU was. They would then boast about how high they were 'clocking' their machines before they had to wind them back cos they'd become unstable. The strange thing was that on occasion I was 'clocking' lower spec machines to higher settings with no problems. And the only real difference was that I always bought reputable but expensive PSUs for those machines.
I saw lots of cheap PSUs in the 250W-400W range that would only supply that amount of power for about 5 seconds after being chilled to near zero in the freezer. As soon as the damn things warmed up you were lucky to get half the rated wattage out of them.
I had a Sun E250 around the year 1999 or so that every two weeks, give or take a few days, would develop a fault on one of the CPUs and crash. Sun were dutifully called out (except it wasn't Sun that came, it was some outsourcing company) and replaced the CPU. The first time I remember the guy opened the box up and looked on the motherboard for the label for which CPU had the fault and couldn't see anything. I had to point out the mobo layout was on the underside of the access panel he had removed
After 3 or 4 CPU replacements someone at Sun started wanting to figure out what as killing CPUs. PSUs were replaced, then the motherboard and still the CPU killer was there. Seems that since the E250 had dual PSU, there was a power distribution board (probably with PMBUS functionality to monitor the PSUs also) hidden under the mobo. Swapping that finally sorted the CPU killer
Never figured out why it was always killing one of the dual CPUs in the chassis
I once had a Sun E450 with Sun's active monitoring (via a very pricey leased line between them and my client). Idea was that if it died, Sun would notice and would: (a) call my client; (b) dispatch the required spares that the diagnostics had indicated were required; and (c) dispatch the engineer, who came from somewhere miles from where the spares came from.
We knew the engineer quite well and gave him a standing order: don't kill yourself getting here. (You could legally drive from his house to my client's middle-of-nowhere office within SLA, but only just and only if it was 3am with nothing on the roads). And we kept getting monitoring fees waived because most of the time the monitoring centre would only notice when the box restarted, not when it had crashed. So the usual sequence of events was: (a) system would die and refuse to reboot because the self-check failed on the broken component; (b) someone in the contact centre would phone to say it was broken; (c) my colleague and I would scoot into the office, pop out the board the POST was complaining about, and bounce the server; (4) the monitoring centre would call to say: "Your system is down" :-)
Had a dough rolling machine that had numerous interlocks to prevent fingers from getting rolled. It had a solid state relay to sequence the turn on and off and monitor the interlocks.
It started being intermittent, not wanting to turn on, and sometimes not wanting to turn off. Started replacing the relay, which would work for a while, but then it would stop working again. This went on for several weeks.
I finally wired around the interlocks to see if it would work properly. All good. Plug everything back in, sometimes worked sometimes didn't.
Checked the resistance of the power leads in desperation. Power less than one ohm, ground, around one hundred. Turns out the motor would run fine between hot and neutral, while the relay was a little more finicky. The resistance in the ground wire was right on the edge of where it would work.
Why 100 ohms on the ground wire? The insulation on it had corroded the wire so that inside was some kind copper salt. Whether it worked or not depended on the humidity.
For some decades, the various pieces of electronics have had local (point of load) regulators.
A typical power brick probably outputs 12V (at some probably horrendously large current). An ATX unit supplies 12V, 5V and 3.3V. The days of having microprocessors that have a core voltage of 3.3V are long gone in the high performance world (even almost 20 years ago there was talk of 0.8V core power).
I have seen processors where the core power feed could also be dynamically adjusted for the speed it could run at; this is quite common now for some processor families - in idle modes, the processor is throttled back and the core power lowered.
Many modern microcontrollers are still nominally powered from 3.3V (although many specify they will run on anything from 1.8V to 3.6V).
it is not unusual in modern kit (and by that I mean anything from the last 20 to 25 years) to have 5 or 6 different power rails (which makes layout interesting).
These local regulators can be quite sensitive to the input (noisy and has long leads from the source are two pretty standard problems) and the monitoring circuits can be quite twitchy (even though we put glitch filters on them, they still have to shut down fast enough to prevent damage if possible). They can also be very sensitive to load transients depending on the architecture used.
From about 20 years ago, these local regulators were switch mode devices and those who have not kept their analogue skills intact (and in particular loop compensation with analogue filters) will have trouble figuring out just how to actually design one (even using off the shelf controllers); without going into details, suffice to say that simulation tools are very useful (the underlying mathematics is not particularly difficult although it can be a bit tedious) but simulations don't take certain things into account, such as PCB layout and that can be the cause of many woes that may not show up until some components have aged.
So it really doesn't take much to upset those regulators which is why designing them requires a thorough understanding of the issues so they can be mitigated - unfortunately, they often are ignored.
I like the intro to this application note
Mine is more a case of malicious compliance.
For some unknown reason, the company I worked for hired a 3rd party company to provide hardware support. A year into the contract, it was then decided all fixes had to be raised with the US office first as opposed to the UK office as it had before. Now I got on well with the UK guys, and they were happy to let me fix what I needed and just replace the parts. After the decision to go via the US. they first had to get the nod from the US Manager (who I did not get along with, but that's a story for another time).
Anyway, all calls had to go to the US manager initially. It was a right PITA, especially because of the time difference!
Late one afternoon, one of my main fileservers blew the PSU. I replace it with a spare, then waited until the next morning to wake up the US guy at whatever ungodly hour it was to get him to OK a replacement. The UK guys finally show up later in the day to find the server working, and we all sat and laughed (they didn't like the guy either).
I wasn't always a saint.....
Thinking of "malicious compliance", a former co-worker noted that the best way of getting rid of any particularly onerous, or poorly phrased, requirement is draconian enforcement.
Fictitious example: Company policy says that hard hats are to be worn at all times. (It's missing a statement about "when needed" or "when in marked areas".) Want to get it fixed? Write up EVERYONE on site for not wearing a hard hat when at their desk. When a bigwig from corporate shows up to find out what's going on, meet him in the parking lot (while wearing a hard hat) and call him out on not wearing his hard hat on site as he's stepping out of the car. Policy gets changed in a couple days...
You'd think coffee makers alone would have gotten this sorted out.
With all the complaints to HR about being expected by management to clean a machine that I've never been trained to clean.
Complaints to HR about snide comments from co-workers who expect just me to know how to clean a machine that I've never been trained to clean.
And ending with a request to HR to be given a signed notice that: "as a condition of my employment,I should regularly be consuming a known stimulant and diuretic on company time (while obviously still abiding by the company's substance abuse policy). This requirement being obvious from the fact that management and co-workers both have expected me to know who to clean the machine for dispensing said stimulant despite never having been trained how to do so by the company."
OK, paranoia over :@)
One of 'our' engineers had a PSU problem to deal with that took weeks to sort out. This was on an aged, but hitherto extremely reliable printing press that was expected to run continuously for several days (until the plates needed changing for the next job) so downtime was seriously expensive. The problem was ascertaining it was the power source, as these machines are festooned with safety devices that instantly stop the entire press, and no-way can you just bypass them 'just to test'. It fairly soon became apparent that the failure was heat sensitive, as it would happen more frequently in hot weather. Also with that kind of crash it takes several hours to clear everything, clean up, reset plates etc.
What clued {unnamed engineer} up was when he was standing near the power unit he heard a click fractionally before the machine went down, and when he opened the cabinet there was a second one... from the thermal trip sitting on a not inconsiderable three phase mains transformer - that was apparently hot enough to fry eggs.
Epilogue
The rewind company found S/C turns on one of the secondaries - a rewind in 24 hours was not cheap.
this one before.
We had a machine that would randomly shut down... not good if theres several hundred pounds of metal in motion, would display any number of error messages..largly due to the ongoing motion with no power to the system.
So..... the engineers came in, figured must be a software fault...... replace HDD with known good one....
worked for a couple of days then fell over again...
Engineers back... replaced said HDD with old one... machine fell over.
Ok fault in the power supply module(this powers the PC module, and all the other stuff)
Machine works for another week...... falls over
Must be the PC module.... replaces that
Machine falls over.
Puts all the original parts back in....
Machine works for a month , then falls over.
Then someone twigs..... theres a timer relay in the machine so that when an emergency stop condition is met , the machine can instantly cease all motion before shutting the PSU module down.
20 quid for a new one and everything is solved.
And now every service manual for that type of control says
"In the event of an unexplained power shutdown while machine is in motion, replace the timer relay first"
When I was a grad student, 40 years back, I remember the folks who built the research machines told me "It's always the power supply. Even after you've ruled it out, it's *still* the power supply."
That being said, the nightmare tech I remember is coax, specifically tapping receivers into a coax cable, before there was point to point ethernet cabling (this was 10 MBit ethernet, so we're going back to around 1980). You'd add a new machine to the network by essentially drilling a hole into the cable, and adding a tap that reached the copper core of the cable. Of course, each time you added a new machine, you'd get different reflections from all the previous taps, and of course you also had to terminate the cable itself properly. The end result is that you'd add a new machine, and some random arbitrary pair of other machines might start having trouble communicating.
Of course, each time you added a new machine, you'd get different reflections from all the previous taps, and of course you also had to terminate the cable itself properly. The end result is that you'd add a new machine, and some random arbitrary pair of other machines might start having trouble communicating.
Hence the markings for vampire taps at 2.5m intervals so as to NOT have reflections in phase.
Had a PC in one of our labs once that was new, and a powerful (for the time) machine. When placed under load, it would consistently bluescreen . As it was under warranty, the manufacturer came in and replaced the RAM (can't remember the blue screen error, but it was generally faulty RAM). It still bluescreened, so he replaced the CPUs. Still bluescreened. We went on for a few weeks, replacing things one by one until he'd replaced everything apart from the case and SATA cables, and it still blue screened. He replaced the SATA cable linking the motherboard to the primary hard drive, and the blue screens stops. The Engineer did try and blame our Windows image. We considered that might be at fault, but the same image was running on 9 other machines with identical specifications (same CPUs, same Motherboard, same RAM, same Graphics card and Hard drives), and none of them hand any problems. Something my boss pointed out when the manufacturer tried to blame our Windows image.
There never was any indication from Windows that this blue screen was anything other than a RAM or graphics card error.
Reminds me of what happened just the day at our place. We have a Planar 65" monitor display. One of the 50KG monsters that can't decide if it's an AIO, a monitor, or a computer. After one phase of our 3-phase power feed crapped out, it refused to connect to the network. We rebooted win10 on it several times and it came up as expected. The Planar could see most of our network except for the one VPN it was to use. Every other device saw it and used it fine. So, as a last resort, I shut down the whole shebang, unplugged the power cord, counted to ten, and plugged it back in. All was right as rain.
Imagine all this story but swap it out to an OEM builder doing new systems in early 2000 with a test workstation. The fault a keyboard that had been used on hundreds of systems causing a blue screen no matter how many other components replaced. Moral of the story use Sherlock Holmes logic...
verify cables
swap cables
check fuses
swap fuses
verify power supplies
swap power supplies
check for corrosion on power leads
check for corrosion on cable connectors
follow the janitor across the data center and stop them from using the rotary floor polisher in front of the disk cabinets (kick data center manager for allowing it)
Insist that a recording power monitor be placed on the lines feeding the data center. Not at the service entrance, but in the data center.
((if necessary, engage with equipment supplier for further assistance))
Recently I was given a laptop (it had been replaced by a newer shinier one). Every time I started it up it turned off after a few seconds. Turned out to be the lead from the mains to the brick. The really weird part was that was the lead that had been happily powering the machine for a few years. Must have been damaged in the post <G>
Back in the day I was the computer guy at a shop that did computers and musical instruments (not necesarily at the same time) - the boss was a bit of an 'entrepreneur'. Had this PC in to fix, had it on the bench - powered it up no problem. Put the lid on - Reboot. Odd I thought, lid off - fine - what's it shorting on.
Looked around it for about 20 minutes and found nothing, had a sip of (now cold - not unusual) coffee while I was thinking, put the mug down a bit hard next to the PC - Reboot - What? used the back of a screwdriver to do a bit of tapping and got to the PSU, tap - Reboot, tap - Reboot. Faulty PSU diagnosed, replaced and PC sent out with relevant bill. I didn't bother repairing the PSU as they were always a sod to get apart and never went back together right.
I'm sure I've mentioned these before but here goes anyway...
Mysterious keyboard faults that were due to staples, paper clips and/or bits of paper (those so-called chads from hole punches) being chucked around desks. Fixed by banging the things on the desks and issuing stern warnings to the people responsible.
My own Amiga developing an intermittent fault with the mouse, which turned out to be a gap in the casing, which admitted light from the great outdoors on sunny days. Fixed with a bit of paper and tape. I suppose I could have closed the blind but I liked the daylight, and since I wasn't well enough to go out and about at the time, it was the best I could do.
I set up my rig in preparation for a gig. Switched on, left the thing on standby, grabbed a pint, came back to the soundcheck. Soon as I hit the On switch, the thing gave out the most horrible hum ever.
Opinions ranged from dud cables, ground loops, faulty switches, faulty power supply on me effects board, to busted pedals. Tried all kinds of things except the obvious.
I felt a proper idiot when I worked out the cause. I have some of my pedals in front of the amp and some in an effects loop, which for those who don't know, is an output from the preamp (Send) to which you can then connect various effects and plug the output of those into the power amp (Return) I'd left the effects loop cables draped over the amp, right across where one of the transformers lived. Moved them away to where they live on the floor and silence (or as near as you get with a big valve amp) reigned.
Yes, I got a lot of stick for it. No, I've never done it again.
An odd one I had recently was a Dell laptop. Randomly (but more often that not) it wouldn't recognise the ethernet port in the Dell USB-C docking station. Recognised the dock itself, and monitor / keyboard / mouse worked fine. Did it with multiple docks. Tried reinstalling the driver for the docking station's LAN adapter. Tried a clean install of Windows. Nothing worked. Diagnostics reported no issues. Occasionally it would work for a while, then die again.
In the end the laptop's motherboard was replaced, which seems to have solved it.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
