Sysadmin cured a medical mystery by shifting a single cable Few make it to Friday without some end-of-week blues, which The Register always treats with a fresh dose of On Call – the reader-contributed column that recounts your stories of tech support contusions. This week, meet a reader we’ll Regomize as “Jake” who told us of the time he worked as the sole sysadmin for an entire …
>Jake didn’t tell anyone about his fix.<
I wonder if the problems with the CT scanner started when someone got rid of a tripping hazard and decided to replace the direct cable connection across the room by properly connecting the CT scanner and the workstation to the existing wall connectors without telling anyone.
"There are many things that we would throw away if we were not afraid that others might pick them up." - Oscar Wilde ...
A very old quote, so I'm just saying that we're not seeing anything new these days. I'm not complaining, I worked for unrelated but similar upsets years ago in the medical environments.
If they have those wall port connections documented somewhere, eventually someone might notice his undocumented guerrilla fix and "correct" it. You can't make changes like that in a highly regulated environment like a hospital without telling anyone. For all he knows there could be a very good reason why the two aren't directly connected, and he caused some type of potentially serious problem that he didn't hear about. Trip hazard, cable tension if the workstation desk was moved that caused it to fall off and break taking the CT scanner out of action for days, who knows!
Sorry Jake, you didn't deserve kudos for that you deserved some education from your boss about the right way to do things!
This post has been deleted by its author
Network switches running on config in memory that had never been saved or even backed up, the first power outage and the whole network was broken with ports inundated by broadcast storm traffic, DMZ switches were directly connected to production network and so was the open internet.
A network at a cult (yes, a real live Christian fundamentalist cult which, I have been told, still practices gay conversion therapy despite it being illegal) which had almost as many switches as PCs in a sprawling Victorian building in Cumbria where all the cabling ran through the suspiciously well secured cellars (big, strong and obviously regularly used metal doors with conspicuously heavy duty padlocks) to which I was refused entry.
The dairy where they'd lost connection to the warehouse from the server/Comms room because they'd demolished a building in the yard, said building had contained some of their fibre patch panels.and yes, the fibre from the warehouse passed through it.
The hospital which still had live 16MbpS token ring bridged to ethernet and back again because management had authorised the expense and refused to accept "the lost investment" nearly 20 years later...
And there are so many more, like the architects who took wire cutters to the WiFi antennas on their broadband router because they were scared of being hacked and losing the contents of their two PCs to "competition"
I think the most ridiculous one I had was back in the days of cheapernet (10base2). I'd inherited the setup which mostly used the make-before-break sockets. For any youngsters (!) the max segment length was 185m and you could only have three segments with attached devices bridged together.
Our site had two main buildings, one occupied by the main company and the other by a secondary operation, and inevitably the computer room housing the AS/400 was at one end. We had one segment running from there to about 2/3 of the way down the building, the second ran from there, across the yard and just in to the second building then the third covered that building. Nearer to the AS/400 was mostly served by 5250 terminals on token ring but further away, including the second building, used PCs with emulation software via the network; the other building also had a file server for the secondary company's use.
Without bothering to consult or tell IT the secondary company decided they wanted more network ports so they got the server maintenance people to simply add a second network card in the server and hung them off that. Not long after they kept complaining about connectivity and how it was all our fault! It didn't take all that long to trace the problem, not only had they added a second network card but they'd configured it to bridge between the two networks, giving 4 segments and a very unreliable network. We insisted the bridging was turned off immediately and that any PCs using the AS/400 were attached to "our" network and not "theirs".
That sounds suspiciously like a company I worked for (American company, this site was in Swindon).
When I was there (placement as a student) we had the link between buildings get routed half way around Swindon rather than direct across the road so that branch of the network was barely functioning, had issues with devices randomly not connecting to the network (too many devices on that controller - fixed by shifting some to the second one), and people moving terminals without telling us - one such move put the terminal next to a high power cable...
Oh, and that's where I had the call to a terminal that wouldn't turn on... it was unplugged at the mains.
I've never encountered an architect who refused to let their total ignorance of how any technology actually works and lack of common sense prevent them from devising and executing their own solution to a problem which inevitably buggers things up in such a way that it takes the qualified person* they should have engaged in the first place at least ten times longer to undo their "fix" than it would have taken to resolve the problem before they took matters into their own hands.
Nor one who apologised for their efforts in the field of buggering things up.
*muggins here
That's reminded me of another one. ACT Apricot had their own networking system which was point-to-point two wire cable. A customer had a new office building in progress and I asked to be able to speak to the person doing the wiring so we could ensure it was correctly installed. I was basically told I was being presumptuous as "they know what they're doing" but reluctantly they agreed to a site meeting, which turned out to be with the site manager, not a sparky.
Of course the wiring was incorrectly installed as a ring (zero information being passed on to the man actually doing the job), fortunately leaving the two ends joined at an accessible socket so I could disconnect and insert the required termination resistors, which had also been omitted, at each end.
They'd also paid a fortune for turned-earth-pin sockets for the computers (so nothing else could be plugged in), to be connected to a separate phase to everything else, as the systems were vulnerable to power issues and in particular didn't like computers on the same network segment being on different phases. The architect had ignored the fact there's a minimum phase-phase physical separation mandated and had put the special sockets right next to the normal ones, so the sparkies had wired everything up as usual, all upstairs on one phase and downstairs on another. The spend got even worse when it was discovered they hadn't provided any plugs for the special sockets and a motorcycle courier had to be used to get some urgently! Had we been aware of what was happening, and been able to pass the info on, we'd have simply put two network segments in with an opto-isolator between to eliminate any ground loop issues.
I achieved 'Magician' status by reversing the wires on a two-wire temperature sensor which was buried deep within a complex piece of machinery. By eliminating an earth-loop we avoided a major shut-down involving site work, cranes, transport, dismantling, reassembly, more transport, another crane, more site work.......
Yeah, we had sparkles who "installed" 802.3 coax for us, they did a lovely job, all channeled in, capped off and hidden, looked like the office hadn't been touched but the network didn't work worth a damn.
Opening up one of the beautiful brushed nickel face plates on the network boxes showed us why, they'd stripped back about six inches of coax, neatly twisted up the braid, covered it in sleeving so there were two pigtails and then (quite nicely) soldered them to the backs of the BNC sockets before heat shrinking them.
What puzzled me at first was why there were two lengths of coax in the channel, eventually I got the the end of the network and found that, as well as the horrendous impedance bumps at every socket, they'd wired it as a ring with the extra coax connecting back to the socket at the other end of the segment.
The weekend overtime pay was nice and I got to play with a TDR for the very first time
Unfortunately due to how I was made redundant at one role (back to my desk only under supervision to remove my persona items) I didn't manage to acquire the TDR network tester. There's no way anybody else would even know how to use it so I wouldn't' have felt very guilty.
There's no "to be fair" about it, if you tell people you're experienced and capable of installing network cabling then charge a premium for the service over your normal hourly rate for electrical work then I expect them to be competent and not to have to redo all their "work".
I dread to think how many smaller networks they'd "done".
In a previous life we had installation technicians, who were just about capable of assembling a disaster zone but having to leave before it had to be tested, wire ethernet cables by just looking at what to them was a random assortment of cables and just connecting them however they felt like. Not to mention cutting the cables to length so it was almost impossible to undo their nightmare.
These were the same installation technicians who didn't care whether it was 240v AC or 24v DC down a wire either... it was highly important that they never turned anything on because it would almost always be a costly replacement job.
Never underestimate the "interesting" things sparkies will do - almost as "interesting" as the things wet-pants will do when there's an electrical element to a plumbing job.
With previous work hats on, I did some network installations - and some fixes of "problems". Often, the sparkies would pull all the cables and I'd terminate them, other times they'd do the containment (e.g. dado trunking) and I'd pull the cables as well. These jobs were occasionally nice, often a p.i.t.a. - examples :
One job, lots of cables pulled in - some by a decent sparky, some by a coyboy working for the firm that added the extension to the building. "Variable" lengths of cable left to terminate into a cabinet - longest went down the corridor to the other end of the building, the other came just 4" out of the wall ! IIRC it would have been around 5 to 10% of the cables had to be extended.
One job, the trunking was there, and I had to pull in the cables - along one wall it ran to 24 cables plus a multi-pair phone cable. The **** sparky had bagged the biggest section for himself, leaving only a section that was too small. He swore blind that additional dividers weren't available (which would have allowed me to create another small section), which I later found out wasn't true. But the priceless bit from this job "it has to be that way round as that's the way it's insulated". A supposedly qualified sparky genuinely believed that a piece of plastic is directional in it's ability to insulate !
Another job where I found one or two faulty cables due to nails through them in refitting the floor boards.
And on fix-it jobs. In a neighbouring office to ours, when the tenant moved out they agreed with the building management and incoming tenant to leave the structured cabling in place, they'd just remove their cabinet. Guess how their sparky removed the cabinet ... yup, by just cutting the bundle of cables tethering it to the wall.
On a science park, a tenant moved who'd had several offices linked into one. So they separated the data cabling and put a small cab in each office. The park management asked me to test the cabling - which didn't work. Clueless ****wit hadn't looked properly at the back of the patch panel, so everything was wired "2 pairs to one socket, 2 pairs to another - with the next cable making up the missing 2 pairs on each". For good measure, a lack of labelling, and some of the cables were swapped - e.g. instead of finding 5&6 paired, you'd find that 5 was somewhere else and it was fun to find (see previously about lack of labelling).
And as someone mentioned architects, the classic note on the bottom of the drawings "data cables are the responsibility of the installer" as a get out for not having made any provision, at all, for such cables to run anywhere.
Just don't get me started on users who would decide to move the office around, not ask if we (IT) were available to repatch them (they'd usually manage to find a busy time), and just unplug everything so we didn't know where anything had previously been plugged in - excess of outlets (I flood wired the place), limited ethernet hub (yes, that long ago) ports, and a digital phone system where the number went with the physical port.
Fancy brand new building on campus in the years Before Wireless but when most students arrived with a laptop. New building had Ethernet jacks installed *everywhere* so students with a computer and a cable could connect to the network from anyplace they might find themselves sitting. Well, apart from the loos.
Low bid for the cabling was an electrician. Did *not* understand networking. Two wings with three floors had to be pulled and redone from scratch.
Two years later wireless arrived and rendered it all moot.
The senior, highly-successful grant-getting scientist and faculty member who called help desk fuming about how IT was so incompetent that now the network was failing. As new guy, I got the ticket. No one else wanted to deal with his legendary crankiness.
After listening to the obligatory cussing-out of IT I took a look. This was back in the days of 10-base-2 with BNC connectors. The ones that look like TV cable connectors, because they ARE the same connectors. And sure enough, when he moved his desk the existing cable was too short, so he brought one from home. Yep, for cable TV. I gently explained about differing electrical characteristics in cables and put in a longer Ethernet cable I had with me. Problem fixed. Red-faced researcher, though to be fair he was a chemist, not an EE.
The team hailed me as a minor hero that day for delivering quiet comeuppance to a major adversary. He was nicer to us after that, too, at least a little.
He should say "well this solved the issue" and let them decide whether that's an acceptable solution or not. There may be reasons it connected via wall ports he wasn't privy to since he didn't even bother ASKING anyone.
You're 100% wrong on this. A hospital is a highly regulated environment, and a CT scanner is a very important thing that they can't afford to have down. He relied on his cable tracing to infer it was a direct connection, but he didn't even ask. What if it wasn't? What if the workstation needed to connect to the network for backups? What if the CT scanner needed to connect so it could contact the manufacturer with information they need as part of its support contract?
You can't do something like this and not tell anyone. There's no conceivable situation where him saying "well I did this and it fixed the problem, is that an acceptable solution to you?" could set off a political fire. But it sure could if he did this and didn't tell anyone.
If you read the article, it says he traced the cabling and it was single point-point connection - just going via a tortuous route.
But I agree, having fixed the problem - or at least determined that removing the apparently redundant cabling did fix the problem - he should have then asked the question as to why it was done that way. If it was 250m round trip, then just adding a switch in the middle of the link wouldn't have put it in-spec, but it would almost certainly have also made it work.
Since he is the only IT person, there is no one that he can ask "why was it like that". And I'd guess that it was "because we do all our networking by cabling every device network port to a wall socket". And a wiring chart might possibly exist, but I know a manager who would hold that a well designed network is its own chart.
We were moved to a room which had previously been a laboratory and subsequently been repurposed as an open plan office. Our network cables had been routed up into the roof void (we were on the top floor) in order to reduce trip hazards and trailing wires. All our desks had a 4-gang plug accessible via a cover at the rear of the desk etc. You get the idea.
Except that the network was unreliable and would periodically either go completely on the fritz or else start transmitting garbage.
This was coupled with a peculiar noise issue and a low frequency throb which only a few people could hear but which several people complained caused headaches. The fan was on a thermostatic control and would vary speed automatically depending on the room temperature.
It turned out that the former labs had a huge fan and motor which were originally designed for fume cupboards and were massively oversized for the office duty. This motor was the size of a standard desk and used 440V power supply with speed adjusted by an inverter. The motor had not been changed but the fan speed had been set to a lower speed range.
Unfortunately, our network cables were routed conveniently close to the motor and using a cable tray immediately adjacent to the fan power supply, motor and inverter. In addition, the lower fan speed caused a low frequency hum, barely audible in the 20-30Hz range.
Eventually it was realised the the unshielded network cables were being affected by the electrically noisy motor and inverter. The problem was worse when the inverter was adjusting the fan speed, which it would do every few minutes.
After a few weeks the cables were re-routed under the hung ceiling at great disruption to all in the office and the motor and inverter were replaced by something smaller and more suited to the task.
Working in Telecoms on a Meridian Option 81 with several thousand lines this was really common and also may explain this situation.
When doing office moves, the box shifters, sorry desktop guys, and the planners forgot to inform telecoms that their help was needed. This meant it was not uncommon for them to use the patch panels to jump between cabinets and even floors.
I think my record to find inter-patching was 6 cabinets. So it may of been the case that the workstation was in a different location before and was a single hop. It then moved and rather than doing a rip and replace some bridge spark went I'll just inter-patch.
AHH but I can't get from A2 to C1 directly, so I'll go A1 to A25, A25 to B1, B1 to B25 and B25 to C1
It's often overlooked, I know that a vendor I once worked with didn't care about the length of a run of cat5, we had some well over the spec closer to 150m. When they were called on it, there was some teeth gnashing and extra data cabs installed at a very reduced cost.
I had some telecoms cable wranglers foist similar on me.
"It'll work anyway" No. It might work, but can't be guaranteed and means if we ever change the kit on the end we'll have to test everything in depth.
And talking of depth, they'd dragged the cat5e through a moat without even attempting to seal the end. Had to cut >5m off it before I found dry wires. At least that meant it was only 35m over spec, albeit in the wrong place.
I've done over-length runs myself - in once case out to 180m. I knew they were over-length, but as they only ran phones (which had a much longer length limit) and serial terminals, there was no problem. Of course, I knew about these - as well as a bunch of links between buildings done with phone cable but terminated on standard patch panels (again, needed lots of phones, would never need that many network connections, and there were handful of Cat5e cables as well) - but I think I never got round to documenting them before I was made redundant. It was back in the days when Cat 3 was still around, and Cat5e was a lot more expensive than 40 pair phone cable, 100M ethernet was considered a luxury, and management weren't generous with the budget.
Where's the "whistles innocently pretending to know nothing about something" icon when you need it.
Not IT, but it might amuse. About 30 years ago, when I was supporting an engineering company that ran small upgrades and repairs to North Sea oil & gas installations, one job that came along was to add some pipe supports to a line on a fairly new production platform - the line had already required repairs to cracks appearing at welds, and the client engineers assumed it was down to vibration induced fatigue. We sent a surveyor out for a day visit to sketch out what was needed (the plan being that he'd return to prepare a full quote for the work). However, when he got there and took a look, he couldn't believe what he found. The pipe run needing attention was between two adjacent production modules but, rather than going by the direct route, ran the long way around each module. He reported to the client that further supports would only be a botch-up and unlikely to solve the problem; since we couldn't offer any warranty for the work, we would have to decline any further involvement.
On investigation by the operator, it turned out that the contract for the modules had included a metre rate for pipework, and the client's engineers hadn't done their job of reviewing the design proposals very well.
I recall spending an afternoon with an ARCNet network resetting all the card numbers and creating a numbering scheme for each building of the now defunct research org.
This meant that all machines in building A had close numbers, all in building B etc. Resulting in the token being passed between closest machines first instead of zig zagging across the campus and traversing, sometimes, many hundreds of meters between machines.
Network speed picked up significantly.
Man I loved ARCNet. Back in the early NetWare days we installed a LOT of ARCNet. Done right it just worked. No it was not the fastest networking system but done right it just worked. Had a local electronics store that got into computers and decided to undercut our prices by getting into networks. They messed up multiple networks that we built. Boss started charging double for troubleshooting networks modified by their "experts". Best part was we real experts were paid a salary plus commissions. Good times.
When "Thin Ethernet" came along they managed to screw up a few of those too :)
Many years ago at a pharma that I have oft mentioned in Dartford I would patch lab test equipment to others for Lab IT via the patch panel, Dr Derek was very greatful for such co-operation & would reciprocate by doing favours for me in return, like bringing up Totalchrom PC's, monitors & associated kit as the labs in Building 225 were closed down, up the steep hill* to Building 313.
Moving away from Pharma - Last year we refurbished a test cell at my current location, a tech from out east came to do the install & his cable management was terrible**, so in addition to tidying that up, the KVM's on two PC's in the cabinets kept flaking out because of the distance & routing next other cables carrying a multitude of raw signals & power.
As we had people in for initial shakedown tests at great expense, I moved the computers closer to the operators consoles & for expediency in providing a workable fix I routed some cables via the closest patch panel. This got them working reliably in the short term but added a lengthy delay of real time data.
I was called to task for my cable management solution around the time I was asked to resolve the outside technicians cable management solution, but explained at least for my solution it was the fastest way to get the external experts operational before their flights back to Poland, as I had to pull up the floor panels & reroute everything & that would probably take the best part of a day (I think it took the best part of a afternoon) & the experts would be extending their stay over a weekend & into the next week.
* As I understand it, the hill was originally instrumental for gravity feeding water & in production "bulk" quantities of drugs without additional pumps to further stages of production or shipping.
** C-Suite got a picture of it & I was tasked with unravelling the gordian knot's of many many cables & power supplies entangled or left hanging within the cabinets by 110V cables, I ended up 3D printing rack mounted holders for the "bricks". Beers thoroughly earned & deserved, hence icon.
We were sub-subcontracted to figure out how to network between two buildings operated by a Fortune 100 corporation that everyone in this forum has probably done business with at some time or other. Turns out that they had already run fiber (ancient multimode, but we didn't need much bandwidth so whatever) so we found the necessary SFP modules for the switches, plugged them in, and.... no data. A more in-depth examination of the connection showed that zero photons were passing through, and the reason turned out to be that the fiber had been run through the air between the two buildings. This was not outdoor-rated or armored fiber. Just basic, plastic-clad stuff intended for indoor use. It was out in the open just dangling and soaking up the cumulative effects of the coastal desert sun, wind, rain, etc.
As a proper cable run between the two buildings would cost a six-figure sum and disrupt operations by tearing up their driveways and parking, we wound up using good, old-fashioned wireless bridging.
This was over 20 years ago so hopefully they have moved on since then, central Australia near Yulara, two site are 10km apart as the crow flies connected over two different technologies.
The Network Path transited from SiteA to Yulara Township to Alice Springs, to Adelaide then Sydney, bridge over somehow then make the journey back via the same route to siteB
When I was a green youngster hoping to get a job in tech support, I went for a job doing IT support for a local private school. As such, they were very well funded, and, IIRC, the job was well paid.
Having applied for the job, I was excited at the thought of potentially working there. I would have been managing a Novel Netware network, with a couple of servers and hundreds of workstations. This was the late 80s/ early 90s, and it was apparently unusual for a school to have as large a network.
The interview was at 1:30pm, so I turned up suited and booted, slightly nervous, but confident and excited. At 1:30, a secretary came up to me and said there would be two interviews, a personell and a technical one. I was quite happy with this, and said so. Then she dropped a bombshell. They had originally arranged the interviews in order of surname, so mine was originally first. They had decided that morning that this was unfair anyone whose surname begins a letter toward the end of the alphabet, so they flipped the list. I was disappointed but didn't see a lot of people in the waiting room, so I waited. And waited. And waited. Eventually, at 6:30, I had both interviews.
The personell one was the generic personell questions. The technical one, which they held immediately after because they wanted to go home, consisted of an aptitude test and also a couple of PCs, one set up as a server, and one as a terminal. I had to diagnose why they couldn't communicate. I correct diagnosed that the server wasn't plugged in properly.
I didn't get the job. Apparently, although I interviewed well and resolved the tech problem (both of which suprised me as I was extremely stressed due to the delay), I didn't quite score highly enough on the aptitude test.
I was angry at the time, but I think I lucked out. Imagine working for a company disorganised enough to flip their interview list a couple of hours before the first interview. I don't get stressed easily, but I suspect the stress would have killed me.
Ok right, where do we start?
"They had decided that morning that this was unfair anyone whose surname begins a letter toward the end of the alphabet, so they flipped the list.”
which logically means that this change was unfair to anyone whose surname was close to the start of the alphabet, no?
"Apparently, although I interviewed well and resolved the tech problem (both of which suprised me as I was extremely stressed due to the delay), I didn't quite score highly enough on the aptitude test.”
Aptitude test, what the fuck...?
No look, they already had a candidate that they wanted to employ, the whole process was a farce. it didn’t matter how well you performed, they were always, always gong to choose the ‘Proctor’s nephew’ or whoever, for the job!
Don't feel bad, you had no chance from the start!.
At Uni we all got the degree of the person to our left (or right, I really cannot remember). There was an epic mail merge failure that the administration didn't notice until quite close to the time, likely needing a new supply of pre-printed certificates, and therefore everybody was given an envelope/tube with an IOU in it and told that they'd get their certificate in a couple of weeks.
Back in the day, when telecoms ran on 2Mb BT branded lines (can't remember the name, arrgh.) Descended to the basement of an office in London of a well known British bank. Had to duck under the cable, stretched across the middle of the room from the multiplexer to a wall socket! This muxed the carefully configured redundant 2Mb links to a single 8Mb line. Did the work on the PABX and left 'em to it.
Needed to share test lab networking between two adjoining rooms. IT was notoriously fussy about their switches and special rules for what traffic they would permit. Inordinate amounts of approvals would be required for them to open the two switch ports to each other (create a dedicated VLAN essentially).
A fluff & buff remodel was done to align with new corporate standards (get rid of offices, deploy cube farm, etc). As the re-networking was happening, and working with vendor after-hours, I managed to plug in a patch cable between the patch ports in the two rooms - eliminating the corporate switch gear. When a few hundred other cables were added with really good cable management, unless someone took all the bundle apart they could not tell the cable doubled back to a different patch port on the same panel. Once the paint was dry I could uplink between switches in each room.
Yeah, a technical violation of the rules. It was a clean solution. Alternative was pop ceiling tiles to run a cable over, or take apart back-to-back wall plates for some rogue in-wall cables. Or simply drill through the sheetrock and conceal behind a desk or something. Bust down the wall and make the adjoining rooms in to a single room....
So for the hero of El Reg's story....There may have been a reason the cabling was done that way, which was long forgotten by they time they arrived on scene.
Andy never told anyone about the fix. And the government never figured out how their system for clandestinely making "backups" of all patient test results stopped getting updates from this one hospital CT machine.
(Kidding. I hope!)
BBC news, June 2025: https://www.bbc.co.uk/news/articles/c1ljg7v0vmpo
"Every newborn baby in England will have their DNA mapped to assess their risk of hundreds of diseases, under NHS plans for the next 10 years.
"The scheme, first reported by the Daily Telegraph,, external is part of a government drive towards predicting and preventing illness, which will also see £650m invested in DNA research for all patients by 2030."
I haven't read the Daily Telegraph article, so I don't know if the point was recognised that this means the government will have everybody's DNA who was born after a certain date, without the individual's consent. I think the Daily Telegraph will be all for that. And also for genetic selection for school or university admission, like "GATTACA". The Telegraph is "that" sort of "news". (I don't mean dystopian science fiction, just dystopian generally.)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
