Linux admin asked savvy scientist for IT help and the boffin blew it Welcome, gentle reader, to another exciting week at the coalface of tech and therefore the anticipation of five joyous productive days ahead and a fresh edition of Who, Me? – The Register's reader-contributed tales of tech gone awry. This week's main character we'll Regomize as "Igor" because he used to work in a lab back in …
I was sent out to install a 3Com 10 Mb/S network card into an IBM Microchannel Architecture PC. Following the instructions, I installed and configured the card. That part all went fine.
Then I noticed it came with a self-test program on floppy diskette. Well, sure, I should test this.
I ran the program, and waited for results. 30 seconds ... 60 seconds ... 90 seconds ... the desk phone rang (I was the only one in this office). I answered. It was the head of networking. "What the hell are you doing?!" I told him. He told me the test program was kicking out malformed packets that were slamming the network, and to not run that program again, to which I fervently agreed.
The thing that impressed me was that Networks detected the problem, figured out the switch, port, jack number, office number, and phone number so quickly.
our BOFH sending new PFYs to hunt for the token
PFY: "Sorry Sauron Simon, your previous monkey PFY appears to have fallen into the lava chamber clutching your precious token."
Sauron Simon: "BUGGER! I FORGOT THAT I CAST THAT MISERABLE WORM FROM THE WINDOW. UNFORTUNATELY UNAWARE THAT GRUB HAD CONCEALED MY TOKEN FROM ME."
As I think Pratchett's barbarian Cohen once remarked to a Troll we don't have properly Dark Lords any more. The upper case in tribute to Pratchett's scythe bearing character who is still as properly fatal in this world as on the Disc.
Well there was cheapernet (10BASE2) which involved the horrors of coaxial network cabling where because it was all in one line, with the required terminators on each end, interruption anywhere tended to take out the entire network. For larger networks there was the further horror, and vast expense, of aggregator boxes which combined multiple such lines. Naturally the maximum speed was 10mb, and that was on a good day when nobody else was using the network.
A bad connection on 10base2 could result in a reflection and signal loss in the middle of an otherwise working run which could drift backwards and forwards by one machine over a day! First time I saw that it did my head in - how could a single cable that worked at both ends not work for 20feet in the middle? TDR=£1squillion so finding the bad connection meant literally checking every T-piece and connector on the line beyond the fault. Part thick/part thin ethernet feeding three floors and running through crawl spaces and ducts inside and out so it was bound to be damp ingress somewhere ... or the joiner someone put under a secretary's desk because they had cut the coax too short ...
Many moons ago we had 10base2 layed across the whole team floor.
Since our boss insisted to go to lunch with the full team, and taking the "we go once the last member comes" approach would mean at least one hour wasted waiting for the last, he would, after 2 mins, take off the 10base2 plug from his desktop, insta-freezing the whole floor, at which point everyone got the message it was better to go lunch.
Good times.
"always easy to fit another box into the middle of the net. All you needed was the T-connector and a piece of cable long enough to reach to the next box."
Until you found out what the 2 or 5 meant in 10base2 or 10base5. (Clearly here 2. :)
Don't get me started on clowns that had inserted (mono)monitor coax (75 Ohm) into the circuit. Or some TV antenna coax. Or some cable from a random lab instrument.
A pocket full of 50 Ohm terminators and binary section to locate the problem but binary section or sectioning of the perpetrators might have been more effective or at least more satisfying. Always wondered how effective a vampire tap would be as an EpiPen® substitute.
I did use a lab. network that was made up of 75 Ohm Coax, TV connectors & all. It consisted of a lot of little boxes with a Z80 in them, each with a D25 for RS232, daisy chained. I think it must have had a box at the other end to break it out into more D25s because at that time all you got on the back of your Unix box was serial terminal connections. Yes, it worked.
I just got fed up with being charged idiotic amounts of money for a 50Ω terminator so I bought a bag of BNC connectors from RS and a bunch of resistors and cooked up a few of my own.
Getting 50Ω was tricky, but 49.9Ω was not :)
I remember transitioning from 10Base2 to 10Base-T. For all its failings, 10Base2 is one heck of a lot easier to crimp on new connectors. Cut to length, spin the wire stripper, crimp on the pin, snap on the BNC connector, and crimp the barrel. I could do it about as quickly as I could type that sentence.
RJ45 connectors will never be as quick.
We don't like them because the passthrough side that's open can get stuff in them, which corrodes the strands inside the cable, or can cause shorts and other issues.
I also don't like the 'combo' crimpers- I vastly prefer my ratcheting crimp frame and it's dies.
Oooooh... and the game Hunt-the-Terminator whenever you arrive on a new site sans any network diagram... most usually the fault was a loose BNC.
I resorted to using insulation tape to ensure the BNC connectors does not get screwed loose by idle fingers...
Always a loose BNC to screw up the network.
[holds up two fingers in sign of a cross]
I worked on Token Ring at Data General and 3Com. I grew to dislike it intensely.
1. The ridiculously inflexible and generally cumbersome "official IBM" cables and connectors
2. The rigid jitter spec (and the whole shared media thing in general)
3. The copyrighted MAC code resident on each and every interface card (Ethernet mac code was free)
4. The difficulty of getting the damn equipment through radiated emissions testing, especially multiport switches with all the outputs clocking in sync with one another.
5. UTP Token Ring was an EMI disaster
I know, "tell us how you really feel", right?
The problem was, it was a great design, just not for general office networking. And it couldn't grow...past 16 Mbits/sec. As soon as 100BASE-T came along, Token Ring died a well-deserved death.
So, what part of it am I responsible for? A small, *very* cheap VME bus Token Ring card, designed for Data General's 88K AViiON workstations (RIP), required for a US Government contract. I'm rather proud of this one, the Marketeers told us that if we couldn't design one cheap enough, they'd go external and buy one. Challenge accepted! The design used the TI TR chipset, which had a switchable Intel/Motorola bus interface. So I set it to 68K bus interface and used a bunch of buffers and programmable logic to make it look like a VME bus interface. Total cost was the PCB, the TI chipset, some RAM and the buffers and PALs. Never saw the light of day, but I have a couple in the attic. Code name was VILYA, one of the three rings of power from Lord of the Rings.
At 3Com, TR was what got me hired after DG laid me off. I worked on some TR switches, none of which were very memorable, except for one, where I implemented a source routing parser in programmable logic, my first Verilog project -- simulated and then used the simulation code as the basis for the synthesizable code. It worked, but, since this was Token Ring, it never went anywhere.
Thanks for the memories...err...nightmares?
In a previous role I'd been moved on from being IS Manager to working on a global ERP implementation, leaving local systems in the charge of my operator (who was ex-finance and had only a few years IT knowledge*).
A while later they decided to replace the thinnet (10base2) networking with Cat-5, an easy enough job except that this was in the days before autosensing on router ports (for the youngsters, you had to use a crossover cable when connecting routers and switches together - the cable pouch in my laptop bag still has one). The installers hadn't explained this, and had left a mix of unlabelled patch cables behind so when my former assistant did some repatching most of the network went down and she couldn't work out the solution or even undo her changes.
I was asked for help and quickly sorted it, then explained the problem and got the cable tester out to label all the patch cables. A couple of days later senior management told me, in no uncertain terms, that I shouldn't have got involved, it wasn't by role, etc. I did ask whether I should have ignored the problem and left most of the business unable to work but didn't get a response! I already knew my days were numbered and was only hanging on for the redundancy payout which came a few weeks later - as I expected the whole place was largely closed within a couple of years.
*One of the senior managers once instructed me to, "Make sure xxxx knows everything you know!" My suggestion that she would have to go to University then gain at least 10 years experience first didn't go down well.
(for the youngsters, you had to use a crossover cable when connecting routers and switches together - the cable pouch in my laptop bag still has one)
earlier this year, while diagnosing an issue, i was asked if a rj45-rj45 roll over cable was needed. I didn't think the guy asking was old enough to know what one of those was!!
Funnily enough, I made one of those when we had a new piece of kit that they were having issues connecting to about 2 years ago.
Got me a recognition & a Amazon voucher for about $20 (The rest of the lower value stuff in the "rewards" catalogue was expensive overpriced crap from Best Buy).
A while later they decided to replace the thinnet (10base2) networking with Cat-5, an easy enough job
An easy enough job in one room, or perhaps one small floor of a building. But an entire building, replacing a single cable run with an individual one to each device could be a major undertaking involving diamond drills, cable ducts, auxiliary cabinets, switch banks and patch panels in racks, cable trays and more.
I worked in a ~ 50s or 60s built 10 floor building with a stairwell with a window alcove running the full height to let in some daylight. Then when they recabled at one point (maybe about 15 years ago?) they repurposed the whole ground to top window alcove as a giant cable run. No more natural light for us when climbing stairs! (but, it has to be said, a better network speed & capacity).
A bank branch had work done to rewire one of the offices with 10base-T. The contractors used a nice diamond core driller to get down to the "IT room" in the basement.
Gravity happened: The core fell down into the basement. The contractors called it a day (hey, it's pub o'clock and now the IT guys can put in their wiring over the weekend.)
What they had not seen (maybe on purpose): The falling core had damaged a sewer line that promptly flooded the terminal concentrator underneath. The IT guys were understandably p***ed by the condition, dunnikindivers were called, and the whole branch was closed for a week for decontamination and repairs.
One of the senior managers once instructed me to, "Make sure xxxx knows everything you know!" My suggestion that she would have to go to University then gain at least 10 years experience first didn't go down well.
"Everything you know"
Including where the bodies wrapped in blood soaked carpet are buried?
These Captain Bollocks* of Manglement expose how little they about anything when they drop these gems of wisdom.
* as Gus Hedges (Drop the Dead Donkey) was once referred to and who famously asked "Are we cooking with napalm? You bet!" Indeed.
I was once asked, as I was leaving a position as the chief engineer for a corporate data network, to "write down the procedure for configuring a router for a new site". This was back when many of our remote offices had been operating as their own islands from a networking standpoint but were now being brought into the corporate fold, multi-protocol networks were common, and we were running DECnet (Phase IV), AppleTalk, Novell IPX, and TCP/IP routed to many locations, and some other protocols that were not routable (e.g., DEC LAT) bridged to many locations. There were some aspects of the router configuration process that were not in the manuals but could be written down as a procedure - setting the name, management IP addresses, and other elements common to all routers, fitting the office into our IP addressing structure, etc. I had already pushed to make the router deployment process as "cookie cutter" as possible, but much of the process of bringing a new site onto the network involved doing a site survey, figuring out what they had and how it worked, and then configuring the router (and/or making the site admin make changes to conflicting addressing, etc.) to match and make everything work as well as possible.
I listed the formal education I had related to networking, reminded the requestor of the locations of all of the network documentation that we maintained, and pointed to the Cisco documentation set (which at the time extensively documented each and every protocol their router products supported in addition to the router products themselves), and said "its all covered in these sources." It would have been nice if everything was standardized to the point where a simple procedure could be written, but that state had not yet been achieved in our evolution.
as soon as loopback cable was mentioned. :)
Crossover cables were bad enough back in those days but at least they were (usually) bright red and detesticulation or defenestration would subsequently attend anyone bringing one into the building.
Fortunately our "boffins" couldn't successfully change a fuse let alone make a patch lead even with a Krone tool which I wouldn't have on site either. Cheaper and more reliable to buy patches in bulk.
Having endured coaxial ethernet and some odd tech run over rs232 wiring I had very little patience for networking by the time twisted pair/10BaseT arrived.
Even with twisted pair/star topology most of interconnection was via (dumb) hubs which were little better than repeaters or switches (adaptive bridges) that didn't do spanning tree so if some clown intrepid boffin were to introduce a loop into the network topology a visit from Mr Cockup rapidly followed.
I will credit that in almost every case said clowns took the simplest approach and managed to plug two ends of a single patch into the same hub.
Igor's loopback cable is the ultimate simplification as you only need to plug one end in. :(
Like any RJ45 cable with damaged plugs, Igor's cable would have very quickly been introduced to Mr Sidecutter and Mr Miniskip if not also Igor himself.
Typical of the environment I suppose, but why didn't Igor or the network admin* use a standard patch and a 5 port hub - the magic lights work the same and even in those days hubs were as cheap as chips? But they didn't ask Evans either. ;)
Youngsters enjoying the luxury of standards based vendor independent autonegotiation (that actually works) and much more just wouldn't believe you. I guess the "joys" of Infiniband await any contemporary bright young things overcome with hubris. (Those the gods would destroy...)
* as a species exclusively either brilliant or ought to have been strangled at birth (or earlier.)
Introducing loops! I once had the fun task of tracking down a loop. The network guy was going crazy trying to figure out why the network was crumbling. As luck would have it, I passed an office earlier in the day, some admin assistants were cleaning up and rearranging some desks. As I passed, I saw a network cable on a desk. Once the alarms were going off, and the phone was ringing off the hook, we put boots on the ground. I heading to a switch closet, and I passed that same office again, and saw that same network cable neatly coiled up and plugged in to two adjacent jacks! It was from that day forward that I physically unpatched all unused connections at the patch panel. I would only patch them once "we" placed an endpoint requiring the connection.
> Crossover cables were bad enough back in those days but at least they were (usually) bright red
You know, you just have to look at the plugs and check how they are wired. Every single of those two to eight cables within a CATX cable is color coded, and the plug are see-through.
This is the one true patron saint of lab techs.
"This is the one true patron saint of lab techs."
I have encountered some breathtakingly peculiar tab technicians but your Igor reminds me more of a few academics I have dealt with who were, if anything, more sodding peculiar (and with lashings of attitude.) :(
The tie and mad Marty Feldman eyes are diagnostic.
At one point I was in a school with a 10base2 network of DOS PCs and it would quite often fall over because somebody unplugged it somewhere along the line. (There was only one line and if you unplugged it without fitting, if I recall correctly, a 75-ohm terminator, the signals were just reflect and collide with themselves and nothing would be able to transmit.)
The IT teacher's method of dealing with this was to walk around the school inspecting the whole line terminal by terminal.
I figured I could probably do that faster by binary chop. I obtained my own 75-ohm terminator and ran straight to where I thought the midpoint of the line was. If inserting the terminator there brought half the network back online, I knew the problem was somewhere in the second half, otherwise I knew it was in the first half. (Thankfully there was usually only one problem at a time.) I'd then go to the midpoint of the broken half and test again, and so on
I did get told off for running in the corridors a few times, but by the time I crossed paths with the IT teacher I usually had enough information to tell him at least which room the break was in.
It also gave me a bit of a reputation as the crazy geek sixth former who was literally running around to save the school network whenever it went down. (I didn't have a costume or action music though.)
You're right it probably should have been 50ohm. I have a feeling my bootleg terminator was 75 ohm though, but the network seemed tolerant enough not to mind in my short tests. Assuming the other terminator was 50 ohm, we'd be looking at a parallel resistance of 30 ohms instead of 25 (although that's then affected by the cable in between; I'm not sure I completely understand the physics)
From memory the wrong terminator resistance worked a lot better than no terminator.
The pain was often where someone enthusiastically plugged the end cable directly into the network interface card. Which was acceptable as long as the card supported this and had the jumper or, for the utterly luxurious and overpriced cards, a dip switch, to connect the internal terminating resistor. Such cards led to other tedious tasks where one had to find the rogue card with the terminating resistor enabled or, once, where regardless of the jumper/dip switch setting the bugger had enabled the terminating resistor anyway.
Damn, I'd suppressed some horrors! :)
We were migrating from token ring to Ethernet.
We also had several segments wired using rj45 cabling. This meant there were a lot of rj45 to ibm type1 (square blocks).
Someone plugged in a cable wired for token ring into an Ethernet switch, the pinout meant the transmit or receive lines shorted out and looped the network.it took me two hours to figure that out with a load of angry sales agents unable to do any work.
As soon as it was unplugged everything started working again.
That wire was treated to a pair of scissors followed by the bin.
Someone ...That wire was treated to a pair of scissors followed by the bin.
That "someone" wasn't similarly treated?
I am trying to recall a humourous sign from years ago in Benny Hill "faux German" along the the lines "Auchtung! Nicht touchen oder sie Ballen geschnipt!"
Probably too politically incorrect now (like Benny Hill) to see the light of day.
Although just last week a young local political candidate during her campaign was promoting a local German cultural festival on social media thought it clever to appear beside her electoral poster with her picture with a toothbrush moustache pencilled in. Stupidity clearly isn't the sole preserve of the aged.
My RS-232C notebook (lost in a move) had the most wonderful photocopy from a page of a book. It was a state diagram showing the basic (no compression algorithm negotiation) modem dial-up handshake sequence.
Though I've looked, I haven't been able to find that state diagram, nor one like it. :-(
Mine was repairing my cheap headphones. Used some of the stack of heat-shrink my dad had.
Didn't find out for years that the heat-shrink was worth more than new headphones at the time! Dad worked where they made it, though, so 1 ft. scraps were fair game for employees.
Our stocks of heat shrink kept mysteriously vanishing. Eventually manglement started buying yellow heat shrink instead of the black stuff. It was easier to see in the back of a rack or inside a unit.
Nobody took the yellow stuff home. It makes all the wiring on your British motorbike look naff.
Icon - trying to start bike in the rain.
Nursing home many years ago. A carer (not from the agency I was with, thank god) wanted to get one of the residents up the lift, but it was fiddly getting him ready for it and for some reason she didn't want to get him ready and then push him into the lift. She knew that if she blocked the door it will stay open. What she didn't realise was that it was an IR beam. A single one at about ankle level.
So she wedged the door open with a mop handle and after a few seconds it tried to close and made a weird clank noise.
Guy in the lift, mop removed and... door failed to close. It looked a little askew, I'm guessing it had popped off it's track?
No worries, this person says cheerfully, I'll hold on and get you up to bed and then I'll write it in the report book so it can be looked at. She turns the guy around, puts herself by where the door would have been, presses the upstairs button and then reaches up and presses something up in the top corner.
The lift starts to go up, and that's followed by a god awful noise. She starts screaming, the resident is screaming, the entire lift is tilted, and don't even ask what happened to the door.
We had to call out the firemen to extract them from the lift.
I went back there a couple of weeks later and the whole thing was boarded up, which meant we had to carry the meal trays upstairs one by one. I asked the charge nurse about it and she described the damage using the word "phenomenal".
Back in the '90s we had a 10base2 network. As is often the case, the network got flakey and there were three of us running around checking all the computers. One of the users got my attention and said he'd unplugged his computer. Worried that it would be a mistake to leave it unconnected, he got a F-F adapter and a terminator...
... You know the rest.
I only gave him a small lecture amounting to "don't do that again, leave the network stuff to the network guys, ask before you do stuff" because he came clean on his own.
Bits of kit you could stick between a PC and a printer, modem, scanner etc.
LEDs to show you what lines were going high or low and jumpers to swap the connections around.
Once the kit chattered into life you knew how the permanent cable needed to be wired.
Or you could see the PC was sending data but the printer wasn't responding and needed some percussive maintenance.
Them were the days.
For parallel ports, there was Joan Riff's parallel port analyzer, named "DIAGS.EXE". It ran under DOS, and displayed the logic states of the parallel port lines in real time, using PC graphics characters. With an appropriate 3-way connector, the PC you ran this program on could show you what was (and wasn't) happening between a different PC and a printer or other device connected to that different PC.
I'm surprised no one has mentioned Starlan or X.25.
I worked for a company that developed TCP/IP software for PCs. To test it, the office had a heterogeneous network with 10base2, TokenRing, Pronet, Banyan Vines, and Starlan.
Writing the driver for TokenRing was a pain. Not actually writing the driver, but getting the specs from IBM (this was 10 years before the Web took off). I spent a day on the phone calling people at IBM. Finally some pointed me to IBM's Librarian who could sell me the manuals I needed.
We didn't have an X.25 network, but we had customers who did. So I got to study the customer's 1" thick bound printout of hex dumped packets to figure out the problem (turned out it was a Telnet bug). I got good at finding 80 45 in the packets.
Man, those old network hubs are why I have trust issues. Connecting a older PC with a 10mbps card to a 100mbps hub knocks everyone else with a faster card down to 10mbps. This is why I keep insisting on switches with dedicated uplink and downlink ports nowadays. I know they don't have this issue anymore but deep inside I feel dirty for connecting a 100mbps NIC to a 1Gbps switch.
"What kind of net access do you have that's over 10mbps?"
For giggles, I looked up 2.5GBps internet access at chez COCM. Lots of providers did it and I *nearly* pressed the button before I realised that, not only would it be about £15 more than my existing 1GBps current service but I'd have to replace *all* my network kit (brand new OpenWRT router, 24-port gigbit POE switch - not because of the switch itself but because it only has 1GBps uplink which would make having 2.5GBps slightly pointless, etc etc).
So I didn't bother.
At some point, when I'm going to be replacing the switch itself then I'll do the speed upgrade.
This reminds me all too well the time we had something similar that caused a full line stop in the factory for three days due to the sole incoming inspection test rig going down with faults on its internal networking. Everything was connected via network switches - internal on the rigs network chain.
Despite having no knowledge of the rig or what it did I somehow found myself getting dragged into helping diagnose the issues as the test rigs used apparently had similar tech and comms systems to the robots I repaired elsewhere in the company. Manglement were getting increasingly desperate (and loud) about the line stop. I arrived to a mess of wires on the test bench and two severely pissed off manufacturing engineers.
We eventually found a RJ45 cable connected into two of the internal ethernet switch's ports. The genius who built the rig had decided to not only make all the RJ45 cables the same colour, but another genius manufacturing line tech had grabbed a spare (identically coloured) cable from the cupboard to connect his laptop to "the internet". Upon finding he got no connection on his laptop, he just tucked both connectors neatly "out of the way" and didn't collate this with why his test station now doesn't work....
I left it to the manufacturing guys to explain this to their colleague (and likely apply a steel toecapped boot to his backside)
That reminds me a story from large company I worked for, which one day sent their 800 home as the whole office network was down.
It took several hours to find the culprit - someone plugged a cable intended to connect notebooks in the meeting room into another socket. Unluckily most switches in this office had firmware without loop detection.
That cable killed the network in the whole office because of the packet flood it caused.
At home I use TrueNAS, sitting on a Dell R720 storage server, which has a quad port network card. All the ports are plugged into the switch but only one port has an IP address. I'd done that because, at some point, I wanted to try bonding the 3 interfaces to get better throughput.
Through a combination of tweaking the config and a failing update (my fault) I ended up having to re-install the OS and re-import the storage pools (which were all intact fortunately).
The VMs on the box couldn't see the data on the box and the solution was to create a bridge, tie the IP address to the bridge. add the main interface to the bridge and then point the VM virtual ethernet devices at the bridge.
Did that, all appeared to be OK so I hit "save config". At which point my network went from 100% availability to about 10% - lots and lots of ARP broadcasts clogging up the networks. Went through lots of "fixes" including ensuring that STP was on on both the switch and the virtual interface (it was).
The fix ended up being quite simple. The 3 unconfigured ethernet ports were each generating ARP packets which the main (connected) interface would then get really puzzled by since it knew it hadn't generated the ARP and it would then respond with an ARP of its own.. Ad infinitum, ad nauseum-network-killing..
The fix was fairly simple - unplug the 3 cables from the switch (fortunately, I'd labelled them).
Network is now back to normal although a bunch of devices had to be rebooted because they didn't reaquire the network after the ARP storms.
> I'd done that because, at some point, I wanted to try bonding the 3 interfaces to get better throughput.
If you use SMB3 you can give all four ports an IP, the same on the other side, and it will auto-bundle the 1 GB connections.
Whether linux does it as well as Windows 2012 R2 (or newer) or Windows 8.1 (or newer) I don't know.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
