Beware the big bang in the network room Cables can be unkind, especially when one has the confidence of youth but not the dark cynicism of experience. Welcome to an edition of Who, Me? to tug at, if not the heart strings, then certainly the RJ45s. Today's hero is a Register reader Regomised as "Colin." Colin had accepted his first proper role as a network engineer …
Traditionally expressed as:-
"Experience is a hard teacher because she gives the test first, the lesson afterwards."
When you do have an expensive mistake like this, you ensure that the maximum amount of learning is done from it by other people, as learning from somebody else's mistakes is a lot cheaper than letting you commit them yourself. An entire new generation of people may be helped to avoid certain mistakes via this article.
Alas, the young engineer here didn't hear "Always multiply your repair estimates by a factor of four." else he'd have had a maintenance window plenty large enough. ;)
Everything's got to be 24/7 now. Even on ancient systems which were designed with a maintenance window in mind and to be offline while overnight batch processes are run, you find sales have sold it as some shiny new always-available thing, so now you've got to persuade it do something to do it was never designed to do.
Hmmmm. "Sales sold it as ... ", and I think that at certain times, the server should mysteriously go offline.
Not to punish the customer, but to alert management to the frailty of the setup. Network errors are normally the least detectable, ahem, traceable to an individual, as they can be done with a single physical cable. YMMV, some setups are more secure than others. but we all know the weak spots.
You might find then that cash will flow for proper redundancy. If not of hardware, then at least people providing TLC.
With any luck, the sales staff who sold the client something as something it is not in the first place.
I can think of at least one such coke-hoover* where I work who has cost the company several weeks unpaid development time in the last year, due to not-entirely-truthful statements made to clients. this has included me having to learn, and make changes to, a product, which in any reasonable universe would have been end-of-lifed at least a decade ago.
*metaphorically speaking, I'm sure none of my colleagues would ever touch the Bolivian Marching Powder, and any behavioural traits that might be associated with such are purely coincidental.
Seeing the prospect of being caught between management and customer in that situation was part of the reason for leaving one job. It had been sold as a replacement for the existing customer's system. Having seen that and knowing the database design of our I could see it wouldn't be the drop-in replacement they seemed to be expecting.
@jake
This has nothing to do with the article, but your comment reminded me of an incident a few decades ago.
I was a dietician/diet cook and had started work in a hospital. I can't remember what I was cooking but I needed something from the 'fridge. Said 'fridge was a big "walk in" ' fridge. On my "journey" to the 'fridge I ignored the couple of people who told me it was not a good time to go to the fridge. I assumed that as I was the new guy they were taking the piss.
So I opened the door and was confronted with a bang. Actually, a more accurate description would be "banging". The company bike was being banged to fuck by someone (I never knew who, all I saw was his back and his arse doing about 3000 thrusts per second).
I mean, what the fuck do you say when confronted with such a situation?
I did the only possible thing. I forgot that I needed whatever and said, "sorry" (yes really) and turned around and shut the door. And then put up with weeks of people piss taking me.
And Whisky is not good enough. I find Vodka way better. At least, that is my excuse...
In our old office, there was a tiny server room in the building's roof, which was more like a large cupboard. For some reason there was a mattress in there. I'm sure it was purely concidental that the network manager and one of his staff, both of whom were married to other people, were "at it"; it was a pretty well-known open secret.
I was expecting much the same at first, but then, after a bit of reading, I thought it sounded more like there would be a server or switch falling off the rack, or the rack tipping over, resulting in the titular 'big bang', especially with that talk of 'tug[ging] at, if not the heart strings, then certainly the RJ45s.' And even that still didn't happen.
Where is the Earth-shattering ka-boom? -->
Did you know a wall of cables from a stack of 5 48-port switches will prevent ALL airflow through a 19' comms rack? Also, if you use separate colours for each type of server (esxi, pdu, hsm, etc), that the waterfall of cables looks like a 1960's bead curtain.
anonymouse because while its been 9 years since this was done, its still in production. and while I don't go all zip-ties and velcro bands, neatness DOES count. for 6-8 port devices proper cable management means if you disconnect all of the ports to replace the hardware, they'll only reconnect, without straining, to the correct ports.
My instant reaction to _ANYONE_ waterfalling cabling in my racks is to remove it.
I simply won't tolerate it and on a new site it's the first cleanup job on the list
Cable management is there to ensure things can be swapped out quickly in an emergency and the horizontal row of rings below/above the switch isn't there for decorative purposes (nor are patch runs so taut that you can't trace a cable by tugging it and feeling the other end of the bundle - something picked up in telco days))
It also means that for switch swaps as described, the plugs aren't going anywhere once removed and you have an opportunity to check the rear of the cabinet
What do you mean you don't have a couple of trouble lamps?
What really irks me about most large switches is the way they tend to require 2 people to install - one at the back to hold the rear up whilst guiding the unit into place. My solution to that is to use ventilated 1U "L" rails(*) (beware the solid ones. Most switches use sideways airflow, not front to back)
* eg: https://www.network-cabs.co.uk/acatalog/1U-19-inch-Universal-Server-Rack-Rails---Adjustable-Depth----450mm-to-600mm-Fitting-AD1UADJSR_300.html
"for 6-8 port devices proper cable management means if you disconnect all of the ports to replace the hardware, they'll only reconnect, without straining, to the correct ports."
I haven't met one that might trip an issue at the 8-port size yet, but on 10, 12-port, or larger devices, that only works if said replacement device has its ports in the same physical configuration as the old, ie, only one row, or in two rows.
Or if the idiot designers haven't shifted the bank of ports from one side to the other.
There are a lot of people who should never be let near a RJ45 patch panel.
I have seen some bad ones the worse was a cabinet at a company my employee at the time had bought. The cabinet sat in a corridor and the patch leads extended in a pile of spaghetti which extended almost a metre in front of the cabinet. I upset the network person on site with my reaction.
I said get a maintenance window and unpick each cable label them and it back beater.
Mind you I have done the same pulling a cable and finding someone has wound something else round it or even worse used spiral wrap on a few cables.
"The cabinet sat in a corridor and the patch leads extended in a pile of spaghetti which extended almost a metre in front of the cabinet."
I can't beat that on amount of spillage, but maybe on time reach spillage. Network cab in a school. Not a big one, maybe 6-8U, about 8 feet up on the wall. The door will no longer close due to the mess of cables. This was only a few week after I'd installed it as part of an entire wiring up of the school. All wall points went back to the patch panel. All wall points were live as I'd carefully and neatly patched every patch panel port into the switches, everything labelled and colour coded. There was absolutely no reason I could fathom as to why someone had gone in and started re-routing cables. After all, we'd designed it all for them, got the sign-offs, set up all the proper VLANS and wired the correct ports to the correct VLANS etc. I can only imagine that the school technician had forgotten all that and possibly after some classroom re-arrangements had re-patched the cables rather then edited which ports were in which VLAN.
" I can only imagine that the school technician had forgotten all that "
You imagine he retained it in the first place, or that whoever was assigned the job even knew about VLANs
The solution to that is to produce photos of the "before" and say "that's not how we left it and this isn't supportable, it'll cost to put right"
Or a big bunch of 'spare' keys in the bottom of the toolbox
A few months back had an idiot manage to lock the keys to his desk inside the desk(*). Couldn't find the correct key on the bunch but found one that worked if you only partly inserted it then 'j-j-j-jjiggled it, Granville'
(* he has a 'cash box' of vital USB sticks inside with the key on the same bunch. Instead of unlocking the drawer and pulling the key straight out, he unlocked the drawer, turned the key back to 'locked' before withdrawing it... and the drawer went 'click' as he nudged it closed)
True, but you'd have to either work standing in glass (which contrary to popular belief about safety glass is still feckin' sharp) or you have to spend a lot of time cleaning the debris up before you can get to work proper. And in either case you have to spend either time, money or both satisfying the boss/customer/client/landlord who's cabinet door you broke.
Glass doors are so manglement can look at the leds. Important if you've modified your optics to reprogram anyone who gazes too long or too deeply.
As for doors, sometimes ignoring the lock is the best approach. Doors often have spring loaded hinge pins top and bottom that can easily be persuaded to open.
Ventilation gets more fun. So if AC's forced air underfloor, then you probably want fan trays in the top of the rack, with panels all round to direct airflow up & out. If it's a hot/cold aisle setup, racks should be set up for that. Becomes FUN! as racks fill up, holes in floor get blocked and doors get left off cos there's to much cabling to close them.
But look on the bright side, most kit these days lets you monitor it's temperature.
Actually, those particular locked doors are to keep Bosses from flipping switches ... To keep the rest of the riff-raff out, we put locks on the doors of the server room (data center, whatever). After many decades, here's my list of folks who don't belong in Corporate Data Centers:
Middle management.and higher (applies to management track folks only).
Any corporate bod with a non IT related Doctorate.
Most children of employees (there is nearly always an exception or two).
Tour groups.
Vendors.
Janitorial staff/cleaners.
Either pick the lock, or use the patented 'yale-latch-defeating-disposable-loyalty-card-round-the-frame' trick to get at the single cable in someone else's comms room that you desperately need to move on a Friday afternoon when they (Corporate IT) do not have anyone available to open the door *.
*Except when the silent alarm goes off and two of the feckers turn up with 'security' in tow 15 mins later. I got away with this as my boss significantly trumped theirs, this ended a previously un-diagnosed and mysterious 6-week outage, and it was Corporate's fault anyway - the 'feckers' did some cable & equipment moves 6 weeks ago (Without telling anyone like wot they were supposed to!) that took a rather important Ethernet segment out to ~130M.. No wonder all them collisions were 'late' :-)
"...and to always declare a much longer window than you think you need."
A factor of four, according to the Scotty Principle
Sounds like the logic used by the maintenance crew around here. Changing a bolt that has worked loose? Five minutes work. Factor in all the farting around, risk analysis, toilet breaks, having a coffee or two, you're looking at hours to change a bolt. Ten hours? I don't think it's been stretched quite that far, but not for lack of trying.
How many years???!!!
It's dice. Die is singular.
But I always believed the railway running each day was determined by a modified version of Owzat! With minutes delay on one bar and if you scored an Owzat! Roll the other bar marked with Bridge Strike, Persons on the Line, Derailment, Signalling Problem, Trackside Fire and "Leave 'em Guessing"
Re: "as a prime example of why you should never go 'big bang' with a swap-out like this, not to work onsite alone, and to always declare a much longer window than you think you need."
Whenever you're calculating a time estimate you always - ALWAYS - put a healthy extra for unforeseen events. We call it the fear factor (that you use to multiply your real estimate by)
It's one of those things you learn by experience.
So switches have lots of ports, which means lots of cables. Which then usually means a rats nest of unlabled cables. And may also include some 'spares', or faulty cables that have been left dangling. There'll probably also be at least one with a dodgy crimp, and some devices will keep their Ethernet ports shut down after LoS.
You need access front and rear. Trying to thread a switch through a nest of cables is never fun, and increases the risk of dislodging or damaging cables. Having some form of console or syslog access can help spot devices going down when they shouldn't.
Then for time allowance, always include enough time to roll back the change, and when to abort.
And ideally, always use patch panels, never cable directly to a switch.
And ideally, always use patch panels, never cable directly to a switch.
If I could upvote you more than once for this alone, I would.
Decent software engineering abstracts things into layers, which can be treated as modular and swapped in-and-out. This is the hardware equivalent, and should be standard practice. It's the difference between having your car's wheels bolted to the axle, and welded on.
"Whenever you're calculating a time estimate you always - ALWAYS - make sure you have full access to all devices by having keys in hand for all locked cabinets"
FIFY
It 100% was not Colin's fault. You assigned him a project yet did not grant him access to all tasks the project could entail - you failed to give him the keys to the rear of a server panel.
Then tried to blame him when he needed said access but could not acquire it.
Blame the management, not the worker. Forcing a mechanic to work on your car but then saying "You can't use the 10mm socket" will NOT get you guaranteed results.
Good preparation, but a number of red alerts were ignored, here.
"I felt some resistance and pulled gently since the RJ45 jacks on Cat 5 cables can take some strain without damage, and all the cables were out anyway..."
Red alert one: if there is resistance and you can't explain it right now, stop, and only restart until you're confident it has been dealt with.
"The Data Centre helpdesk team had not shared their step ladder due to "health and safety""
Red alert 2: if you don't have every tool you'd potentially need, refuse the operation.
"Colin hadn't been given the key to get into the rear of the rack."
Red alert 3: like #2
Transformation is about sorting out years of mess, so you'll always step onto some crazy temporary "fix" from a long time gone permie or contractor.
It's really like walking on a mine field. Every bizarre thing counts.
Also, red alert #4, it sounds like the first time he looked at the setup was when the maintenance window started.
So apparently he agreed to do something and set a maintenance window time without even eyeballing the situation.
There's lots of times when "it's a simple printer switchout" has turned into something like "the printer has been permanently installed in a custom built alcove" or the "printer is something from the '70s that uses neither USB, parallel, or other bus known to modern man, and has a custom interface board."
So there's been times I've have to NOPE the hell out, and agree to fix it on a "best effort" basis.
Red alert one: if there is resistance and you can't explain it right now, stop, and only restart until you're confident it has been dealt with.
I'd say thats more of a Bronze alert. Or mauve.
Even if it does mean changing the bulb.
A subtle jiggling of the unit at "below plug releasing levels" will give you geographic indication of the issue.
The trouble with IEC13 plugs (ie kettle leads) is that they sometimes manage to work their way out of the socket, to the point where something as gentle as another cable brushing past it can break the connection.
I managed it once while just swapping some network cables into a rack, one knocked a power lead in a blade enclosure.
Now, this enclosure did have multiple redundant power supplies, but the one that ended up drawing all the load was bad, because it immediately popped, and the remaining two couldn't handle the load and shut down, taking all twelve blades with them.
All of this was silent, so the first I knew about it was my boss bursting into the room, frantically asking what I'd touched.
I've seen it happen often enough that I insist on using self-anchoring cord sockets. These solve a LOT of aggravation later on (think: someone fumbling in badly lit back of rack knocks out black power lead from black socket in a dark environment. Do you think they'll notice it?)
Agreed.
The moment the stepladder was denied I'd cite my own Helath and safety plus "local obstructions" as a reason for cancelling the job (and charging for the hours spent onsite or XYZ minimum charge)
When manglement get a few bills like that, they tend to start asking pointed questions of underlings
Off topic...ish
A shipyard in Scotland is trying to build two ferries. They are over budget and late. The latest problem is the data cables are not long enough.
The cables were long enough when they were ordered a couple of years ago, but someone decided to move the displays and instruments, so the cables are no longer long enough. Swapping the cables around is not an option, so even more will have to be ordered.
The obvious answer would be to move the displays back again - but that's another can of worms.
I know someone who carefully specified 'line termination should be within 2m of the router' because the Cisco routers came with a 6ft(1.8m?) cable for free.
He learned his lesson (order 5m cables, just in case!) after discovering that at more than one site the line termination had been mounted half way up the wall opposite the comms cabinet because the line guy had thought "I'm about 6ft fingertip-to-fingertip, and it can touch both the line termination and the router!"
It was obviously quicker to pick up a few 5m cables from PC World than argue with BT/Virgin/whoever(*) to get them to re-site the termination somewhere sensible
(* Virgin were taken off the list of 'approved suppliers' because... well, life's too short!)
> (* Virgin were taken off the list of 'approved suppliers' because... well, life's too short!)
If you think that's bad, try getting an outage fixed when Virgin and BT are pointing fingers at each other, so won't go out in less than 24 hours (on a 24*7 4 hour onsite contract)
Mind you, that whole project seems to be jinxed.
True dat. The sunk costs fallacy kicked in years ago: it would now be far, far better and cheaper to sell both hulls for scrap and get someone else (the Poles, probably) to build a couple of new ferries, but far too much political capital has been invested.
Are you sure about the display movements? I haven't seen that reported anywhere. As I recall, they have already had to replace all the power cables in the Glen Sannox once because Ferguson's put them in out-of-order when they were waiting for other problems to go away.
The only good things about the ferries clusterfuck are (a) they give us all a good, if expensive, laugh and (b) they make the Edinburgh trams project look pretty good.
Look at it this way, the company has now just realised that their biggest client has no redundancy. They should be thankful it was only a cable that was unplugged and not the switch being busted with some of the lead times coming from people like Cisco now, we're getting quotes of delivery in a year now.
I wasn't directly involved with this. But a new electric substation was built at a utility I used to work for. Racks were installed in the control house and the electricians dutifully installed the protection relays and wired them per the provided diagrams. But they failed to leave enough slack in the cabling or utilize the (provided) hinged cable brackets which would allow the equipment to be rolled out for maintenance on its slides.
The relay technicians came to the job site next to calibrate and commission the equipment. Upon finding that the cabling had not been installed properly to facilitate future maintenance, they proceeded to take side cutters and cut every cable in the racks. And then call the electrician crew back to do it right this time.
and labels.
Every cable tagged with a code number , C93 for the arm overtravel sensor for example.
So when the arm overtravel alarm goes off, it must be a doddle to swap the sensor out.... except the sensor is fine and cable is blown somewhere.
Open the control box...... C93........... C93 .............. C93............ ahhh there it is.... coupled to the 24 volt DC supply ......... eh?
OK resort to the wiring diagram......... C93 .......... C93 ............. C93............. theres a C92.. but that says its on the HP sensor..... but its wired to Wrist motor encoder line on the arm.....<looks on the arm> that says D12 and the manglement are waiting .. oh so waiting..... with the looks.... the glances..... the wrist tappings.....
resorted to tying a string to the cable , pulling it out, and seeing where it was realy plugged in... A47....
Beer because I needed one
All of our PCs have a non-removable sticker with the host name in big friendly letters, so if there's an issue the user knows what machine they're on. And if the PC isn't accessible, for example inside a kiosk or behind a wall mounted TV, then the sticker is put on those.
Except the workflow changed, and now all the stickers are put onto the PCs when they're received, so we now have to keep look-up tables saying what they used to be going back three generations.
My work laptop didn't have a label on, so I stuck one on there. I was forever having to go and look up the host name (seemingly random alphanumeric string) whenever I needed to put the local SQL Server instance name into a connection string to test something (and no, "localhost" doesn't work in all situations). I got fed up with it enough to get the label printer out from the back of a drawer.
I remember a guy getting some Saturday overtime to hardwire some RS232 connections (~10 wires per connection) to a remote test desk
Come Monday we ran a few checks but just got weird results. Looked at the tag block in the back of the test desk, expecting to see a neat pattern just saw a jumble of colours... same at the other end of the cable
NOW he tells us he's colour-blind!
This reminds me of the time when I was working late when my colleagues and boss had gone to the pub. I was just about to leave to join them, and when I walked past the server room I could hear lots of UPSs complaining. I went in the server room, traced the cables back and found far too many devices were powered from a single 13A mains plug, which had partially melted. After reorganising the mains cables so they used more than one socket, I was then very late for the pub, and no-one seemed bothered that there could have been a fire or something. If I didn't have such sensitive hearing and wasn't so dedicated, I could have had more beer (and probably had to perform the BCDR process the next day).
Or better yet, the only person who either had permissions and/or knew how to do the maintenance quit long ago because the company across the street was offering 2X the salary and was offering benefits like not being on-site 24/7/365 because you were the only person on staff. And BTW, he was the only one that knew the key-codes to get in the server room door and the only one who knew how to change them. So, as time goes on, the company has less and less access to its computers.
Or, how about maintenance is required but nobody knows where the machine is? Indeed in one case they didn't even know what CITY the machine was in.
I get to the customer's site to troubleshoot something in their control room, a.k.a. IT closet. Hard start, 0600 hours (Eastern, AKA GMT-0500). Not allowed to be in the control room after the store opens at 0800. I call the number I was given for their help desk, and get a recording along the lines of "Your call is very important to us. Please leave a message, and we'll get back to you as soon as we get into the office, at 0900 Pacific". Which would be noon, Eastern time.
Aarrggh!
I remember doing a cabinet tidy up which involved upgrading some servers. All went well and as we were also rationalising cabling, we decided to move one server to a different network port. This was not part of the plan, but all was working just fine when we packed up for the evening.
Fast forward to the next day - some, but not all client machines on part of the network were falling offline.
It took some time to realise that the reason was the port we'd moved the server to was untagged on the correct VLAN but also tagged on several others.
The DHCP server on this machine didn't understand VLANs and was responding to every DHCP request both on its own VLAN and all the others too.
As there was a DHCP server on the other VLAN this explained why only some clients were failing. Some swift switch reconfiguration fixed the issue and that server's DHCP was fixed some time later.
Long time ago, during one migration the main DHCP server was switched off and on again. Turned out to be running the DHCP client itself, hence not getting the IP address and of course no one knew the local password. Like someday it got the IP address via DHCP, kept it after the old DHCP server was gone, eventually got promoted to the DHCP server role and no one spotted anything until the reboot who knows how much later. A temp DHCP server quickly got it back online but the whole situation was quite hilarious. It was probably seen too important to be rebooted at all.
Years ago when I was working (mostly) desktop support, facilities wouldn't allow IT to have a trolley because we might break something by bumping into it. So we had to carry everything ourselves, back in the day when desktops, printers and of course servers could be massive. LaserJet IIISi printers were a favourite at over 40kg each. Apparently IT staff were more expendable that the doors and furniture.
" facilities wouldn't allow IT to have a trolley because we might break something by bumping into it"
I've solved that one in the past by invoking elfin-safety and insisting that facilities staff carry the heavy stuff instead of risking expensive technician injuries
The moment they started using trollies it got photographed. Of course it was all about "demarcation" and some petty manager pissing over everything to maintain his empire. That's fine but if it now takes 3 weeks to move heavy items, that manager will be the one who frustrated end users (and higher manglement) get directed to.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
