BOFH: The evil guide to upgrading switches

BOFH logo telephone with devil's horns It's five in the morning and the PFY's at Mission Control making a few changes. His final task will be a firmware update for the core router – which will, if we're lucky, stop a six-monthly out-of-memory issue. He'll have already updated the firmware of our security camera system (and …

[Chris 15]

rule 1

Make sure to remove any incriminating codes of any description to cover youre tracks

(rule 2 is to put an 'old' asset tag on the device insinuating that it has been about for ages, cost peanuts, and was sent foc as an afterthought by a vendor with some other kit 'to get it out of their inventory as they had been sitting on it for ages')

[b0llchit]

Re: rule 1

Always save those old asset tags from deprecated equipment. They are recycled, responsibly.

[Yet Another Anonymous coward]

Re: rule 1

>Always save those old asset tags from deprecated equipment.

I have a box full of old laptop bottom cases with asset tags - we've never found the procedure for disposing of a corporate PC, so officially I have all these laptops

[DS999]

Asset management is a pain

Way back when I was in charge of IT for a division of a big research university, and the previous occupants of my position were a guy who was only there about a year, and before him the role had been performed since the Dawn of Time by a CS professor who treated it like his "spare time" job (which it was, given that teaching was his priority)

So when an audit of assets was to be done the one year guy before had been lucky to avoid I found that pretty much everything purchased in the past 25 years was still on books as an "asset", even if it had long ago been disposed of. So naturally an audit that asked to locate a couple dozen random items was doomed to fail when over half those items were so old they were obviously no longer in use.

Even some items a professor who collects "antique" computer hardware was sure had been disposed of via the "surplus" store which is supposed to handle removing it from the asset rolls were still on it - he was sure because they had purchased several items from surplus that were still on the asset rolls.

After much back and forth I managed to get the powers that be to agree to a one time ability to trim the asset register in exchange for a full audit. They agreed anything older than 10 years would automatically be removed (fortunately stuff like desks was on departmental asset registers, so pretty much everything that old on my IT related list would have been essentially valueless even if a few of them like green screen terminals were still in use here and there) as well as anything older than 3 years that had an acquisition cost less than $500. That brought the list down to a manageable enough size that my staff and I were able to conduct that full audit, and they accepted our list of "this is what's left on the list we know we have and can locate" as the new list.

[Sampler]

Re: Asset management is a pain

Just how much didn't make it on to the list and in to your car boot?

[Dave314159ggggdffsdds]

Re: rule 1

Ime you get rid of them by stacking them in a wheely cage, taking them down to the basement, and watching the facilities guys (who insist you can't take anything past the doors yourself) push them off the loading dock. At that point it's their problem.

[Anonymous Coward]

Data destruction certificates

With anything with a hard disk you need to be sure its been securely wiped or destroyed. As desktop populations increased I moved form a screwdriver / frill through the HDD case shattering the platters, an internal HDD grinder to using external companies where the cost of the destruction and certification was paid for by the scrap value of the desktop kit.

PDA's / Phones brought further complexity and were often wrth so little as scrap that I would end up paying to have them destroyed

[NoneSuch]

Re: rule 1

B: Admit nothing and cover your tracks.

The mantra of the IT professional. AKA CYA.

[Anonymous Coward]

Re: rule 1

Just be very careful doing so if the 5S fad hits your workplace.

OTOH, a great BOFH side gig would be hauling scrap equipment from a business doing their first 5S. Get paid to haul stuff away, store the stuff somewhere, then sell it back as they realize how much they really needed that $20,000 custom tooled jig that looked like scrap metal.

5S:

Slimy consultant

Sells the concept

Scour cabinets, closets, storerooms, etc.

Stack anything that's either dusty, or not understood by manglement

Shitcan the lot of it...

(Other commentards can likely do better definitions...)

[Bogbody]

Re: rule 1

Ah yes 5S ......

Vital notes thrown away having been left in a tidy pile on a desk.

Yellow carded - remove in a week - shame the owner was away for 2 weeks.

:-)

[Anonymous Coward]

Re: rule 1

Shit, shower, shave, sleep, sex are my 5S's. The boss does not approve, though.

He's trying to impress the importance of 5S on the Helldesk guy to clean up the IT room, but Helldesk guy is having none of it. It's been an 8-month game of cat and missing mouse.

[Anonymous Coward]

Re: rule 1

I did do some 5s like sorting when I took over a desktop team. There were no filing cabinets available as they were all full of obsolete parts. Mainly for epson fx80 and fx40 printers which were being used in fire stations for printing job tickets. These were critical devise but were constantly failing due to age and rough handling.

I bought a small stock of replacement dot matrix printers (they used multi part stationary) and played the bad boy next time a printer failed insisting that a replacement was bought and installed (the same day) lo and behold that printer didnt then fail again 2 weeks later as had been happening with the lx40's over the next few months every printer was replaced and we wend from 5-6 critical printer calls a week to none.

[Blackjack]

Re: rule 1

Even better, use an older casing, replace the QR code label with one of those old"Intel Inside" stickers and if the thing has evident "Gaming kit" lights, disable them.

[Doctor Syntax]

Re: rule 1

I had a screwdriver set with an Intel Inside sticker. It fell of the case of the kit I was working on and stuck to the screwdriver lid so I left it there.

[iron]

Re: rule 1

I had a wallet with Intel Inside and still have a microwave with Designed for WinXP stickers. Neither has ever been on a PC, I used to work for the company that manufactured the labels.

[ShadowSystems]

At Iron, re: case stickers.

I once got my hands on a roll of the "Intel inside" & "Made for Windows" case stickers from a previous employer.

My friends & I took a trip to the local Fry's Electronics (an American electronics megastore that is no more) & spent an afternoon wandering the aisles applying stickers to random items.

We enjoyed the confused/amused expressions on customers' faces that found "Made for Windows" stickers on Apple products, "Intel inside" stickers applied to rolls of thermal printer paper, or both stickers stuck to something completely inappropriate like the forehead of the manniquin modeling the latest dummy models of wearable tech.

In my defense, we were young, dumb, & bored at the time. =-Jp

[Arbuthnot the Magnificent]

Re: rule 1

My office bin has a "Windows 8" sticker on it. Much more useful than the installation it came from.

[David 132]

Re: rule 1

Let me guess, you saved the "Windows 11" sticker for something white and porcelain in the restroom?

[Loyal Commenter]

Re: rule 1

One of the toilet cisterns in our old office had a "Made for XP" sticker on it. Wasn't me that put it there, guv'.

[Richard Pennington 1]

Re: rule 1

I have an abacus with an Intel Inside sticker.

[tezboyes]

Re: rule 1

Maybe that could be engraved ;)

[Uk_Gadget]

Virtual Friday BOFH

Damn needed that.....

[chivo243]

Re: Virtual Friday BOFH

I thought my calendar was off! I was about to reach for my phone when I finished reading ;-}. Will we get a bonus episode tomorrow??!!

[A.P. Veening]

Re: Virtual Friday BOFH

Will we get a bonus episode tomorrow??!!

That will truly make it a Good Friday.

[diodesign]

Re: Re: Virtual Friday BOFH

I could lie here in an attempt to make you all return on Friday

But no, we brought BOFH forward for those who want to do other things in an Easter break other than check out IT news and the internet

But hey you're welcome to drop by tomorrow anyway, half of us will still be working ;-)

C.

[John Brown (no body)]

Re: Virtual Friday BOFH

"But no, we brought BOFH forward for those who want to do other things in an Easter break other than check out IT news and the internet"

Are there other things to do? What other things?

[Anonymous Coward]

Re: Virtual Friday BOFH

Eating Easter eggs? Bloody painting walls?

[Oblivion62]

Re: Virtual Friday BOFH

Painting bloody Easter eggs? Eating walls?

[HandleAlreadyTaken]

Re: Virtual Friday BOFH

>Painting bloody Easter eggs? Eating walls?

Painting Easter eggs (often the color of blood) is an ancient tradition in Eastern Europe. Eating walls, not so much...

[Tim Hines]

Re: Virtual Friday BOFH

"... half of us will still be working ;-)"

So about the same as usual then? ;-)

[John Brown (no body)]

Re: Virtual Friday BOFH

Yeah, I just spent 10 minutes looking for Dabbsy's SFTW and was worried when it wasn't there! Hopefully that will turn up on "real Friday".

[John Brown (no body)]

Re: Virtual Friday BOFH

Still no sign of the column for this week. Is Dabbsy taking a week off? Is he still hung over from last weeks 10th anniversary celebrations? Have El Reg decided 10 years is long enough to suffer Dabbsys Drivel and sacked him?

[Doctor Syntax]

"we can probably fix it with a firmware flash in a couple of hours."

This was the switch which was just borked by a firmware flash so are we looking at a previous Choose your own adventure?

A. Go without a new coffee machine.

B. Break into the server room at dead of night by fixing the security cameras and access control to administer a firmware update that will bork the switch recoverably.

[Blackjack]

Maybe they have a spare in storage?

[John Brown (no body)]

Nah, the BOFH said they can do a serial port based firmware flash that takes hours to do because it usually fails 4 times out of 5 and is slower anyway, rather than the quicker network port based flash because now it's borked the port based flash is no longer an option.

RTFA, as they sometimes say around these here parts :-)

[Giles C]

I had to do that once, deleted firmware as the switch hadn’t enough space to hold two copies (long time ago) got distracted and typed reload before the new firmware was copied.

Had to get someone in the office (other side of the country) to upload it via the serial port.

5 hours later it could be started again.

[Arbuthnot the Magnificent]

If you haven't spent four hours waiting for zmodem are you even a BOFH?

[Anonymous Coward]

switch hadn’t enough space to hold two copies (long time ago)

I am only now (2022) awaiting delivery of some new Aruba switches to replace some Cisco 2950 which have exactly this problem. Never having had to deal with Cisco (or indeed any other "big name") before, the concept that a firmware upgrade carried the very real possibility of brickage, even back in the misty depths of time when I first met these beasts (2006), was utterly alien. They have (with one exception) been faultless performers over the years, but I shall not be sorry to see them go. Apart from anything else they don't talk STP properly with my newer switches, I've had problems with VLANs, and as for port aggregation...

Probably just me not being clever enough :-)

[Hot Diggity]

Choose Your Own Adventure

I think I am older than that Big Bang dust on the routers.

I loved the Choose Your Own Adventure books by Steve Jackson and Ian Livingstone back in the dark mists of time.

Thanks for that Simon. 40-odd years gone just like that.

[Anonymous South African Coward]

Re: Choose Your Own Adventure

Lone Wolf by Joe Dever... liked it when it came out. Got the first 3 or 4 books of the series.

And, yes, there's a whole subreddit devoted to the Lone Wolf adventure books.

Project Aon on Android is where it's at if you like to have a stab at defeating dark and evil forces.

[tezboyes]

Re: Choose Your Own Adventure

Yeah I downloaded the Android; of the Lone Wolf versions a few years ago, good nostalgic fun :)

[Blakey]

Re: Choose Your Own Adventure

The Steve Jackson and Ian Livingstone ones were "fighting fantasy", "Choose Your Own Adventure" was actually its own brand of gamebook - without any stats to track, die-rollijg or combat mechanics. I didn't like them because they were "for little kids", unlike my own very grown-up copy of "DEATHTRAP DUNGEON".

[Anonymous Coward]

Re: Choose Your Own Adventure

Deathtrap Dungeon.

The absolute best ever. That boody room with the ruby and the ceiling that dropped on you.

[Paul Crawford]

Year and a half old switch without trouble. Recently firmware updated and a month later all PoE power drops off, though "switch" is still working for traffic that is not dependant on PoE, unlike the security cameras for example...

Soft reboot recovers PoE. New firmware bug, or random glitch? Makes me wonder how often such upgrades are worth the risk of new exciting bugs versus any genuine fixes they include.

Time for some of this =>

[Flightmode]

A layer-2 office switch sitting inside a campus LAN with RFC1918 management address, port security (if not 802.1x) and unused ports disabled? If it ain't broke, don't fix it.

(Routers, however, are a different beast. Stay on top of your vendor's CVE publications and upgrade regularly. And not to the bleeding edge version of you value sleep over the overtime compensation.)

[Ozan]

Never trust point 0 release and wait for point 1 release.

[Oblivion62]

...wait a fortnight after the point 1 release...

[l8gravely]

I waited until Service Pack 11 of my backup vendor's software before I upgraded and it *still* broke NDMP backups of my Netapps to tape. I hate hate hate hate hate hate all backup software. Though I have fond memories of Legato Networker and it's fairly sane design and command line interface that actually worked, or could be scripted if you needed to do anything more interesting.

CommVault? Sucks.

Netbackup? Sucks.

Bacula? Less sucky, but still not ideal.

All of these tools emphasize the backup, not the restore. Networker had this wonderful tool for browsing your backups and finding what you needed and letting you restore it easily, and told you which tapes you needed without hassle.

CommVault... no such damn luck.

Can you tell I'm bitter?

[Anonymous Coward]

You should update your switches as well because nearly all of the old ones can be subverted with random malformed packet data (specific to each bug in their packet forwarding engines...).

On the other hand, port security, 802.1x and no unused ports enabled are DEFINITELY a good start.

[Flightmode]

I should have added "and no way to pivot from that switch to other network devices" to my list.

But I basically agree with what you say - if there are known vulnerabilities out there for your device, for the love of dog patch it. You don't want bad actors in your network (though I'm sure you never have infected bots on your LAN, right...?) being able to eavesdrop on or manipulate traffic.

[Triphase]

This is familiar... Switch dropped PoE randomly. Yours wasn't a Net**** by any chance?

[TeeCee]

Well done. For a brief and weird moment you managed to get me to wonder who made Netfuck switches.

[tezboyes]

Micro$haft of course!

[Screepy]

We very rarely updated our switches unless there was some really annoying bug that we had to solve.

However, a recent insurance audit indicated that in fact the unpatched switches could be grounds for the insurance company not to pay out in the event of something unpleasant happening.

We now have a remit to keep all switches up-to-date :(

We have a mix of old Ciscos, Extremes, and some newer Arubas.

The Arubas should be easy right? Just deploy from Aruba Central and sip tea while they all update seamlessly... Wrong! A good chunk just refused to update, some updated and rolled back, some successfully updated and then disappeared from Aruba central - never to be seen again. We now have a priority call logged with Aruba asking for them to get us visibility back on the switches (they appear to be running fine though).

Haven't even looked at the Ciscos, and the Extremes only get mentioned in hushed whispers.

Would be interested to hear if any other techs have similar remits...

[Giles C]

Patching process

Yes

Couple of companies back…

I worked for a large insurance who were pci compliant

We had a policy (it is probably still in place but I left 4 years ago)

Cve less than 5 check code and make sure not affected

Cve 5-8 2 weeks to get fixed

Cve 9-10 patch within 48 hours assuming a fix/workaround was available.

The high rated ones were great when the the alert came in on a Friday afternoon.

They did pay overtime thankfully….

[ShortLegs]

If Aruba are cloud managed... no no no no no

Merakis are cloud managed. That great, until someone manages to "disrupt" the outbound interface. Like, maybe, changing the port from "auto-negotiate" to "1gb". When the other end is a router with an interface that 100mbit.

Because Meraki has not SSH or telnet capability, so even though a seperate network may be unaffected, you cant ssh from another device to the Meraki and undo the well-intended but out-of-scope change.

[Martin an gof]

If Aruba are cloud managed... no no no no no

Have had a couple of Aruba InstantOn on test and intending to buy a few more for a fairly simple network for the simple reason that they are (even now) well under £200 for 24x1G and 4xSFP+. They can be cloud managed, but they don't have to be, but if you want to manage them locally (yes please) there is no CLI, just a web interface. Did firmware updates on the two on test pretty simply, but then the real test doesn't come until something goes wrong, does it?

M.

[Paul Crawford]

In this case Zyxel. Reasonably cheap, Taiwanese (but assembled in China), have a not-too-sucking admin interface, and seem to work quite well most of the time.

As well as preventing any incoming connections, we also firewall off the switches' admin IPs from outgoing connections so they can't phone home (same for our web cameras, IoT tat, and special-task Windows 7 box) which may, or may not, be a factor in the odd glitch.

We have the older/cheaper models that don't have cloud management and are determined it stays that way!

[BenDwire]

Serial port settings ...

... in case the new firmware's changed the port characteristics – which vendors like to slip into firmware updates without telling anyone.

Here's looking at you, pfSense !

[TRT]

Re: Serial port settings ...

Or like Dell who offer a BIOS configuration at build time option for something like £5 a setting. Who wants to add thousands onto the bill when refreshing an entire building full of learning suites? Except I think the deliberately set the parallel ports (this was a while ago) to some random configuration deliberately because half the machines wouldn't run Quark due to license issues. You would think brand new machines would all be the same, but not so. Half of them were in a unidirectional mode so the dongles didn't work. The working mode was EPP which the other half had. It's all a big con!

[Michael H.F. Wilkinson]

"They're over here, on the shelf by the window …"

I can sense a Database Normalisation Warning, and a problem with a windows install coming up

Brilliant episode, once more

[dajames]

Shelf by the window ...

... which happens to be open, affording a bird's eye view into the skip below ...

[Charlie Clark]

Re: Shelf by the window ...

Or carefully timed refuse collection…

Of course, the boss should have used the "emergency" to fill his own boots for essential IT upgrade. How about a coffee machine that also brews beer?

[tezboyes]

Re: Shelf by the window ...

Everyone loves a pinter, and an early morning breakfast stout is the ideal start to a long weekend of "upgrades".

[Yet Another Anonymous coward]

Re: Shelf by the window ...

Went to Google (other better search engines are available) to find the name of that stupid expensive counter-top wifi-enabled automatic programmable beer-making machine.

The first result was a comparison of 12 different models on the market....

[Mark 85]

Re: Shelf by the window ...

... which happens to be open, affording a bird's eye view into the skip below ...

Or the boss's new car.

[karly]

Re: Shelf by the window ...

I loved the last line

[Coastal cutie]

Re: Shelf by the window ...

Damn - you beat me to it - have a beer or beverage of choice, on the principle that it's 5 o'clock somewhere...

[Anonymous Coward]

The old switch and bait.

[Pirate Dave]

So tempting

a BOFH episode on a Thursday? But that screws up the flow of the weekend, and just makes the universe seem a little sideways for a couple of days. So must not read until tomorrow. Oh, so tempting. But no, tomorrow! Tradition! But didn't the BOFH used to come out on Thursdays? Shut it, that's a lie! Tomorrow! Yes, yes, tomorrow.

Is it Friday yet?

[cosymart]

Re: So tempting

Here on the correct side of the pond it's Good Friday tomorrow and a bank holiday, hence the early BOFH :-)

[Yet Another Anonymous coward]

Re: So tempting

It's a bank holiday here in Jesusland as well. Although possibly it's the killing of a middle-eastern man that they approve of

[chivo243]

I loved total control

Over the asset management database... What record? No, you must have been looking at similar unit...

[bombastic bob]

Re: I loved total control

I imagine the asset management database was unrecoverably lost as part of the switch failure...

And ALSO: How do you get "that irritatingly bureaucratic tracking pcrocedure" to go away?

But luckily change-control was one of those management fads easily stifled with a relentless campaign of punctiliously observing the change control procedures for even the tiniest change.

Summary: Eliminate the idiotic bureaucratic "solution" by adhering to it as strictly as possible

[Joe W]

Re: I loved total control

A few moons back there was an episode about getting a business case, a change request opened, doing a risk assessment, etc. Didn't work then either...

I have to be honest, in some cases a change request is useful. You can then harass the other guys about whatever needed to be done - and I found that some groups in our company use it excessively, and then they get the work done I asked of them. I just don't do the other stuff, nor do I use it for my own tasks. Risk? What risk?

[Adrian 4]

Re: I loved total control

>Summary: Eliminate the idiotic bureaucratic "solution" by adhering to it as strictly as possible

'Work to rule' is the most genius tactic in the history of labour relations.

[David 132]

Re: I loved total control

>Summary: Eliminate the idiotic bureaucratic "solution" by adhering to it as strictly as possible

AKA "malicious compliance".

[TRT]

I see the BOFH has...

Invented the long drop espresso.

[Pirate Dave]

"then perform an excruciatingly slow serial upload of the old firmware (which will fail four times out of five, with buffering issues)."

It's no better with tftp if you're trying to run the tftp server on a Win10 machine. Sometimes praying to the electron Gods will help, sometimes not. Sometimes an offering of a spot of blood on the Mode button appeases the angry electron pixies, other times, they can't be subdued by any known means.

But run the tftp server on a Linux box, even an old, outdated, insecure CentOS 6.x box, and Bob's your uncle every time. Apparenlty penguins can eat pixies.

[Hero Protagonist]

Dust from the Big Bang

Couldn’t the entire universe be considered dust from the Big Bang?

Icon because deep question

[bpfh]

No leapfrogging updates

Oh yes. Dell comes to mind here.

I always thought that a bios flash would update everything in its preset predefined "operating system" of the basic input output system with a monolithic overwrite of the previous code... but no. I've got to flash Every. Bloody. Version. from 1.03 through 1.17.

Oh and Dell's driver and patch download server is slow as sh*t... and I'm sitting there invoking the wrath of a thousand demons into the flaccid cock of Michael Dell for forcing me to spend 2 or 3 hours downloading, flashing, rebooting a corporate windows that takes 5 minutes to cold start... and do it 15 odd times.

Companies that force you to do this: may your ear holes turn to arseholes and shit over your shirt arms and may your genitals be infested by the fleas of ten thousand camels.

[Giles C]

Re: No leapfrogging updates

I found that with applications, Cisco stealthwatch (or secure network analytics as it has now been rebranded) version 7.1 to 7.4 (so not even a major verision)

Patch 7.1 to latest service pack

Upgrade to 7.2

Patch 7.2 to latest service pack

Upgrade to 7.3

Patch 7.3 to latest service pack

Upgrade to 7.4

Finally patch 7.4 to latest service pack.

After looking at that lot, decided to rebuilt entire system onto 7.4 and move the configs over manually lot less work and I can redesign it to make it work better at the same time.

Never had that with firmware - although most firmware I work with is Cisco switches or firewalls.

[Anonymous Coward]

Re: No leapfrogging updates

“Companies that force you to do this: may your ear holes turn to arseholes and shit over your shirt arms and may your genitals be infested by the fleas of ten thousand camels“

That’s a bit excessive. Five thousand camels is more than enough.

(In my experience.)

[SuperGeek]

Mental health!

Nice bit of gaslighting at the end! "It says 2022!" "Oh, that's for the beans!" I feel sorry for the Boss. He gets gaslit so much I'm surprised he doesn't spontaneously internally combust!

[Dyspeptic Curmudgeon]

Re: Malicious Compliance

Read The Good Soldier Švejk by Jaroslav Hašek

He follows orders *exactly*. Hilarity ensues, except for the officers, who end up with high blood pressure...