Undebug my heart: Using Cisco's IOS to take down capitalism – accidentally Welcome to another edition of Who, Me? where this week a typo manages to send a hub of rampant capitalism into meltdown. Our story takes us back a few decades and concerns an adventurous time in network support. "Mort" – for that is not his name – was working for a well-known stock exchange, the network of which was running …
So he properly issued an "undebug all" command at the console, and said console decided that no, it was going to do a "debug all".
I don't get how that is possible. There must be some shoddy programming behind that thing.
You've never had the fun of tinkering with IOS have you?
It wouldn't surprise me at all if it didn't bother listening to the commands lovingly mashed into either the SSH command or if they're more locked down via a RS-232 to ethernet adapter and minicom software. Janky does not come close.
It's not quite as rage inducing as handling HP's iLO cli but close.
I have done Cisco training, but thankfully, don't use iOS day to day, and, as a result, don't have admin access to our Cisco switches at work, so couldn't do any damage even if I wanted to. Not that I do.
I remember the days well. Entering a whole load of commands, only to find you have missed one, and as a result, changed a whole load of configuration options you shouldn't have, and quite possibly taken out a large chunk of the network..
The manager not only saved 'Mort', but saw an opportunity to get funding to upgrade the IT to something that could actually handle the high pressure trading environment.
Few things persuade those 'god-like' traders to spend money on menial things like them suddenly not working and shutting down the entire company.
The CLI command buffer is normally quite good at accepting input from 'copy'n'paste' but it does occasionally get side-tracked executing a command and character get dropped.
Generating a crypto key can take 30s on some of the older gear and that could be 30s of characters piling up in the buffer waiting to be processed, and Sod's Law says your mega important command is the one that gets mangled.
Cisco CLI is not immune to losing characters from the input if the system is busy and/or the buffer fills up.
What jumped at me in the story is that with such careful planning of what to do, why on earth not do undebug for the exact debug command you know you have executed earlier just to be safe.
Anyone with any experience in Cisco IOS, especially on a more heavily loaded device, would be acutely aware of its tendency to occasionally miss characters from input as it quite understandably prioritises shuffling packets to what is going on in the CLI.
If you have the luxury of accessing the device in-band as well as out-of-band, it is usually good practice to open a console session and type in "undebug all" (not pressing enter); then executing the debug command in-band through a regular SSH session. Then it's just a matter of alt-tabbing and hitting Enter like a maniac if things go TITSUP*.
*) Terminal Is Totally Slammed Until Poweroff
At a certain large UK ISP, the <Cisco rep of the day> walked up to my desk and said "You don't like Cisco do you?" "No, it's not that. I just don't like things that don't work". The sales reps used to be replaced very regularly.
And a joke from back then: What's the difference between Cisco and Huawei? Well, they both design routers, but Huawei actually make them as well.
But first, a boss who definitely wasn't. I was working for a well-known company in one of their manufacturing research teams. My boss decided we needed to fix a process problem with a few simple changes to the equipment; I argued against it as I saw it would cause new problems - but he was the boss. We ran it and it wrecked the equipment. He called me into his office and tried to tear off a strip - I stood my ground and told him, in no uncertain terms, how my report on the trial would be worded (and copied to his boss - who had initially hired me for another project a couple years earlier, and asked me to move with him to the new one). I should add that our facility was in a security area and you needed clearance to enter; my wife also worked for the company and came by one afternoon to say she would be working late. My boss called her into his office to demand what right she had to be in a secure area - she (not so quietly) pointed out that she had a higher security clearance than he did!
Anyway, I soon moved from there to a new company and a boss who was quite the opposite. He knew his technical skills were limited but knew the company politics (and was on first-name terms with the bosses from head office); he hired people with the required skills and let us get on with our jobs. If anyone decided we'd crossed a line into their territory (as we frequently had to do in order to fulfil our responsibilities, he would step up to take any and all the flak (and usually fire a bigger salvo back). Afterwards, he might call us into his office for a private chat and ask if we were sure we had done the right thing. His overriding policy was that whatever his staff did was done in his name. If we had made a mistake, it was his and he would defend us. It generated loyalty and, on the occasion when one of us thought we might have got something wrong, we would tell him straight away. It also meant that, when he was tackled on something he hadn't been told about, he was confident he was right to defend us to the hilt. I was sorry to leave that job, but the offer to double my income from one of our big customers was too good to turn down - and if I had, my divorce would probably have come about quite a lot sooner!
...boss called her into his office to demand what right she had to be in a secure area - she (not so quietly) pointed out that she had a higher security clearance than he did
It pains me to say this, but the boss was probably right in this instance. Just because you have a certain level of security clearance, does not mean that that you are are authorized to enter a secure area.
Many years ago now, I was working in a hastily put-together ISP NOC that also doubled as a door-opening service for anyone wanting to get onto the technical floor. One of my colleagues went to answer the door one time and refused to let the person in - he didn't have a badge on him and my colleague didn't recognize him. There was a bit of shouting and even profanity involved at the door, but eventually the guy walked away. Later on he sent an email to the whole NOC team commending my colleague for not standing down. Even if he happened to be the CTO.
A long time ago I heard a story, unvouched, about Thomas Watson (I can't remember if it was Sr or Jr) being denied entry to an IBM facility because he wasn't wearing a badge. The enraged entourage wanted to fire the lowly security guard on the spot but Watson said 'No, he's only doing his job'.
In more ancient times, my employer, a medium-size university, had a big ice storm. The power went off and back on. A notoriously dodgy but still un-UPS'd fiber transceiver refused to come back up. This was during the university Christmas break. It was in a telecom room in a basement of one of the buildings. Nobody in my group was ever given keys to it; we were told we needed to be escorted at all times by the campus gendarmerie. So I drove the hour or so to work, entered the police office (staffed 24 hours) and asked to enter the telecom room, while presenting my credentials and stating the reason. The officer on duty deferred to his sergeant, who approached me and told me that there was 0 chance I was getting in that room today. He said he didn't care who I was. I called my immediate supervisor, who called the sergeant, and got a similar line. My supervisor called his supervisor, whom I remember he was an ill-tempered person when subject to administrative BS. I saw the sergeant pick up the phone again and then hold it about 2 feet from his head while the supervisor-of-supervisor screamed at him. A few minutes later the sergeant quietly approached me and asked me "to never do that to him again." I was escorted to the room without incident, and had the offending unit restarted in 30 seconds. Later we put a UPS on it, and never had problems again. I did, however, ask supervisor-of-supervisor what had transpired. This individual was very well-connected politically, and had simply told the recalcitrant sergeant that, unless he let me in, the next person on the phone would be the University President, who was on vacation in the Caribbean at the moment and would be extremely unhappy to be interrupted, and if he still held out, the person after that would be the State Governor, whom the supervisor-of-supervisor was friends with. Needless to say I never had another confrontation like that, but then again we had fixed the root of the problem with the UPS anyway.
I worked for ICL in the 80's as a software support consultant om mainframe systems. My ICL pass would get into any dC with our equipment in with no questions asked. At the time this included several secure sites. They were used to engineers turning up at any time of day or night.
If there were any difficulties we were instructed to ask to speak to the Chief Exec / CIO so that we could inform them that a critical fault was not going to be resolved and there would be a large bill for the abandoned visit. If that didn't work we could contact the account manager 24/7 , they always had personal contact details for the senior customer exec and would call him/ her at 2 am if required.
Explaining this always worked for me. The one time I got pushback was when the security guard locked the car park with my car in it. I asked him which was the best hotel in town and pointed out that the cost would be billed to the customer. needless to say he did then open the car park
I was sent to a secure facility in the west side of Birmingham (UK). The gate guard let my car in, and I parked in the Staff Only section of the car park as I had some heavy equipment to lug in and didn't want to have to carry it across the car park from the Visitors' area, and in the pouring rain as well. When I returned to my car, there was an A4 size notice glued to the right hand side of the windscreen, which would have obstructed the view of the driver. I got in the car and drove out of the automatic exit gate, and as I passed the security hut, I turned on the windscreen wipers. The by now soggy notice was swept off the windscreen and deposited at the security guard's feet, it hadn't inconvenienced me as they hadn't noticed that I was driving a LHD car.
"A4 size notice glued to the right hand side of the windscreen"
I had that many years ago when a colleague let me "borrow" his parking spot which was a LOT closer to the gate whilst he was on holiday and an over zealous security guy pasted it on my screen.
A squirt with WD40 promptly got it off!
Ah yes, those were the days, when engineers were completely trusted to have access anywhere and do whatever they wanted, unsupervised and without checks.
Horrendously insecure, of course, and there were a few 'anomalies' like when it transpired that one engineer working for one company had inserted a cable between his company's server and an adjacent competitor's rack for some ongoing real-time data transfer. Discovered after only a few weeks, I believe.
Nothing to do with the poster 'PM from Hell' above, I'm sure.
Might also explain why I was totally unable to find any HMG accreditation for a specific ICL data centre when they wanted to host a Department's servers and claimed they were already hosting lots of other HMG equipment.
Happy days. Where's the 'Nostalgia ain't what it used to be' icon when you need it?
Another comment on secure areas vs. clearances: At least in the US, there is "need to know". I know folks who had more-than-sufficient clearance, and used certain rooms, but didn't have need to know for other rooms [1]. Shirley said wife didn't have need to know for husband's project.
1. All I officially know was the name of those rooms (nameplates by the doors), but when the official (yet public) [REDACTED] Tech Manual says that if [REDACTED] is damaged you must not touch or look at it and report it upward, and a certain nameplate references said item, you can surely put 2 + 2 and get 5, with the extra bit being the part that could land you in jail if you drop too many hints in a public forum. Everyone know *about* it; only those allowed in the door knew the juicy details.
What strikes me always about the "need to know" environments is that people seem to be too attached to wanting to know secrets.
I'm the opposite - I am aggressively desinterested in secrets, because I have handled them for too many years. The less you know and the less data you have, the better. What you don't have cannot be lost or accidentally leaked or shared with parties who should not have that data.
If you're dealing with secrets, a multi-layered defense should be second nature.
AC: "people seem to be too attached to wanting to know secrets"
Two neat stories. in Clifford Stoll's 'The Cuckoo's Egg'*, he recounts visiting a secure US Government site and when his host left the office, he played with the classification stamps left on the desk by stamping a blank sheet of paper with "SECRET", "TOP SECRET", "US EYES ONLY", you know harmless stuff. Anyway on being searched as he left, the guard was very unhappy and summoned his contact for an 'explanation'.
Another story, which might also be true, was about the Colditz escapers in WW2. One had a wife who was very intelligent and good at puzzles. She became convinced that her husband's letters were coded and went along to Military Intelligence to talk to one of the high ups. In the TV episode he briefly leaves his office, leaving a folder marked "SECRET" on his desk. Being inquisitive, she opens it to see a single piece of paper covered with the word "SECRET" in various orientations.
*Excellent book, if you haven't read it, you probably should.
Best thing you could do before running them is issue the commands to ensure it doesn't dump to console (and to logs only), but even then some commands would cause spikes in CPU usage and make the CLI sluggish.
"debug spanning-tree all" is another one you don't want logging to console in a switch that's part of a live/prod environment. Every STP broadcast, event, topology change, uplink change or error thrown onto the screen, and getting it to turn off once it's running is almost as bad...
Also a minefield - making sure your colleagues are aware of what debug output looks like and what it means. A former junior associate of mine was running DHCP debug on a pair of campus distribution switches (troubleshooting an IP address allocation issue), another engineer saw it and assumed it was a problem - he responded by rebooting one of them. Thankfully quicker hands managed to stop him from rebooting the other at the same time (which would have taken out the entire site whilst the switches came back up...)
This post has been deleted by its author
My manager has our back... And back long ago, during a maintenance window, upgrades and patches went so well, I reminded my then manager, also a lovable manager that there was another niggly bug that we wanted to fix, he fired up a utility to fix bug x, and due to a poorly worded dialog box, he proceeded to take the RAID system down, wouldn't come up again, and quickly found out the backups were not up to the task due to the best effort job done with the woefully lacking hardware. In a month we had shiny new Dell PEs. Nobody was shown the door, but I did get some OT out of the exercise.
Hmm
Two network engineers sitting in an office, one (not me) types in a debug command that was missing a parameter so it started overloading the console.
I glanced over at him as things started slowing down - his glanced towards the door a quick nod and we both got up with laptops and quietly walked to the comms room. Where we found a switch stuck in a debug loop
Rebooted switch and as it was spooling up management burst into the comms room
There was a panic when the network went slow followed by a bigger panic when both network engineers had vanished.
We both stayed working their for another 7 or so years - management put it down to human error…
To me, that sounds like a great opportunity to use the "We noticed that the networks was slowing down, so we worked together to isolate it to this one switch and it's already coming back now."
That's a Network Career Achievement Unlock right there: "The Pretender" - Managing to receive praise for fixing a problem that you caused yourself.
Reminds me of a former colleague in 1st line support that would occasionally remotely run pssuspend on outlook.exe on the CEO's laptop, be first to answer the phone when it rang, then would magically fix the issue. The CEO loved how responsive and attentive he was, and always managed to fix the issue within a few minutes,
I'd like to think I am that kind of boss. I've never thrown anyone under the bus. If something has gone wrong, I endeavour to find out what and how etc and even if it was something silly/avoidable, we use it as a learning experience.
I don't try to pull the wool over the eyes of customers - I've forestalled more than a few conflicts by saying "yep... we messed that up, but here's what happened, what we did to fix it and what we've put in place to ensure it never happens again". Luckily if the conversations have turned to "compensation" then I have the "above my pay grade mate - talk to x" to fall back on, though it's rarely actually got that far.
Generally I have have very high standards but if you are working from me I am pretty easy going: I don't care where you work from or what hours you keep (within reason and if you are needed for meetings/on site etc then I expect that much). I do expect deliverables to be delivered when agreed and if there are problems brewing then I want to hear them from you, not when I get a customer calling me to complain/rant. Basically don't take the piss and we will get along fine.
I once basically 'threw my manager' under the proverbial bus. Well, actually all I did was point out that he'd basically jumped:
It was not dark, stormy or night when I 'took over' a project for a financial institution concerning PKI infrastructure. The customer was wondering where their pilot scheme was and when it would be delivered. The senior manager was not aware we were providing one.
I finally tracked down copy of the contract (on the manager's laptop) and read it. In the section headed "Deliverables" there was a bullet list of items we were delivering (about 5 or six items) which we had or were in the process of delivering. After the bullet list, was a paragraph mentioning a 'pilot secure e-mail system'.
Now, me being me, I could see how the customer (who also had a copy of the contract) might just interpret the fact that a "pilot secure e-mail system" being mentioned in the section headed "Deliverables" could conclude that we were contracted to deliver a "pilot secure e-mail system".
The senior manger had interpreted it as just being the bulleted list items, and not contractually bound to deliver the 'pilot secure e-mail system'. That would be on an 'if we feel like it' basis.
There was a meeting, which I was 'theoretically' chairing with the client and said manager about this. My manger was literally shaking, as the client had us 'by the small round things'.
It cost us a bit to provide the pilot, but the techies really enjoyed it. What we didn't enjoy was using a technological start-up's PKI, which was cryptographically sound but so craply implemented that, and here Register readers of a sensitive disposition regards security should take a deep breath, you could never change the System Admin password or account name after setup. ('What never?" - "No, NEVER." as the song goes.*)
*Gilbert and Sullivan, 'HMS Pinafore', the head of the Queen's Navy Song.
A component falling on its rear end is why every 'shop' i've worked in has 2 of everything in each dc.
type debug all & the device falls on its rear, the other device would assume control & by the time an issue is noted everything is still working albeit with a device that is isolated & no longer in the active path. a benefit of that isolation is that it processes less traffic so the cli should be more responsive so an 'undebug all' should then work.
its possible to have splt brain where a device under stress doesn't relinquish control but as soon as its power source is removed you should be good, just remember to amend the metric on what was the secondary kit to ensure it remains primary when the borked system is back online.
HSRP, VRRP & others etc exist to ensure overall systems can remain available in event of an issue. If the risk is loosing millions then spending a smaller sum to ensure availability is well worth it. Many people don't understand that. Its also why you buy diverse MPLS private circuits and not just ADSL internet connections.
Had a manager when I worked for a bank that would throw you under a multiple pile up.
Anything that went well, he crowed from the rooftops that it was all because of him, anything wrong, he threw you under the bus by name.
It's how I ended up with the head of european settlements screaming screaming at me down the phone.
Oddly I was moved out of his team, one Friday afternoon at 5pm and told I was working for a different team at 9am on Monday.
He was the proverbial Richard Cranium.
But then, I've had managers that took all the flak and wouldn't identify the person that made a mistake.
The late Sir Terry Pratchett's book "Mort" is one of my favourites.*
"Death comes to us all.
When HE came to Mort, HE offered him a job..."
(Heaven knows what Sir Terry would have made of the current situation with Brexit, Covid-19 and such.)
*My other favourite is "Small Gods", essential reading for theologists.
Who would become an incandescent a55hat at the weekly production meeting. I suppose it didn't help that I wore a tie (mandatory for that meeting) festooned with cartoon characters.
One fine day, he came storming over to my desk screaming that "Your tests aren't working". I was a test engineer at said company designing hardware and software that ran on both the tester and the target.
So I checked out the failure results and thought simply that this can't be right. Every single piece of the production run had the same fault but I knew precisely what the code did and there would have to be a major tester hardware fault for it to be the test equipment, but the results were coming from two independent test stations.
We had our manufacturing done in a facility about 100 miles away (the USA is a big place…) so I drove over there and collected a few samples that were failing the test and duly brought them back to the office.
One of the 'associate engineers' (a real snotty type who looked down on those who had become engineers without a degree) said he would test one on his setup and declared the circuit just fine (for those who wish to know, the circuit was to fire the escrow relay in a payphone - remember those?).
Well, I am a bit of a stickler for a real analysis, so I hooked the unit up to an oscilloscope. Normal operation had a circuit charge up to 120V and then fire. This unit was charging up but then the circuit would break down and only about 40V was actually getting to the relay.
The actual root cause was that the contract manufacturer had set the reflow oven temperature too high and it was delaminating some capacitors that were there to store the charge until dumped into the relay (which was clearly visible when I took the part off the board). That means their breakdown voltage was not the rated 250V but about 40V.
That had two very satisfying outcomes.
1. My test were working properly as they had detected the fault. The manager in question was given a few words by the CEO.
2. The rather snotty associate engineer was shown up as being sloppy in the engineering sense.
I was a bit lucky there in some ways as the manager was not my direct manager (I worked for the CEO).
Debug all, seems like a good idea, but the "sir, yes sir!" response of the router is dead on.
Anytime IOS is mentioned, I have to discuss my favorite IOS command: "rel in 5" (reload in 5 minutes). Saved my bacon on more than one occasion, especially when working on a remote router.
Issue the command, do the commands that will possibly drop the route that allows you to access the router, then see if all is well. If you borked a comnand, you sweat out a brief period of "suboptimal route selection resulting in degraded performance". If things are good, do a "rel can" (cancel the reload). At that point, if you're me, you log off and go home happy (until the next time the power cycles and you realize you forgot to do a "wri mem").
Anon, hoping that my old bosses don't recognize my router excuse.
Before any change I dump the config into notepad ++
Then write mem / copy run start (depending on age &/or flavour)
Then if I could loose connection do a reload in 10
Juniper land is better with its commit confirmed, just a config roll back, no waiting for the thing to boot.
I've worked in financial institutions for around 20 years, including 15 in stock exchanges. Cisco gear would not be found at the trading platform, the latency is too high. You'd find arista gear mainly.
You might find cisco on the edge or in the private mpls networks that connect the customers to the trading platform in a cursory role, as that gear is usually juniper.
... that takes me back to the mid-90's. I once used three Cisco 2503i routers to link three sites of a large engineering company via 64K ISDN lines. Novell Netware on each site. IPX isn't routable so this was my introduction to the wonderful world of TCP/IP routing. Took the best part of 6 weeks to get it working, including a visit from a Cisco engineer. Turns out they'd never tried a 3-way link before, but a few IOS patches and tweaks later and they got the job done. The resultant phone bill was also very impressive!
Back in the early 2000's I worked primarily with Cisco switches, and the most common problem saw were duplex mismatches. And this does cause packet loss and bad throughput problems.
Thing was, at that time the autoneg specifications were pretty ill-defined, and not always implemented the same between vendors and versions. I believe especially Nortel switches had issues when connecting to Cisco.
Cisco also had one crap IOS version, where even if you configured your interfaces to full-duplex, the underlying software with still revert back to autonegotiate.
That one cost me a trip to Paris, as people were puzzled for a day or two on what was going on with their network traffic.
By then I already only had to take a quick glance at the blinkenlights on the front. Green going flashy amber can only happen in collisions.... IOS upgrade fixed that one.
Post 2005 the standards got better defined and followed, especially with GigE standards, but Cisco was still forced to put in IOS commands for manually configuring speed and duplex on GigE links. They did so reluctantly, as it's a really bad idea. Autoneg does more than just speed and duplex in GigE...
If is Citrix from somewhat 2017, where I experienced this oddity first, up to the newest current version it breaks the universal law of causality.
1. move the mouse
2. click
3. move the mouse
4. click
If done too fast Citrix executes:
1. click
2. move the mouse
3. click
4. move the mouse
And don't get me started including the keyboard in this...
Maybe older Shitrix installations have the same issue, but I have a large gap from from about 2000 (Metaframe 1.8 NT 4.0 based) to 2017 where I didn't need to use or administer it.
I had the privilege of having such bosses several times. The first fell in disgrace as the holy pope of manglement wanted an overtime sacrifice at all costs and our team leader refused. But we HAD to make overtime, the machine had a fixed delivery date, the customer was already enraged by our "communications" which were - lets say - "reality incompatible"... Long story short, manglement wanted alibi overtime, team leader explained that you cannot program and test segmented profibus without the dp/dp couplers and the terminal resistors which at that time had yet to be delivered (ordered too late, once in a while not even Siemens is to blame).
Management was so enraged by his firm stand against useless hours that he got transferred to a different team in a totally different field, then made redundant with another transfer to a fictitional post that was then cut.
Profibus again, this time with safety modules (PLC I/O hardware in PITA mode). Team leader, programmer and engineer in one person, fired by manglement because he refused to exchange safety hardware against cheaper components, again a case of ordered too late but this time with an additional flavour of incompatible hardware due to manglement infections with marketing mould. Too bad a safety I/O module that already does his own short circuit detection sees an issue in a safety sensor that ALSO does his own version of a short circuit detection. The original sensors chosen by our engineer would not have interfered but those chosen by management, probably out of the "best buddy ever" line of products, simply were not compatible at all.
Replacing the I/O modules is definitely not an option if you have to deliver performance level d+.
Words lead to shout which lead to firing the engineer and team leader in the last month before delivery and also prohibiting him from entering the company in the mandatory 4 weeks termination period. Up to the last second, he stood his ground and made sure that whatever axe management wanted to grind, whatever example they wanted to make, it would be his head on the proverbial platter. What fun it was for our literally beheaded team to try to understand the whole machine in a limited amount of time while also trying to finish it and wrestle in last minute wishes from the customer.
Currently i have a boss who actively refuses to ever make a decision. No worries here, tools like that can be "handled", praise regular readings in the church of BOFH. As long as you make and keep thorough documentation of what was decided or at least ordered when by whom...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
