Who, me? Techie with three months' experience was sent in as the Cisco expert and proceeded to blow everything up

WHO, ME? “Where to begin?” is a marvelous question. To get you going at the start of the working week, The Register thinks the answer is a new instalment of “Who, Me?” the column that shares your admissions of error. This week, meet a reader we’ll Regomize as “Martin” who told us about his very first job in tech. “I had just …

[Korev]

That downtime must have cost a packet

[Anonymous Coward]

This is Muddy

From TFA: "AS/400 at the time had its connections from the hosts to the mainframe through SNA over TCP/IP emulation".

Why would you need to emulate TCP/IP in this setup?

Are we talking about connections to an unspecified mainframe from AS/400 boxes?

The more-likely seeming setup would be PCs with sessions to (an) AS/400 host(s).

[Giles C]

Re: This is Muddy

The sna emulation was to allow the terminal sessions to talk over tcp to an sna server (the as400). Sna was a flaky protocol, we used to have replication traffic between as400 boxes on either side of a firewall. (This was 20 years ago so some of the following may be wrong due to time passed).

SNA opened a connection and kept using it, if something interrupted the connection the process had to be restarted, and you had could have connections up for months.

When I first started at budget insurance there they had windows 3.1 machines using SNA and IPX stacks and a terminal emulator called rumba (I think) running on a token ring network. Most of the staff used terminals not PCs

But yes they moved over slowly to first having tcp as the native stack (no dhcp for some reason which escapes me), running sna in the background before moving to tcp based comms, at which int I upset then as they ran it in telnet and I told the admns what their passwords were (then they realised that ssh might be useful)

Fortunately I don’t deal with those sort of boxes anymore but I think sna should be mostly gone now.

[BartyFartsLast]

Re: This is Muddy

"Fortunately I don’t deal with those sort of boxes anymore but I think sna should be mostly gone now."

I bloody hope so. Used to cause nightmares at a national contract because their entire, time sensitive dairy products distribution business relied on badly installed, knackered old patched up token ring which dropped for fun at some sites

[Anonymous Coward]

Re: This is Muddy

This will be somewhat vague because I might just still work for the firm. I once had the pleasure of using some software that was very very new, in fact unbeknownst to us we were the launch customers. So after we’d populated some of our data into the system it was put on internal live status. This worked but it was not perfect and we in the team had some doubts but the Head of Department was confident. Then one Friday afternoon all hell broke loose, it stopped working, totally borked and there was nothing we could do. So we called the supplier who said they would send someone round to look at it on Monday. HoD says no they’re coming today not Monday and put the phone down. Two people then turned up and looked in bewildered horror at the error messages which made no sense to us and even less sense to them.

HoD stands around the two as they shook their heads and typed commands which were either ineffective or generated yet another cryptic error message. I was volunteered to be the babysitter for them and was told I shouldn’t stay too late. So at 19:30 I enquired as to how they were getting on and then told them they’d have to come back after the weekened. On Monday afternoon with no progress from out dynamic duo, the HoD takes me and some other team member into a conference room and is visibly annoyed, on the verge of a rant. He tells us that this is ridiculous they cannot have been atthis for over 8hrs and nave nothing to show for it. He then said that this next bit was confidential, and that he is going to exercise our right to terminate the contract. He is not going to do so immediately, he’ll wait until they fix it, then he’ll break the bad news. He wanted to see how long it was before it was fixed, as this was embarrassing. I was told to make contact with a rival vendor and get them up to speed, and that we’d like a demonstration asap. Friday lunchtime the duo (or rather one of their global teams) had found the problem and a patch was being applied, they saw the HoD with me in attendance too. So the dynamic duo explained that a combination of 2 characters in a database entry had brought the system down. One was an unexpected “€” in one field and an “@“ in another, they hadn’t thought those might be used in a business name.

The bad news about the break clause is given and they look stunned, suddenly they pointed out that they had now found the problem, and it had been fixed. Yep said the boss and what happens when another unknown rogue character kills it again? Points out it took them 7 days to find and fix the damage this hand grenade had caused, so no there is no going back, not now and we will inform them when we have finished using the software. Meanwhile the other vendor had been in to the building and done a successful demonstration which we liked and got them out again unseen. We migrated everything over had it running in parallel with the discredited software. Much more satisfactory and have been using them for years since. It was later leaked to us that they were going to make a big thing about us being the first customer to use this abomination of a system. They had designed web pages with our logo, dummy quotes saying how satisfied we were with system and the vendor etc. - Oops. We had also checked whilst they were there and found people who we were dealing with providing the “tech support” in person were not very technical. They were just dealing with various other people around the world who were actually doing the work. One of them was listed on their website as marketing/support, the other didn’t even appear.

[Dabooka]

Re: This is Muddy

Not payroll / HR by any chance?

[Androgynous Cow Herd]

Re: This is Muddy

"Then one Friday afternoon all hell broke loose"

I stopped doing any config work on Fridays at all.

And Thurdsays are questionable anymore.

You're just asking for trouble (and a lost weekend)

[JimboSmith]

Re: This is Muddy

We had a situation back when you received a software patch via Royal Mail not the Internet, where a program suddenly refused to print. This was kind of really needed so not a good thing at all. Fortunately there was a bloke from the vendor on site doing training who was summoned. He said yes he had applied a patch to fix something else as we hadn’t done it ourselves, and that may have done it. My boss who didn’t mince his words said and I’m slightly paraphrasing here, without raising his already low gravelly voice, like an East End gangster from a film:

“Nobody gave you permission to install anything, and you’re on very thin ice. If you don’t roll that patch back now, I will have security here in minutes to escort you from the premises, permanently. You will not be allowed to collect your stuff, it will be handed to you outside the building or sent on to your company for which you will be billed.”

Low and behold it was rolled back very quickly, he apologised profusely and everyone made it to pub after work with no one having to stay behind to work.

[Anonymous Coward]

Re: This is Muddy

SNA has hung around in in situations where there is confidence in end-to-end security, hence some confidence that you don't get man-in-the-middle attacks, and hence the originator's authentication can be propagated without sending passwords for authentication. Certificates are now providing that reliability - at least until quantum computers breaks their encryption.

Db2 for z/OS has supported SNA until now, but IBM has announced that the next version (** future directions and all that stuff **) won't - TCP/IP only.

[Anonymous Coward]

Re: This is Muddy

@ Giles C:

Your explanation makes more sense than the quote from TFA ("AS/400 at the time had its connections from the hosts to the mainframe through SNA over TCP/IP emulation").

Hmm. Perhaps TFA would have been clearer were the critical phrase written as, "SNA-over-TCP/IP emulation".

I had worked at place running SNA-over-TCP/IP, and also over Token Ring, the latter via Rabbit gateways to our mainframe.

[ComputerSays_noAbsolutelyNo]

Brittle

If a single packet loss causes this, well, they had it coming in some way or an other.

[Giles C]

Re: Brittle

A known problem with sna unfortunately it was extremely fault intolerant

[BartyFartsLast]

Re: Brittle

It was bad if your environment or hardware was flaky, but in the places it was used the networks tended to be very well installed and maintained.

Until they weren't.

And, to paraphrase, no well planned network survives first contact with the users.

[fibrefool]

Re: Brittle

once upon a time by misconfiguring a Cisco router I discovered a ping-of-death that could make my employer's mainframe's FEP (Front End Processor) crash. I can neither confirm nor deny that, having been surprised to see the mainframe disappear when I tested the config change, I waited until it was back before attempting to re-connect to confirm the behaviour was reproducible. happy days...

[Anonymous Coward]

I started working for a large multi-national in London that used Peoplesoft. In my previous role I had access to run queries but they were always one day out of date. As you will all agree (can't see anyone not doing) that was sensible because it wasn't the prod environment. The other point was I didn't have a schema and had to work out all the fields for myself which in itself isn't a problem because I can use trial and error plus common sense/checking it against the accounts in the system and it didn't take me that long to work out where everything is.

After I got a rough handle on what I wanted and how to get it I set off joining the tables and setting it up for a nice big query so I could take the data out and manipulate it elsewhere because there is no way I'm building loads of queries in Peoplesoft when I can do it elsewhere in a fifth of the time it takes to run it from there. Sure, the big query will take a little while but once it's done it's done.

I'm guessing you know where I'm going with this. They didn't have a dev or a staging environment just prod or they only had access to run reports in prod. I never actually found exactly what their setup was as I only ever had access to prod. Off my report starts running and I go off for coffee. The office was shared with various back office functions so I'm there getting my brew and I'm starting to hear people complain about the slowness of the system. This was actually a common occurrence (wonder why?). I quickly realised that was me, whoops. Then the phones started going from the various branches. Whoops... I cancelled the report as quick as I could. A lesson was learnt that day. Never assume you aren't in prod even though that's the dumbest shit I've ever seen. I got my data but had to limit it to much smaller batches. Nobody knew it was me either as I'd just started and I think the IT department were a little incompetent to be fair or probably didn't care as it fixed itself.

It was actually one of my most frustrating roles because after a few months I spotted errors with reports everywhere even the one that went to the board to make decisions which I took over running. I pointed this out and offered to fix it so it balanced correctly but was told if it's always been wrong just leave it that way. I was just a contractor so did as I was told but to this day that still bugs me.

[Bebu sa Ware]

"if it's always been wrong just leave it that way"

If it's broken, don't fix it.

Surprisingly common mindset. Could be the motto of governments and bureaucracies everywhere.

I can understand this position when a work flow has a faulty component whose output is corrected later in the process. (Of course then arguably two+ broken components.)

Uncoordinated and unsupervised changes could wreak havoc. Effective change management is still virgin territory for many organisations.

Spanning tree in all its various forms has been responsible for a good many footshots.

Networking hardware - switches in particular - has grown incredibly more complicated over the last four decades to the point that a degree of expertise that is absent from your average systems' chap, is required to configure switches correctly to operate in anything other than a trivial environment. I bail at vlans with spanning tree on Cisco kit.

[Giles C]

Re: "if it's always been wrong just leave it that way"

Judging by the technologies in use I would put the time frame at the 1990s at which point basic 802.1d spanning tree would have been the most common failure.

What probably happened was that two devices had the same cost (32768 was a common default) and when the new switch was installed it had a lower MAC address than the existing root bridge.

I am not sure if Cisco’s MAC address ids where/are lower than nortel.

This would cause a recalculate which in 802.1d was up to 50 seconds.

Nowadays with rapid-pvst we can reconverge in a second or two and also have hopefully planned the root bridge costs properly.

I saw this happen at budget insurance (again) where whilst the spanning tree moved things around we could watch all the terminals go offline running down the office bank after bank of desks.

[Anonymous Coward]

Re: "if it's always been wrong just leave it that way"

I had a play around with Cisco kit a couple of years back just for fun doing some infosec training which I never actually got to use. I must say it's not the most intuitive of systems. Once you work out it's logic it's fine. I stayed mostly in the cli though because I find that much easier no matter what I'm using. I tell it to do something and it does it. These days at home I have a QNap router with all the bells and corporate whistles such as vlans and whatnot because it was the best router option for 10gb ports when I got it.

[Evil Auditor]

...after a few months I spotted errors with reports everywhere even the one that went to the board...

Did you ever flip through those reports chronologically and discover the PACMAN-like animation in the data? There might have been a PFY at play.

[Anonymous Coward]

That's a good one.

In the world of analysis I had one rule a bit like fight club. You don't talk about this rule but you do teach it to junior analysts. Never and I mean absolutely never do you report an exact number on a report. Even if it is exactly 10,000,000 or whatever exactly rounded number it is. It raises far too many questions and you will have to explain this number in great detail to the higher ups. It's just not worth it. Adjust the report in a way that can still be explained through should it be required but for the love of dog don't report an exactly rounded number unless you are reporting something that lends itself to exactly rounded numbers.

[Pascal Monett]

A bit hard to provide a financial report without numbers.

And you should always report exact numbers (unless officially required).

Let the higher-ups do the rounding.

[Scott 26]

bit like the survey of Mt Everest. From Wiki:

Peak XV (measured in feet) was calculated to be exactly 29,000 ft (8,839.2 m) high, but was publicly declared to be 29,002 ft (8,839.8 m) in order to avoid the impression that an exact height of 29000 ft was nothing more than a rounded estimate.[30] Waugh is sometimes playfully credited with being "the first person to put two feet on top of Mount Everest".[31]

[Anonymous Coward]

I'm in pharma. You report the data you have. Rounding to a previously-established, documented number of decimal places is appropriate, but altering it is an ABSOLUTE NO, NOT UNDER ANY CIRCUMSTANCES.

[disgruntled yank]

Peoplesoft

@AC

Peoplesoft had its quirks. On the other hand, it had meaningful names: PS_PERSON, PS_PAYCHECK, etc., and you knew what you might find in PS_PERSON.FIRST_NAMES. I thought fondly of this when I had to do some work with Great Plains (or Dynamics): every table and every column named as if the developers were paying serious $$ of their own money for every character used in a name. I kept a copy of a spreadsheet with Great Plains table names.

And I can't imagine running Peoplesoft with just a production instance. Somebody liked to live dangerously.

[Throatwarbler Mangrove]

At $ork[-2], I got a call late at night about a bunch of services falling over. The issue was rapidly isolated to a DNS outage, specifically the DNS for our parent corporation, a very large financial institution. Over time, it came to light that:

* There was no escalation path for the DNS outage, so the manager of the DNS team had to woken up by a phone call from my boss.

* There was a single point of failure in the form of the master DNS server.

* The root cause of the failure was a failed switch port. That's right, the SPOF only had a single network interface.

* There had been a change ticket filed to set up redundant network connections for the master DNS server, but it had been repeatedly deferred or cancelled due to it being considered too high-risk to implement.

I'm not sure what the financial cost of that outage was, but it certainly exceeded the cost of a second switch port!

[I could be a dog really]

I'm starting to hear people complain about the slowness of the system

Ah, that takes me back.

A few hats ago, we had a system that ran on SCO OpenServer 5 - yes, back when they did good tech, and before self destruction. Unfortunately, SCO OS had a hard limit of 560,000kbyte for disk cache - statically configured in the kernel. Yes I tried, set it to 560,001k and the kernel wouldn't boot. We could put more memory in the system, but we'd still have that 560,000k limit on disk cache.

The system we used had a reporting tool that was "interesting". You'd set off building a report and it would run in a flash - using indexes. But make a small change, and it would decide indexes were for wimps and do a full join and filter across the tables. No, remember that limited disk cache ? Well our sales order detail table grew to 1Gbyte - so when the reporting tool decided to do a full join with it, it a) made the disks very busy, and b) flushed everything else from the cache. Result, 99+% WIO, 0%user, phones start ringing as everyone's terminals freeze as the program gets stuck in a queue waiting for information from the disks.

We had one report that could only be run at the weekend, and took 40 hours to run. I rewrote it in Informix (being careful with indexes), and it ran in under 2 minutes without bringing the system to a halt.

[Anonymous Coward]

SCO never, ever had good tech. As your very own tale of woe proves.

[I could be a dog really]

I disagree.

For the time, it was fairly good tech. It was very reliable, fairly affordable, Unix - and you could have a free personal licence IIRC. At a time when other vendors had "interesting" pricing, and Linux was still in it's infancy. The woes we had weren't anything to do with SCO, but to do with the software we ran. OK, being able to throw more memory in and use it as disk cache would have gone a good way, but by the standards of the day, that wasn't really a widespread limitation.

As it happens, we did have a plan to update to a later version of the software suite, and that had the option to run on Linux which is where we'd have gone (and thrown in a few bucket of extra RAM to get a pseudo "database in RAM" size disk cache) - but that didn't happen while I still worked there.

[IGotOut]

I'm impressed...

...given that Cisco and Nortel switches were completely different beasts (I much preferred Nortel kit), I'm impressed he managed to get anything working. But then again I found Nortel more more logical and user friendly, so an easier journey than Nortel to Cisco.

[trevorde]

Worked at a very small software consultancy who were asked to help out a company with their custom systems. We were introduced to their IT team who consisted of two 'apprentices', hopelessly out of their depth, and no supervisor in sight. They had some problem with one of their Java based apps and asked us to 'have a quick look at it'. Even though my expertise is C#/C++, I managed to fumble my way through and fix it. To the bewildered duo, it must've looked like god level wizardry. We decided to not push our luck and declined to take it on.

[SnailFerrous]

The expert

If everyone else at the company knows nothing about a subject area and you know next to nothing, then you are the company expert. Experienced that too often.

[Excused Boots]

Re: The expert

The definition of an expert or consultant is anyone who knows even slightly more about a subject than you do.

[just4this]

Re: The expert

Experts know, consultants claim to.

[Richard Pennington 1]

Re: The expert

I'm retired now, but more than once I have become a "subject matter expert" by being the only one (on either side of the table) to have read the documentation.

[James Wilson]

Re: The expert

In the land of the blind the one-eyed man is king

[David Hicklin]

Re: The expert

Which is how I got into IT in the first place

Work had some new PC's installed for the first time (Compaq 286/386, this is early 1990's) along with a Novell 2.x network and the lovely BNC based co-ax daisy chain networking. Nobody knew anything about Networking, and very little about PC's bar a couple of early IBM 8086 machines so they had an outside contractor/consultants in to install and configure it, after which they left....

Now I was very interested in how this stuff worked and with management asking "what do we do now" I found myself with a secondary job of looking after it all...and unless it was a simple or urgent task it got left for the end of the day on overtime.

And that is how a 30 year career in IT started.

[Terry 6]

Re: The expert

I became a the tech support and trainer for my own and a range of other educational services and local schools because (in the 80s) I had a decent understanding of computery stuff. A combination of high school coding classes and then in adult life, doing stuff with an Acorn Electron. I joined another teacher in running central trainning for school staff and took over from him when he got promoted to a headship.

By the standards of what was neeed a few years later I knew nothing. But in that period I was a true guru. For a good few years I kept up to speed, so my guru status just about survived. It lasted well into the period when school IT started to get professional. But it couldn't last. I did have my proper teaching job to do as well.

[Anonymous Coward]

Re: The expert

Obligatory reference: "The Expert: Seven Red Lines"

https://m.youtube.com/results?sp=mAEA&search_query=the+expert+short+comedy+sketch

[pts1967]

I'm old enough now that this warms my heart. There's actually nothing better for the new guy than to let him absolutely tombstone a whole department and then showing him you've got his back and getting things all sorted out. With the right crew, it quickly builds excellent team cohesion, well done!

[FeRDNYC]

"How come no one's yelling at me?"

"You know how bad you screwed up, right?"

"Yes."

"Is there anyone who feels worse than you do?"

"No."

"Then I doubt it'll ever happen again."

– Aaron Sorkin's "The Newsroom" (2012)

[Groo The Wanderer - A Canuck]

The most incompetent individual I ever encountered was in the late '80s, almost early '90s. They claimed to have a degree from a small US University in the midwest and to have been employed by a top ten firm in the IT sector doing "C" programming. When we found their code littered with a "#include <stdio.h>" before each and every single stdio function call, we realized we'd been scammed... an investigation followed, they were allowed to leach out the remaining term of their contract doing mindless drudge work that was never merged to the RCS repositories of the day, and everyone was thankful their initial term had only been three months and that we caught their ineptitude before any commits could happen.

[Caver_Dave]

Standard MO when working with Indian subcontractors

When manglement talk to the prospective sub-contractors they talk to the experts.

When we see the work produced, it was performed by the juniors!

[l0pht]

I recall a time when I was in a large Bank and our MSP sent an alleged "certified" network specialist to reconfigure the core network gear with lots of DMZs' and new VLANs.

The "specialist" took down the network, overwriting the existing configurations without a backup and then couldn't work out how to reverse the changes; the result being the Bank being offline for two days.

Needless to say, he was never allowed back into the building, and we ended up sending the MSP the bill for revenue losses.

[David Hicklin]

So he was "certified" then, just not in the way you wanted !

[Anonymous Coward]

That sounds similar to one of the networking projects I was involved on. The "consultancy" (whose name *might* sound something like "Daprimini") delayed implementation because their expert was still doing the basic installation course.