Junior sysadmin’s first lines of code set off alarms. His next lot crashed the company Welcome again to Who, Me? It's the Monday morning column in which readers of The Register admit to making big mistakes and somehow swerving the consequences. This week, meet a reader we'll Regomize as "Logan" who told us he acquired a psychology degree that somehow led to a role as an IT generalist. A few years into that gig …
I've had a few managers who took the blame for the department over the years and several that tried to push the blame further down the tree...
I've tried to become one of the former over the years.
When I was working on a job for a finance department at a customer, we had to recalculate Essbase hypercubes twice a days... Process:
1. Backup bottom row data
2. Delete all data from the hypercube
3. Load bottom row data from backup
4. Recalculate
The delete data and reload was standard OP for Essbase back then. A clean calculation from an empty cube took around an hour, a re-calculation of a full cube took around 8 hours...
I went through the process a couple of times, then the next time through, I somehow got distracted and started at 2... And, oh, no current data to load! I asked the junior dev who had been responsible for the database what we do now? He said, load the previous backup and blame it on the users. I was horrified, so I went to the Finance Director and my stomach was really rebelling, but I told him exactly what had happened, that the database had a transaction log and we would load the previous backup, replay the transaction log and the users should double check the data, once the cube had re-calculated, and that it would take around 2 hours, instead of 1 hour.
I loaded the backup, played the transaction log and recalculated. It turned out it worked and we lost a total of 2 transactions.
My honesty gained me the trust of the finance director and we had a very good working relaitonship after that.
I once had a manager who had risen through the ranks of the engineering department, didn't hold any formal qualifications beyond school, but knew the company operations well, was politically astute, and on first-name terms with the corporate VPs across the pond (we were the UK arm of a very large US multinational service company - one that does business with other multinationals and also direct with governments). My manager admitted, privately to his senior team, that he didn't know the technicalities of our work but that he trusted each of us to do what was necessary for the interests of the company. Doing that didn't always win us friends (especially when we had to stop production or block contracts) but, whenever another manager tried to take any of us to task, our manager stepped in. If they had an issue they were to take it up with him for, as far as anyone outside our department was concerned, everything we did was with his full authority. He took a lot of flak but always came through unscathed as, if logical argument didn't work, he probably knew where bodies were buried (and, of course, had the ear of staff several levels above anyone local). He might chew us out afterwards, if we could have been more tactful or delicate in doing our job, but only to ensure we would know a better route in future - he never countermanded anything we did (unless we agreed that it was a marginal call and there was merit in doing so).
Very rare.
So the senior was an arsehole then.
Reminds me of the arse that I worked with who was sat in the office reading the paper. That's fine, do what you want but I was working and then needed a tool he had. So I asked for it, he continued to ignore me and read the paper. I went to get it out the draw and he stopped me (he really deserve a punch in the face for that). I argued with him, telling him he was doing fuck all and I'm trying to work. His phone then rings, he pretends its the director of IT because he was a cunt but so was the director of IT. I could clearly hear it wasn't. Eventually he handed over the tool I required, leaned over the table (as I was a temp at the time) and said "You're never going to be perm with an attitude like that".
If I could of afforded to walk out at that point I would of. I never spoke to the cunt again and have always wished him dead. Shame on him for making me think that.
Some people are such cunts.
I've got a DBA become a sysadmin, who then does things like listing printers in CUPS by IP ADDRESS, and then is amazed when the DHCP server hands them a different address and things shit the bed.
Oh and blames it on the DHCP server, of course. FML.
They've all got perfectly serviceable names, used by Windows and everything else in the business, and suitable labeled on each one.
Real sysadmins just don't become developers. They're hard bastards with more than a supçon of sadism†; Not soft buggers twaddling on about pointless syntactic preferences in the latest obscure programming language de jour.
† more like a tonne of bricks should you mess with their systems' uptime.
Real sysadmins are also real hardcore developers but they do all the hard work and loath their so called "developer" counterparts because Real sysadmins always have to clean up their so called "developer" counterparts' crap. Also, Real sysadmins know to handle it all, including a LART and know how to handle the so called "developer" counterparts by having a good planning knowledge of basements, elevator shafts and windows (of opportunity).
Quote
"Real sysadmins just don't become developers. They're hard bastards with more than a supçon of sadism†; Not soft buggers twaddling on about pointless syntactic preferences in the latest obscure programming language de jour."
Too right. Had a report of my PFY dealing with someone by leaning back in the chair, putting her fingertips together looking over the steeple at said errant member of staff and saying "You fooked up badly... I think its time you saw Boris"
According to rumour, he pleaded "No not that! I'll be good I promise , I'll even cut off one of my own legs with a rusty hacksaw rather than see Boris"
He is NOT wrong, and I am on the DBAs side here.
We put all printer on a fixed DHCP reservation. This way you see in one UI a good overview of the IPs. And you spot printers which get removed without ever informing since the last lease update was two month (sometime two years) ago... You CAN use DNS names, but then you have more work taking care of DNS as well, especially if you don't have a fixed IP, the IP changed, and then the name still resolves to the old IP for a while due to various caching mechanisms.
(And no, Servers don't get DHCP IPs, they get fixed...)
And I was talking about big multinational companies and government scale customers. Does not mean that printers were not added to the LDAP directory too depending on the customers, or who ever administers the site wish - even Windows AD has that option. But if you want a procedure for printers working on a large scale with over hundred other admins, the print servers talk directly IP address. That is the only way to be sure every admin can handle it. Oh, and this is not about ipv4 or ipv6.
Aren't the most useful details of a printer where it is and what kind it is?
I've been asked to support a printer by someone who couldn't tell me anything about it at all. I resorted to sending test print jobs to the printers I could reach online asking whoever found them to phone me.
> I've been asked to support a printer by someone who couldn't tell me anything about it at all. I resorted to sending test print jobs to the printers I could reach online asking whoever found them to phone me.
Was the incident "printer won't print?". Just askin'...
Sigh, that's because Windows, with its WSD or whatever printer driver, or the PCL printer drivel*, _doesn't work_.
How many, many, many times I had to go to peoples' computers and reinstall printers with the same settings, to get Windows to see the damn thing again. IP address worked better, hostname worked occasionally, samba name... not really.
Eventually I started installing the printers. with TCP/IP socket, DNS hostname, and Postscript drivers. Every single one that I did like that *never* had a printer reachability problem.
About 20ish years ago, I worked as IT support guy for a small PPI claim back company to provide support for their Windows PC's. I was only there for a few months but during that time, my boss got fired and I was passed to another manager. A manager that had absolutely no IT or technical knowledge whatsoever and who sacked me within a couple of weeks because 'I didn't re-program their Mitel (maybe) phone system quick enough.'
At the time, I already had years of experience in IT support but the closest thing I had to phone support was plugging them in. So for this phone system project I took my time trying to understand the changes that she was asking me to do. Knowing the risks involved, I told my new manager that I had no experience in making the changes she was asking for. Her reply was simply 'It can't be that difficult. Go speak to the phone provider. They'll help you.'
Yeah right. Which company is going to be stupid enough to help a customer to complete the work? The work that they would normally be paid a couple of grand for them to complete themselves?!
I called them up anyway and they provided very basic support. This is how to sign in to the admin console, what each section is used for etc. etc. Why would they provide anything more which, as I've already said, would take food from their table?
Whilst I was figuring out their phone system, I found the call reporting feature which showed that they were losing a lot of business because prospective customers were calling in the evening. The office was only open 8-5:30.
But before I could bring this issue up, I was summoned to a brief meeting with my manager and was basically told 'you've not completed the project quick enough. It couldn't have been that difficult so here's your P45 and there's the door.' So I kept my mouth shut about the report I've created and the company folded a few months later.
When we moved offices and had a new switchboard installed, I made friends with the installation engineer who let me have a copy of the software the company used to program it. DOS. Luckily I had an older portable with RS232 interface. Never any problems after that - and no communication with the manufacturer.
"So I kept my mouth shut about the report"
I might have been tempted, about a week after being fired and preferably after you've found another job, to go a couple of levels above her head point out that just before you were fired you were about to tell her the company was missing whatever amount of business it was but didn't get the chance. And emphasise that as you don't work there any more the only way they can find out how they were missing it will be to work it out for themselves - which they obviously won't have the skills to do.
Someone let me loose on the MicroVAX console... I had little to no sysadmin experience and a fairly interesting (I have mentioned her before, regarding tape spools; this incident was before the tape spool incidents) analyst wanted her job doing at a slightly higher priority than everyone else's equally mundane tasks.
What could possibly go wrong, you may be asking... well, young and impressionable idiot me decided that, as priorities went from iirc 16 (do anything else first and if you feel like it run this job for a bit) to -16 (real time) what's wrong with -16 for a quick job?
Let me tell you, dear reader, that -16 means everything else (terminal IO, other proceesses, Swap, everything!) stops until the -16 job gets finished.
The job crashed/terminated after a few minutes and released the cpu to do other things else but it was a "learning experience" at the time and something I still remember nearly 40 years later.
niceness levels, not priority. How nicely that process behaves with other processes.
A high number: very polite. Will hold the door open while everyone else goes through.
A bag negative number: absolute arsehole. Will barge people out of the way to get where they want, leaving a trail of destruction.
>>niceness levels, not priority.
Nope - not Niceness. Priority. Wasn't *nix but (Open)VMS. Very different - I knew about niceness and assumed that priority was the same.
Have looked it up, just to be sure; n was, apparently, 0 to 31, with anything above 16 being realtime.
Sigh - I remembered it wrong, but the effect still stands. Anything in the highest bracket (16-31) always gets executed.
For reference I include some OpenVMS Guidance on priority
SET PROCESS /PRIORITY=n
SET QUEUE /ENTRY=xxxx /PRIORITY=n
When real-time processes (those with priorities from 16 to 31) execute, the following conditions apply:-
They never receive a priority boost;
They do not experience automatic working set adjustments;
They do not experience quantum-based time slicing;
The system permits real-time processes to run until either they voluntarily enter a wait state or a higher priority real-time process becomes computable.
So Yeh crash and burn if you set a priority to >16 because, IIRC, not even OS processes run higher than 15 - it's the co-operative in co-operative multi tasking - everything promises to be nice and not grab the CPU for (potentially) ever.
Let's not forget that other axiom any sysadmin should adhere to: never, ever, run a script in production without a thorough understanding of what it does AND testing it first. It does not really matter whether you plucked the script from internet, or it was handed to you by a trusted colleague, or even if you wrote it yourself. Be aware of what yu are doing - the Golden Rule in system stability.
When I first worked nearly 50 years ago disk space was expensive, and I worked on DOS on the mainframe.
I was in the build group compiling a product. We had the build system and the live system.
I followed the instructions, deleted the old product, and started compiling the new code.
10 minutes later someone wandered round saying there was a problem.
I was working on the live system and not the build system.
My supervisor sorted out the problem and every one was happy.
They changed the instructions to say change this line.... to point to the build system.
Next week, I duly followed the new instructions and 10 minutes later someone came round...
I was building into the live system again.
My supervisor sorted it out - and changed the instructions to say "change >both< occurrences of..."
and that's one reason why I was moved to another platform with read only disks.
They also rewrote the instructions to include "what everyone knows".
These problems sounds quite a bit different than a prior story you posted with a similar outcome:
https://forums.theregister.com/forum/all/2024/04/08/who_me/#c_4840709
Are these in-fact different incidents, and you just had a talent for taking down VSE production systems over and over?
I was pulled into a meeting because I was being blamed for a late delivery of a project. The manager demanded I explain what I thought was the delivery process and who was responsible for what part. I did so, just calmly pointed out I did not have the access for the pre-reqs needed for my part of the delivery. I had chased it several times. He looked at me, said 'You're right - you may leave" and then proceeded to chewout the guy that tried to drop me in it. I seriously respected that manager. Funnily enough, I applied for another job years later and he was in charge - as soon as he saw my resume, he just said "hire that guy, he's good". I miss that kind of manager.
On the flip side, I was managing a shift and one of my guys did something a little silly (back in the days when we could set our session login names - he used something inappropriate). Someone spotted this and took offence and wanted to discipline the guy (where a talking to was more appropriate than a disciplinary). I refused to name names and took the heat myself - it happened on my watch, I'll deal with it and it won't happen again.
Years ago I started on my first full-stack position. I had experience writing web stuff in PHP and PostgreSQL but this was my first role doing those things for real.
I found that the codebase was using some deprecated API to connect to the database, and so I rewrote it to use the modern versions and cleaned up the code. My manager thought it looked good and we merged the change.
A day or so later (on a Friday), another engineer is confused because the mock database in his development machine went blank for no obvious reason. After both him and my manager spent hours trying to figure out what could have caused it, they realized my change had neglected to preserve logic that checked whether the code was instanced to run in production or in a test environment, and if it thought it was running in a test environment, the first thing it would do is… drop all tables.
Now here's where it gets nasty: every Saturday night an automated system would run to backup the production database. If this change hadn't been caught, it would have wiped the production database, forwarded the wipes to the replica server, then created a backup of the replica. Which means it would wipe all the customer data for the week.
We tweaked both how we ran tests and how we performed backups after that :P
but had let me go ahead anyway so I could learn by doing,
That is the exact opposite of me working for a large private contractor for a goverment dept .
I.T was outsourced and the remaining Gov it manglers overseeing it had devolved into insane Nazis that would trawl the network for the slightest thing out of place and take delight in demanding somebody was kicked off site and sacked for it .
Which happened to me and turned out to be the best move of my career
We had one of those. Bifurcated enterprise with main installation in capital city and ours in biggest earning city.
Fsckface would log on and monitor transaction rates. If he spotted an issue, he was on the phone calling and cussing like a trooper.
Then they amalgamated the two operations and ran everything on one mainframe in capital city, under Fsckface's direct supervision 'to save costs'. Fsckface had been working for this for literally over a decade, and had sweet talked any number of his managers into believing how simple the switch would be.
Problem arose on day 1, about half an hour into the business day. Despite Fsckface monitoring our transaction throughput for over 10 years, he never registered in what passed for his brain the transaction numbers, only the momentary flow rates. Capital city mainframe required an extra processor be bought for mucho denero and I mean mucho before the thing would run faster than a snail with gout once loaded with our transaction loads, and Very Loud People got involved before the cost-saving was working as advertised.
The joke was on him. Many of his colleagues had returned as consultants after a mandatory 'down time', and he fully expected to do the same, but it turned out he was as loathed by his capital city colleagues as he was by those of use working where the bills got paid, and that sweet consultancy was never offered.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
