Teens maintained a mainframe and it went about as well as you'd imagine One of the joys of Monday mornings is arriving at work to find messes made over the weekend. The other is reading a new edition of Who, Me? It's The Register's weekly reader-contributed column that shares your stories of somehow recovering from failure. This week, meet a reader we'll Regomize as "Serge," who, when he was just …
I remember entering a LISP assignment late on evening using on of those (all the VDUs [CRT] as usual were taken.)
My efforts worked 100% first time a circumstance that must have had cosmic significance as on leaving the lab I noticed the full moon was extremely dull and orange. A lunar eclipse. ;)
Like Serge I had a night job looking after a HP3000 MPE minicomputer and HP1000. After a power outage and UPS missing in action had to bring up the 3000 from toggling the row of front panel switches (top right in image) to enter the bootstrap into memory right through to multiuser.
Knew nothing of this stuff until then but locating the manual and the (over)confidence of youth, I pulled it off but unfortunately probably doomed me to a life of BoFHry but then defenestration and unreported basement interments are among the perks.
Needless to say after moving on I avoided HP kit, MPE and promptly forgot SPL but looking back the system was probably quite advanced for its time. The command lines were extremely verbose I recall something like listUserCatalog against DIR or ls but possibly the system manager had configured it in moron mode as the capabilities of the entire user base were rather 'limited' in just about every sense.
We had a DECwriter as the console on a VAX 11/750 running Berkeley unix. Every minute a cron job printed the date so that we could tell when any output had happened.
One day we suddenly started getting an error for every command - something like "out of processes". We found that the console had run out of paper, and several hundred "date" processes were stuck waiting for their output to complete.
As a student I remember dumping the whole VMS 2.x "HELP" output to the DECWriter in our lab (we didn't have a set of VMS docs there). That gave us a nice 2" deep pile of fanfold reference material.
A few days later the DECWriter stopped working, apparently the dot-matrix printhead had burned out... We never owned up.
My first IT manager had the ability to show both, even though he wore glasses and had the requisite dwarf beard. It was a couple of weeks into my new job, I got the short straw, and had to visit the office of the new Director, and it was a hot PRINTING issue just before a meeting. As a n00b, it took me a few minutes to track down the cause(s) first was the fact that the new director wanted to use his personal laptop he brought from the states, instead of the company issued laptop. Second was our friend A4 vs. Letter. I explained the difference, and how the work computer was already configured for printing AND backed up and left the decision to him.
The next day I got the gleam and smirk, and was told just to report stuff like this back to him and not to give the user a choice! I later assisted this director with other issues, and earned a letter of appreciation.
In high school, we had a PDP-8/e, connected to a Teletype, model 33 ASR, which is essentially a solid-character, moving-head, impact printer with keyboard, paper-tape reader, and paper-tape punch.
The beast was fed 8.5-inch-wide rolls of newsprint paper.
One evening after school, I completed the assignment due the next morning, and the paper ran out. The spare rolls of paper and paper tape were kept in a locked supply cabinet, to which only the teacher had a key.
I kept with me a set of "interesting-looking" keys, but had none which fit that lock. In a burst of inspiration, I dashed down to the W.C. found a key which worked on the paper-towel dispenser, removed the 8.5-inch-wide roll of paper, returned to the classroom, mounted it on the Teletype, and made my printout. The characters were smashed-in on the soft(-ish), (semi-)absorbant paper, but the instructor accepted the printout the next morning.
You cannot imagine how sad it makes me having a 33ASR explains. But I suppose we have a younger generation to educate.
It was an unbelievable feeling sitting at one of those and realising I had the whole of a 1904 at my command (I don't think it was supposed to be the whole of it but when it proved possible to crash it it really was). How I sit with this laptop on my knee with equally unbelievable times more power, memory and storage. Particularly storage bearing in mind that the fixed disk needed a building of its own and had to be properly aligned with the Earth's axis to prevent precession ruining it and individual allocations were 100k words.
(I don't think it was supposed to be the whole of it but when it proved possible to crash it it really was)
That was typical for a 1904 running George 3, which was the set-up for the 1904s I was sysadmin on.
George 3 and any jobs it was running all had separate address spaces, each containing its own set of accumulators, address pointer, entry points, etc. and any job could have additional subordinate address space(s) containing the G3 scripts and/or binary programs it controlled.
There was another process that managed task switching, creating/destroying jobspaces, etc but this was essentially invisible to both users and operator unless George 3 was being started, stopped or reporting a system crash.
For a while I had a surplus Model 15 (I think) KSR. This was the version which used 5-bit character encodings, had three rows of keys (plus a spacebar), and [Letters Shift] and [Figures Shift] keys.
It had been left out in the rain for months, and some of the parts had obviously corroded.
After pulling out the plant detritus, I brought it inside and let it dry out, wired it for loopback operation, and turned it on. Unsurprisingly, it did not work properly.
I unplugged it, sprayed the hell out of it with WD-40, let it sit a day, then, each afternoon when I came home from school, I'd spend 5-10 minutes typing on it, then unplugged it, and lightly re-sprayed it with WD-40.
After a week or so, began working properly.
"When faxes were new, we asked the salesman "do you need special paper?";"
It must have been a unique fax machine. The first ones I can remember used a thermal paper. Any printouts I have come across in old files are illegible or very crumbly. As soon as laser print fax machines came out, I got one of those and when I could turn an old computer into a fax machine and could review faxes without printing, I did that. Towards the end of my manufacturing company, I received far more spam on the fax than legitimate communications to the point where only faxes that had a code would be accepted.
Whereas one of my managers received a fax with something critical to the business, would be referred to often and decided that lamination was the way.
The lovely piece of double sided toasted fax paper was then pinned about the fax machine as a warning to others. Fax, Photocopy, THEN laminate if needed.
The problem I found with them was that the aluminium that was vaporised would deposit itself close to the printer. The inside of the printer would get covered with black 'stuff' which was mildly conductive, and eventually enough would build up to do something bad, and it would either stop working or start printing printing gibberish (the pointers would catch a 'tail' of the stuff, which blurred the dots). My father had one, and I was forever trying to clean up how the prints looked by cleaning the muck out.
There was an elastic band driven by a motor with two wire pointers, which moved across the paper. Power was passed to the pointers, sparking across to the paper, which being covered with aluminium, was conductive. There was a wire brush that contacted the surface of the paper to complete the circuit. So each complete rotation of the band would drawer two horizontal lines across the paper. As the paper moved continuously, the lines were drawn at a slight angle, which made the characters slightly skewed, not that you would notice, as the timing and position of the pointers was pretty inexact, and often adjacent horizontal lines were offset from each other in a jagged manner.
The operation was very simplistic. The ZX computer would start the paper and head motors running, and would then turn the power that produced the spark on and off rapidly for each dot on the print scan line. There was some method of the ZX computer knowing where the pointers were (I can't remember exactly what it was, it may have been a black line on the semi-translucent band read by an optical sensor), and this would initiate the sequencing of the scan line. The entire thing used the Z80 CPU of the computer to drive the printer. It was noisy, smelly, and you would never use the print for anything remotely official.
The other thing is that because it was inherently electrically noisy (no listening to the radio while you were printing!) there was no way that it was ever going to pass US FCC interference rules, so it was never available there. I understand that there was a more traditional thermal printer using something like a till roll mechanism marketed by Timex in the US, but I don't think that it was a Sinclair design.
"The spare rolls of paper and paper tape were kept in a locked supply cabinet, to which only the teacher had a key."
I've seen plenty of stories of supplies being kept in locked cabinets/rooms leaving some poor sod with little choice (in their mind) but to improvise. Many times the creative method used to overcome the lack of proper supplies has cost more than what it would require to replace the whole cabinet full of stuff. Countless times I've had a cashier that's had to call a supervisor to bring an new roll of receipt paper as none was kept at the till. How much did that cost in lost productivity? Since it's become common to get a receipt loaded with advertising, it happens even more.
Countless times I've had a cashier that's had to call a supervisor to bring an new roll of receipt paper as none was kept at the till. How much did that cost in lost productivity?
If you define productivity as "completing the day's sales", and assuming customers were content to just wait longer or use a different register, the answer is "none" — which is probably the crux of the problem.
If the cost of a decision is merely to inconvenience (but not lose) customers, most businesses are more OK with that than you'd prefer, sadly.
One fateful night, Serge's pal forgot to get extra paper, panicked, and started shoving anything faintly paper-related into the machine – including cardboard.
WTH? Suddenly this story veers into bad sitcom territory. I can't really grasp the mindset of someone who panics THAT desperately, at any age. It doesn't seem like the sort of thing you grow out of, either. I shudder to think what that guy is up to these days. I'm sure the hijinx continue to be "hilarious".
Some people panic when confronted by problems they would otherwise be able to easily solve, but are overwhelmed by the dreadful (sometimes, only-imagined) consequences of the situation.
In this case, the "obvious" solution was to take the DECwriter off-line, let the spooler hold the print jobs as it ought to do, leisurely find replacement paper, load and align the fresh paper, then turn the DECwriter back on-line.
OK, so what? Offline the DECwriter, set it up with a new box of paper, turn it back online, and you can once again type system comnands to rhe computer.
I never had a DECwriter, but I had a surplus ("Just haul it away and you can have it.") Data General Dasher TP-1 dot-matrix printer+keyboard, and a Nova 4 minicomputer to hook it to.
This account confuses me. The DECWriter II was purely a sprocket tractor feed printer. It was not like a typewriter or cut sheet ink-jet printer, where there were rollers that pinched the paper to act as a friction feed. Unlike later printers like Epson FX-80 (to quote an early PC printer) it did not have a friction feed mode. If you were not using continuous fanfold paper with holes for the sprocket feed mechanism, the paper wouldn't move vertically. Also, I don't think that the position of the head was user-adjustable beyond 6-part stationary, so putting anything thicker than this would probably cause the fabric ribbon to drag across the surface, leaving horizontal black stripes, and possibly getting the ribbon shredded by the print head (and I've seen this happen - inky black fluff everywhere).
If you needed to print on something a bit more like paper sheets, you would use fanfold paper with vertical perforations allowing the sprocket holes to be torn off. I used a DECPrinter (a faster cousin of a DECWriter, but without the keyboard) to print my first year project at University like this (original roff on UNIX Edition 6 - nroff did not appear until Edition 7).
What I normally saw happen in situations such as described was that previously used paper would be turned over, and printed on the back.
I can totally see someone (and by "someone" I mean the 1980s workplace comedy-of-errors stereotype that this other kid is portrayed as) manually dragging cut-sheet paper through the feed mechanism, advancing it by hand as each line of output is printed. I'm sure the off-kilter output lines, swooping roller-coaster hills and valleys, and partially overlapped text just added to its... "charm".
"This story comes courtesy of a reader we shall Regomize as Leon. As luck would have it, it seems Leon was left alone one night, in charge of a once-great country's entire government, when he accidentally pushed an update onto the production server, which deleted all personal rights and freedoms. Unfortunately, the country hadn't been successfully backed up for over a hundred days, leaving him with only six hours to retype the entire database. Luckily, earlier that year he had been cruising around the undergraduate common room and had some friends on speed-dial..."
Ah, the days of early interactive computer use. Compared to the steam-driven Teletype machines I started on (writing BASIC to a PDP-11 fronting for an ICL 4130) the appearance in select areas of campus of DECwriter II's and even VDU's was revolutionary. And of course, an opportunity to waste time playing STRTRK on the new DECsystem 10.
Being of the "I might need it later" persuasion, I may even have some of those 50-year-old dot matrix printouts squirreled away somewhere. And let's face it, without the opportunity presented by all that tech, I might never have had a software development career to retire from, shuffling instead through the stinks of chemistry.
We had classes using the Novell network for coursework, and each student needed a page with his account name, server, password, login instructions, etc
I was the bloke what printed that, as a "student assistant" (aka cheap labour)
This was something like 8K-odd pages IIRC
So here it was 3am and the printer runs out of toner. It was a very expensive high speed laser printer back when laser printers were expensive to start with.
I scoured the place for the toner for the printer, but found nothing except these generic white bottles labeled "TONER"... no one had ever shown me how to take care of the printer, but I managed to pry open the door to the top of the toner compartment.
I knew the jig was up when the pages came out covered with a thick sheet of black plastic.
I went home and presented myself to the director next morning, who understood the situation and noticed my air of complete defeat. Fortunately, a new roller/fuser cartridge and an actual new CARTRIDGE with toner in it, and we were back in business.
I did a very similar job at age 18, running an IBM mainframe computing centre overnight as the sole operator on the night shift. The jobs were mostly remote, but early one morning, shortly after I started, a crew from an oil company showed up and said that they had booked one of the mainframes for exclusive in-person use for an hour. They had data tapes in hand and were ready to start at the scheduled 5:30 am time. Nobody had briefed me that this was a possibility, so I wasn't sure what to do. I could call my immediate manager and wake him up a 5:15 am, but that didn't seem like a good idea. The oil-company crew said they had done this many times before, and they seemed to know what they were doing, so rather than holding them up and having IBM appear unprofessional, I told them to go ahead and left a detailed note for my boss. At 7 am the day shift operator came in and said that it was fine, a routine job. So I went home and collapsed into bed to sleep. An hour later I was awakened by an aggrieved call from my boss, asking why I hadn't properly recorded the starting meter reading on the mainframe so the customer could be billed properly for usage - never mind that the procedure wasn't written down and nobody had briefed me on that part of the job. *sigh*. He should have praised me for taking care of the customer! Anyway, there was no way he could legitimately blame me, so I didn't lose any additional sleep over it. :)
Most of my duties were receiving paper-tape, reading it, punching cards and passing them through a mechanical sorter and then collator, and then reading these back onto the tapes on the 360 where they were further massaged. An incredibly tortuous way that had been slowly improvised on over the years.
One day the consol popped up a message about SYSVOL being out of space. Since I had spent idle time reading every IBM manual I could, I knew that our particular operation didn't need to have a fancy BAL library that was stored on the SYSVOL. So I deleted it. Everything continued to run fine until one day a chap from Chicago shows up to build and load some new software. Boom!
I learned enough in those few months to start a 50+ year career - and still loving it!
" Nobody had briefed me that this was a possibility, so I wasn't sure what to do. I could call my immediate manager and wake him up a 5:15 am, but that didn't seem like a good idea."
It's a great opportunity to get your boss out of bed at an unholy hour and you should have used it. No entry on the calendar, no memo, no procedure, etc. A random lot of people showing up before the chickens wake up and asking for mainframe access these days would scream "danger". Ahh, the kinder gentler days, but time on a big machine was valuable and billable. I see it would have been an opportunity to sit down with your manager and outline the procedures (on overtime, of course). Projects like that look good on a resume a lot more than only being a night shift button pusher.
My last job mostly involved fixing printers and projectors at a large independent boarding school.
The school was spread across many acres, with buildings dotted around the town. One of the boarding houses was almost a mile from the IT office. Thankfully, as this house was mainly occupied by rugby players, they made little use of IT, so we were rarely called up there.
One day I was summoned there. I remember it was cold and wet, and I cursed the house master every step of the way up the muddy path.
Upon arriving, I was ushered into the library, this was a little used room, which was a shame, as it was oak panelled with a real fire. Standing on a card table in the corner was a Brother laser printer with all three lights flashing. It was only a paper jam, so easily fixed whilst enjoying a cup of Taylor's Italian blend coffee, which all the houses served, and was one of the many perks of working at this school.
5 minutes of swearing and scuffed knuckles later, I retrieved the jammed paper. It wasn't printer paper, it was a sheet from a lined A4 refill pad, and unless this printer had a remarkably accurate "barely literate teenage boy's handwriting" font, the paper had been written on before feeding into the printer.
I put the printer back together, tested it and reported my success to the housemaster, he thanked me with a bottle of 19 Crimes, another perk of working at that school...
Working for a large company, as I did for 10 years, they were strangely stingy with their IT budget on occasions. I had "inherited" a Compaq III luggable, one of those chunky box shaped things that were portable only between mains wall sockets, and with a little red plasma screen that could be tilted upwards and a small keyboard. This thing had a whole 20MB HDD and a 360k floppy. It did not take long for me to fill the HDD but also 2 boxes of floppies. This was getting tiresome so I requested a new PC.
I think I was one of the heaviest computer users in the group, being in the middle of designing a £12M chemical plant at the time.
Go and see Phil (his real name) over in Block 10, he does the procurement, I was told by one of my colleagues. So off I trotted. Phil's comment was that there was a limit of IIRC £800 on new items, but that we could get round this by ordering a separate system box, monitor and keyboard / mouse. Great! I thought, and a few weeks later received a call that my machine had arrived and the IT team were setting it up for me, but I could pop down and have a look. My mighty Compaq Deskpro 386SX20 with 80MB HDD was sitting on a bench and next to it a full fat IBM tower case PS/2 Model 80 with maths co-processor and graphics card plus a big screen monitor and IIRC 120MB HDD.
Whose is that? I asked. Oh, that's JC's (his real initials and head of our section of 500 people). What does JC do with that? I asked, somewhat puzzled, as JC's leggy blonde Cheshire Set secretary did most of the work for him.
Oh, he reads his emails and writes the odd letter, I was informed.
So yours truly had battled the system to get the minimum spec machine to do the job, and this powerful (for 1992) machine was being used to read emails and type letters! Rank hath it's privileges I was told.
Later on I was given a fairly gentle but firm chewing out by my line mamager for getting my machine by the back door, even though I was in desperate need of it to do my job.
"Don't do that again" I was told, very clearly.
PTSD flashbacks: Working for a hospital one time, at the end of each financial year they would spend all the money they'd held in reserve for emergencies.
"Get 3 competitive quotes for delivery in the next week". Sure, boss!
- We got around the "competitive quotes" BS by requesting quotes for $BRAND PC (our standard) from $Preferred_supplier, and a couple of resellers who were more expensive and didn't have the stock.
- $Preferred_supplier knew the game, and would invoice by the deadline, even if deliveries were delayed.
Then came more stupidity; Directors had to get the latest and greatest, even if a boot logging script showed they never switched the PC on. Their PCs had to be cascaded to their secretaries. Whose PCs would be given to other admins. And so on, until we had spares for clinical departments that were desperate for anything.
- At each step, the handed down PCs had to be wiped and rebuilt. I can't actually recall why, IT security wasn't even a thing then, but it meant the cascade took months.
We did it the other way round where I worked once. If you had a new computer, everyone asked questions about budget, where it came from, spawned jealousy etc. But if you kept your tatty old case but had completely new internals fitted, well in that case you still had your 'old' computer and no-one noticed or cared. So we spent a fair bit of time stealth upgrading people and the management were none the wiser
We did pushdowns, but a team of seven or eight of us could do a whole department, with multiple new PCs and 30 pushdowns in two or three days! Caveats:
* This was during the Windows 3.1 era
* We didn't have to wipe the discs then
* We used Binary Research's "Ghost" to lay down our basic Windows 3.1/MS-DOS image on the new PC
* We used Frye Utilities to transmit the data files (only!) from the older PC to the newer PC over the network, via IPX, and used LapLink for computers lacking network cards
* Users had (usually) very little dara stored on their PCs, as we continually exhorted them to use their network-resident home drives and deparmental shared drives to store their data on, as those network drives were automatically backed-up
I worked at a PC builder during the windows XP era. Back then if you tried to put a harddrive with XP installed on one set of hardware, into a different machine, it mostly wouldn't boot.
So, this company had to have a different Windows install for every different configuration the sold. Machine B is identical to machine A, but with a soundcard? New image. Different motherboard? New image. Each build image was numbered, and once it had been built, a Ghost image was made, and stored on DVD-R. During my time they were into the 6000's.
So, when a new batch of PCs was being built, someone (often me) would have to go pick all the different hardrives from the warehouse, and then painstakingly write the correct Ghost image to each one. There was an IDE/PATA harddrive duplicator, but this was just as SATA drives were being introduced, so we had several PC's with SATA ports, and each drive was just stood upright on a shelf and plugged into the open PCs. I discovered that SATA was effectively hot-swappable, which saved boot time (Ghost was booted off a floppy disk in each machine), and also that a drive-to-drive clone was possible, which was much quicker than imaging from DVD.
I could of course then use that saved time to sit on my arse and do nothing :)
> end of each financial year they would spend all the money they'd held in reserve
Same for almost all public organisations, in the 1980's I was working at a maker of Diesel Generating sets and sure as hell every February/March we would get "need to spend money now" requests for a new genset. These were always invoiced by the deadline even if the deliveries were later in the Summer, they were a nice little earner as well as the money "had to be spent".
I have never understood the mentality of this and the bean counters, surely they must know what goes on but they just keep on doing it.
Yeah, we've referred to this numerous times on El Reg.The incredible, wasteful inefficiency of ( approaching) end of year panic buys or budget clawbacks. Good managers penalised for carefully managing budgets, which means they lose the (allocated but not yet spent) cash prematurely and then get a reduced budget the following year. Else cupboards full of useful, but unneeded stock items- or creative purchasing ( buy stuff this financial year and return it for a credit note and so on).
So fucking wasteful of money, and time.
In the N Ireland Civil Service this was known as the Spring Sales and was, I understand, mostly money kept by DooE against the possibility of a bad winter needing a lot of snow clearance and road gritting. I was able to use some to get in consumables for the next year - microscope slides, cover-slips, methylumbenlliferyl phosphate...
The mentality is simple : if they haven't spent their yearly budget, they are rewarded with a smaller budget the following year.
Since increasing the size of the budget for any "special" project is months of discussions and, quite possibly, all of that ending in a refusal, you spend the effing budget.
Kind of like the fighter jets who, at the end of the fiscal year, spend some time taxiing to and from the runway (without taking off, because that would become "flying time" and treated differently) just so that the yearly fuel budget was spent (obviously, CO2 emissions are not a military concern).
I once started up a sort of combined student club/computer lab at my uni. The department Chair and school's Dean gave me a (very nice!) room and a small budget, most of which went for a wall of whiteboards and proper electrical service.
Which left the small matter of stocking said room with computers. I knew the IT staff, and they were kind enough to donate several dozen machines that had fallen off the bottom of the upgrade cascade. Slow CPUs and antique Windows versions, but they could run Linux when the drives and motherboards weren't flaking out. Which was fine, since Linux and FOSS were the whole point of the thing. The students got plenty of practice repairing hardware and cannibalizing parts, but they kept the lab going and the organization growing.
A few years later they got a decent equipment budget, and a few years after that got some nice server room hardware donations from alums who'd gone off to work in research labs for several big name firms, so it all worked out. It's still going, 20+ years on.
Always the way.
Back in 2017 the if-it-breaks-no-work-gets-done lab PC I was given upon joining my then employer was a Intel Core Solo machine, running Windows 7 32-bit. 1GB of RAM and an ancient 60GB HDD all mounted inside a somewhat battered micro-ATX tower - it appeared to have been thrown together by the IT contractor over his lunch break from bits he had lying around rather than the company buy anything. It also came complete with a rollerball PS2 mouse and a 15" 4:3 LCD monitor....it did the job, but the HDD would usually be going mental with all the swapping going on if you tried to run Outlook / Office and any of the lab software at the same time.
When I went to collect it with the lab trolley, it was sat next to a rather nice and shiny i7-7700k system - 16GB RAM, 500GB NvME SSD, Windows 10 etc.. Plus two brand new 24" Dell monitors and a fancy Microsoft ergonomic wireless keyboard and mouse set. Turned out that was for the ops director (MD's stepson) who I seem to recall spent most of his day reading emails.
The elderly HDD in mine turned out to be very knackered, and died after a few weeks of use - sadly overnight on a test run while generating a lot of rather irreplaceable QC data for our biggest client. Thankfully a week before I'd seen the warnings and already had a ticket raised complete with screenshots warning of HDD failure.
There was some grumbling from my boss about why such important tests were being carried out on what in his words was "junk".
After a few further grumbles I got upgraded to a nice little i5 HP desktop mini (and joy of joys it had an SSD), a similar model now sits at home serving me well.
I guess I'm lucky, never really had to deal with that.
In my previous life with a big blue computer company we were working with used equipment, and if some of those parts went into our work machines no one cared... I had a SWEET souped up IntelliStation.
In my current job I was given a bog standard (and dull), Dell desktop, and it 32 bit word just couldn't handled the large documents I worked on. So my boss ordered a Dell workstation for me. Nice dual Xeon machine with lots of RAM, PLUS it had a PS/2 port so I brought in my trusty 1994 vintage Model M keyboard (which I am using even now).
Fast forward, it was time to "refresh" my PC, and as I had a big Dell Precision already I got a new one of those.
Meanwhile most of the office people are using Dell laptops, while I've got this big machine, my Model M, and three monitors. :)
"Whose is that? I asked. Oh, that's JC's (his real initials and head of our section of 500 people). What does JC do with that? I asked, somewhat puzzled, as JC's leggy blonde Cheshire Set secretary did most of the work for him.
Oh, he reads his emails and writes the odd letter, I was informed."
This story still plays out today where the barely conscious exec that equate the whole of the internet as "the web" since all they know is Chrome or Edge, is set up with a high-end gamer class PC and a giant bandwidth allocation so there's never any lag that isn't external. All they do is email and check that their wife/mistress aren't ordering lorry loads of junk from Amazon using their corporate card, maxing it out, which will be embarrassing when it's declined when they take a client out for dinner that night.
AC because the company is still around and doing business...
Company of significant size, 2nd largest in its field worldwide. 99% of its business relied on bespoke software backed by an oracle db. Sun hardware, at a minimum the environment consisted of one db server and one app server, which was what it was in the dev environment and QA. Production was naturally bigger, it had 3 app servers and a BIG db server. Lots of zeros on the scary "business lost per hour of downtime" number.
It came to my attention that the marketroids had been promising all our customers that "Oh yes, the system is fully redundant and resilient." It wasn't. Not a trace of it. Even those multiple app servers didn't provide any redundancy because the devs had built several critical modules in such a way that they spectacularly shat themselves if they actually HAD to fail over to another node. Lose an app server and you basically had to shut everything down, wait for the DBAs to unbork their mess then start it up again without the failed node before it would run right. Which took long enough that about 50% of the time I'd have the failed node fixed before it was time to start it all up again.
First step was when the app servers needed a refresh I made sure to "save some money" by making sure dev and QA environments had two smaller app servers, rather than one large one. Then I bribed the QA mgr with a nice bottle of whisky to make sure a failover test was included in the test suite for the app server software. The devs never found out who was behind that, thankfully, because they had to postpone a couple of releases to fix all that and caught a huge amount of crap as a result. OK, now the app had redundancy - because I'd made sure there was enough capacity in prod to lose an app server and still handle the volume.
Then came the hard part. Getting that database properly clustered and redundant. I couldn't hide that one in BAU stuff. It needed budget. So I made the business case, gave the usual "fast, cheap or good - pick 2" options and put in in the next annual budget request. DENIED! - of course. Next round of budgeting I tried again, same result. And Marketing were still fibbing to our customers.
Then the outage happened. It was a nasty one, one of those "drop-ship a replacement server and restore from backups" ones. That big db server was doornailed. The single point of failure had, in slavish obedience to Murphy's Law, failed.
Cue the screaming from customers, rage in the corner offices and many attempts to roll stuff downhill onto my team. At which point I forwarded both email threads where my requests for budget were denied, pointed out that this was exactly what those denied budget requests were supposed to prevent and that the lost business for the duration of the outage was actually more money than I'd requested that they spend to prevent it.
It got approved in the next go-round.
Had someone decide to re-write a shift handover (no idea why) - only they forgot to copy the bit that said "Do NOT print off the stock request print".
I come in on the night shift (just 2 of us as the other shift member was on leave and I am in charge), and we cracked on and started to print off the stock request. 40 boxes of fan-fold paper later, we'd spent all night having to sort out the printers (old ICL mainframe printers - can't remember the model) every 10 minutes because the crappy paper refused to stack properly. When the boss came in, he was NOT a happy chappy, but as we went back through the shift handover we saw where the original instruction had been crossed out and not copied. I was relieved - I was only 19 at the time.
heres comes the PTSD from having to deal with 16-17 yr olds who know everything and can wreck the most simple of jobs.
My personal favourite? was the guy unloading the robot pallets into boxes and putting freshly loaded pallets into place.... of course this is physical work so he had the bright idea of just leaving the parts on the pallet and hitting restart.... come the end of the day he reads off the number of parts made according to the robot and puts that on his pile of filled boxes. I look at the paperwork and it says 2000 parts, then I asked him to count the number of boxes(10) and how many in each box (100), then explained to him that I know what he did and not to do it again.
2 weeks later... he's sacked for doing the same thing only this time with the boss watching.......
theres the easy way to learn and the hard way to learn.... some folks insist on going the hard way
At uni we had in the basement a box with pens. Students and staff alike took one as needed. If the box was as good as empty a few dozen new ones were provided by our institute's secretary - no questions asked.
I once plotted the number of pens in the box as a function of time and got a perfect exponential fit.
My college didn't have enough Video Terminals my freshman year and we used some of those DECWriter IIs. Actually you could probably obscure your tracks by backspacing and retyping commands with corrections several times... The ones we had would backspace and type over the print again and again... made it hard to decode what you did when you made too many typing mistakes when reading the output.
My college had both Teleray video terminals and DECWriter LA36's. The Telerays were just glass TTYs and had no advantage over the DECWriters, at least with our mainframe. I liked the DECWriters. They were reliable, and you ended up with a complete record of your session. Their only shortcoming was a tendency to blow fuses. As I recall, I went to a hardware store and bought a few fuses so I wouldn't suffer disappointment.
One generous employer I worked for allowed staff to leave at 3pm on Fridays.
I'd take advantage of this time to run minor updates on servers, routers, firewalls and test.
It was a great chance to do some sysadmin work during normal work hours and have free weekends.
The "youngsters" that were hired used this time to set up a game server and play under the guise
of "going the extra mile" by also performing sysadmin tasks for our employer.
I was wondering why the network was sluggish each friday.
After a few weeks of noticing this, I fired up "ethereal" and saw a huge amount of traffic from the admin subnet.
A quick trip to several offices and the server room exposed the game playing.
I fired up a packet injector and flooded the network with UDP packets aimed at the game server.
After two fridays of doing this, the youngsters went elsewhere to play their games and I happily
continued using the time for work related activities.
No one ever found out.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
