If we can't fix this printer tonight, the bank's core app will stop working Thank Arioch it's Friday, because the working week is nearly over and we can ease you into it with another instalment of On-Call, our end-of-week trawl through readers' memories of awkward jobs. This week, meet “Jonny G”, resident of Canada who “Once upon a very long time ago, worked for a large IT vendor in a role as an on- …
You may scoff but it is more of a common problem than you think. Many a time I've seen a printer break and someone has had to fix it. In fact on more than one occasion a printer has broken and it has had to be replaced entirely.
We've even taken to keeping spares just for that very scenario.
Yes, but we all hoped to read about an heroic trip to the printer in a cold, snowy night among wolves, and then an epic fight with a daisywheel printer made when God slept in times of old, won just before dawn came an carried away all the PHB vampires who were trying to break the door...
Instead this guy just told to call the printer technician? <G>
In my time I was lucky if I wasn't strangled the ribbon.
Ribbon? You kids today don't know you're born. In my time, we had to wrestle and strangle a squid with our bare hands to get the ink for the printer. Flailing tentacles everywhere, and I tell you, there's nothing like the smell of freshly squeezed squid. I still bear the sucker scars. Thanks a lot for giving me flashbacks :(
> Many a time I've seen a printer break and someone has had to fix it. In fact on more than one occasion a printer has broken and it has had to be replaced entirely.
I've personally witnessed a few printers displaying the dreaded PC LOAD LETTER error. Even a hard power cycle wouldn't fix that one.
I had a situation where the year end print of the general ledger involved multiple boxes of fanfold and a cranky chain printer.
I would get the print job going and would sleep on the carpet (thank Arioch it was not in a real data center) until awoken whenever the printer would go quiet.
I coded the software and ran the jobs, thus I was a DevOp prior to 1980. And a bit of a BOFH.
A benefit to the job was a location not far from Times Square (pre-Disney) and a pub that served litres of German beer.
I was that printer repair man! Well, not quite. Mine was a hospital telephone switchboard system configured to record all incoming and outing phone numbers and print them out. No one knew how to switch that function off and the printer had failed. The switchboard box could only store about 500 numbers. Not fixing the printer would have borked the entire hospital switchboard. I remember it being an Oki Microline 390 but not what the fault was except that it was a 200 mile round trip and it took about 10 minutes to fix it.
What a stupid system design.
How did the printer catch back up all the logs while they were all still trickling in on a nearly-full buffer? Luck.
Not heard of speccing out storage to cover such eventualities? Only an hour or two of grace, or even a day or two? That's just ridiculous.
Someone should be fired - the guy who designed it, the guy who managed it, or the guy tasked with renewing it that failed to put that in.
Keeping the damn printer is even more ridiculous, in this day and age.
'For some reason "Once upon a very long time ago" doesn't quite mesh with the above comment.'
Absolutely. Some people find it hard to believe there was a time when new fangled tech like CRT terminals and monitors didn't exist and all text output from a system came via a printer. Hint - there's a reason why you login to a Unix box via a TTY session, and it's called TeleTYpe...
In my Computer Museum
I have the stack of cards that I wrote my first big program on sitting in my display cabinet. All 1467 punched cards plus the George III JCL.
Next to them is the punched tape that contains the source to me 1975 Degree project.
Finally there is a flyer for the rather special 32bit Mini that I worked on in the mid 1980's.
My grandkids shake their heads in disbelief when they hear about how we loaded a bootstrap into a PDP-11 through the front panel.
Things have come a long way in the past 45 years.
"My grandkids shake their heads in disbelief when they hear about how we loaded a bootstrap into a PDP-11 through the front panel."
Heck, sometimes you were lucky and the boot code wasn't overwritten... and was still in core on the next power-up. It's probably just as well that I've forgotten the RP-02 boot code (about 8 words if I recall correctly).
Let's face it, by the sound of it this system was probably written some time in the 60s. In COBOL. The guy who wrote it probably has long retired, if not already being dead.
Remember, in the 60s they expected these systems to be replaced within ten years. Hence the millennium bug...
"Remember, in the 60s they expected these systems to be replaced within ten years. Hence the millennium bug..."
Eurm, not really or not only. 6 digit dates were used to save space on storage media. An IBM 2311 disk had a capacity of 7.5MB. A 2400 tape stored, depending on blocking factor and recording density, 113-170MB.
So shaving two bytes off each date stored for an application could save quite a lot of (scarce) space. Also Y2K was 30-35 years down the road. I don't remember many people thinking that far ahead. The assumption too, as you pointed out, was those current at the time applications would be replaced some time in the future.
I remember doing a gig in the early 1970's for a large insurance company whose master file resided on 100+ tapes (tape because the data center's raised floor couldn't handle the weight of the necessary disk units if the file had been stored on disk). So cutting down on storage requirements was a must.
"Remember, in the 60s they expected these systems to be replaced within ten years."
Not necessarily. But they expected it to be SEP. They probably expect storage to be cheap enough in the distant future so they'd eventually be able to afford a couple of extra characters per field.
Well considering that a chain aka line printer could keep up with the output log I'm guessing it didn't generate a huge i.e, 1Mb or more of data which I expect would be the limit of the cheese platter style disk drive unit I guess the system was using.
BTW line printers were really awesome for doing charatcer based artwork on.. If you could handle the noise.
I know some of us do, but I wonder exactly how many of the readers really know what a "chain printer" actually is.
Back in the day, these were capable of 600+ lines a minute. The speed that the chain which carried the letters moved was such that if the chain broke, it could seriously damage the heavy metal acoustic cover. There was a reason why there were cover switches to prevent the printer operating when the cover was open, and it was not just the noise.
In my first job, the single chain printer that they had printed all of the council's rates demands and payment books, the car parking fines, the council house rent and backlog reports, the payment cheques etc, well pretty much anything that the council sent out in the post bar what the typing pool typed.
The printer was on a special I/O channel, and the particular model of Sperry Univac 90/30 could only have one of these printers attached. The printer itself was the size of a large wardrobe, and could not easily be swapped out.
They were a significant part of the cost of the computer system, and it really was a severity one call if it did not work for more than a day. Oh, and when it was down, none of us programmers could work either, because we needed it for the printed output from the batch (decks of punched card) compilation runs for the programs we were working on.
Peter
"because we needed it for the printed output from the batch (decks of punched card) compilation runs for the programs we were working on."
You might need to explain to some of the youngsters here (I guess about 50% of the readership) what a "punched card" is... I'll back you up, when they start downvoting you in disbelief. You could also mention that new-fangled high-speed technology, "punched tape", which, believe it or not dear readers, not only enables high speed parallel I/O but also eliminates the danger of "card stack drop" for greater software resiliency. Marvellous invention, I tell you. I just hope it catches on.
I love all this retro stuff, so it's no bother to explain. What I am talking about here is the 80 column IBM or Hollerith card that was common in the '50s, '60s and '70s, but was mostly obsolete by the '80s. I learned to program PL/1 on them, and also used them in my first job.
A punched card was exactly as it sounds. It was a card measuring 187mm by 83mm. It had rounded corners, with one or other top corner cut off to allow quick card orientation checks.
They had rectangular holes punched in an 80 column by 12 row matrix, with each column representing one character. Not every manufacturer used the same encoding type on the cards.
Generally speaking each card represented one line of information or a line of source code. Most card punch machines would punch the holes representing the character, and would also print the character at the top of the card, so that you could read the card.
Most languages (but not all) reserved the first or last six columns to hold a card number, so that cards could be ordered. When punching the cards, it was normal to step the cards number by 10, so that it was possible to insert cards into the deck without having to repunch the whole deck. Most compilers would allow you to miss the numbers altogether, but if you ever dropped the card deck.....
Punching the cards was done on a desk-sized machine with a keyboard and a transport mechanism which allowed blank cards to be fed in to a hopper (normally on the right), typed one column at a time, and then moved into a hopper on the left (cleverly turning the cards over to keep the order). The better card punches also allowed you to copy a card, one column at a time until you got to an error, and then punch the rest of the card.
Alternatively, you could get hand punched that allowed you to punch by hand the correct holes for each character, but you had to be desperate or very clever to use one. Some people claim also to have blocked cut holes by carefully placing a 'chad' (the cut out rectangle of card) into a hole to 'edit' a card, but I'm a bit sceptical myself.
Column 7 in most languages was a card-type indicator. Typing a C in this column normally meant that the cards was a comment. This may sound wasteful, but the comments would be included in the listing from the compilation. In some languages, like Cobol and RPG, the card-type indicator was used to identify the section that the line was in (Input, Calculation, Output, Exception).
Cards were read in a reader that pulled one card at a time (at a rate of 2-10 cards a second) across either an mechanical, air-jet or optical reader, which worked out which holes were punched. You kept card decks together with elastic bands, and just for security, most people would use two bands, in case one broke.
Creasing a card, or allowing them to get damp or worn at the edges would often cause the cards to jam in the reader, normally meaning that the whole deck would be rejected. Having a deck rejected meant that you missed the slot and have to put your job to the back of the queue once the problem was rectified. Remember that systems were mostly single tasking (at least for compilation streams), so they processed the queue in sequence, one-at-a-time.
I'm not sad to see the end of those days, but having lived through them, I feel that it imposed a rigour that would benefit modern programmers if they had to experience it.
"Some people claim also to have blocked cut holes by carefully placing a 'chad' (the cut out rectangle of card) into a hole to 'edit' a card, but I'm a bit sceptical myself."
Never used cards much myself, but I did hand edit paper tape so people editing cards sounds very plausible to me. If you were lucky, you just added the relevant hole(s). Less lucky, and you taped over the relevant column (usually also the columns either side, and then hand punched the new codes back in. This was originally on 5 hole tape, so one had to be careful to read through the tape for Figure Shift and Letter Shift codes since there was only 31 hole combos available. A splicer kit was a must have too.
If you'd been a bad boy, you got to use this monster instead of the ones with actual typewriter functions instead!
"Some people claim also to have blocked cut holes by carefully placing a 'chad' (the cut out rectangle of card) into a hole to 'edit' a card, but I'm a bit sceptical myself."
It's true - we used to do it occasionally (but only for single use decks). Put the chad back in and then go over it with pencil - compressed the card slightly to get a nice tight fit and supposedly filled in any gaps with graphite - Oh the things we used to do in the 70s.
In 1961 I was a brand-new RAF telegraphist working in the communications room of an RAF Thor missile site. There was an IBM Data Transceiver which sent & received 80 column cards. The cards were mostly equipment demands and were punched by the storemen: the most soul-destroying job was the verifier who had to take the recently punched deck of cards and re-punch them using exactly the same paper work that had been used to punch them in the first place. The theory was that any errors would be detected as two people were unlikely to make the same mistake.
We had a chain printer on the IBM 360 at university. It would let you know it was out of paper, or of other issues, by raising the lid on the acoustic enclosure. VERY effective at getting attention.
P.S. I know what punched cards are and what Hollerith is. Also the difference between an 026, 029 and 129 keypunch. "Bite my EBCDIC." :)
The IBM 1403 N1 printer model could print at 1100 lines/ minute or even 1400 lines/minute (when using a reduced print chain a.k.a. Universal Character Set).
When VIPs were visiting the data center a program was run to cause one of the printers to make 'music'. Other programs produced 'pictorial art'.
BTW line printers were really awesome for doing charatcer based artwork on.. If you could handle the noise.
Now consider the guy servicing these beasts, and doing hammer flight time adjustment. You could not always rely on the service kit containing hearing protection.
I've worked (occasionally) on belt and drum printers, never seen a chain printer AFAICR. For really torturing them you had a text file that had the characters on a line arranged so that the hammers would fire all at once. This had a fair chance of blowing the fuses or even the power supply itself after a few pages.
As a student I found myself in the possession of a HP drum printer one day. The fact that the drum had rings of characters for only one in three character positions quite baffled me initially, until I figured out that the paper transport could move sideways. So, left, print, middle, print, right, print, line feed, print, middle, print, left, print, line feed, and so on. Apparently saving 88 hammers (now 44 instead of 132) and hammer drivers made up for the additional mechanics.
BTW line printers were really awesome for doing charatcer based artwork on.. If you could handle the noise.
Yes they were. Long time ago I did a stint at Nixdorf maintenance department. Plenty of printed ASCII art on the walls as those were used to test printers.
I learnt to have healthy respect for line printers as they will make you pay severely if you don't show them the respect they deserve.
What a stupid system design.
In what way? Do you have a full list of requirements? Do you know what they were trying to acheve?
These were financial transactions on an old system, maybe there was a requirement to have a hard copy of the transaction within a fixed timeframe? So stopping the transactions from proceeding if this can't happen seems very sensible?
Not heard of speccing out storage to cover such eventualities? Only an hour or two of grace, or even a day or two? That's just ridiculous.
You do know how expensive storage is?? Even in the years long after systems like this became outdated storage was still very very expensive.
Someone should be fired - the guy who designed it, the guy who managed it, or the guy tasked with renewing it that failed to put that in.
Yes. You're right, you've got a half remembered story about a system you know sweet f/a about, but you're right... someone should be fired.
Keeping the damn printer is even more ridiculous, in this day and age.
Some day you may be the person in charge of making the decision to replace a system like this; You may then realise it's not quite as simple as you think.
you know Jim, the one that fixed the printer that time! Been telling the boys how Dave saved the day by phoning the vendor to come support their broken printer. That Paul is a heck of a guy, super support hero. The board where so impressed with Tims heroics late that night he got fast tracked to management, johns story has been used as an inspirational story ever since.
On a slightly similar note, once upon a time I worked as a junior sys admin for a telco with a logo which quite resembles a Death Star. I was working Chrismas Day as it offered tripple time and I had a few bills to pay.
During the night something had gone wrong with one of the telephone switches and it wasn't sending along whatever data it was meant to. As a result its own storage was being filled up with the backlog. This was made even worse by the fact that the storage was pretty small. Apparently it was possible to back up the data (to tape I assume), but the restore process was somewhat untrusted. In fact the rumour was it wouldn't work at all. This was a problem given the time of year and the amount of phone calls that would take place.
Fortunately, for me, I didn't deal with any of the telephone switch stuff, so I just had to spend most of the day answering the phone and trying to co-ordinate various things. The worst thing I had to do was call the head sys admin and ask him to fiddle with the firewall to allow some access for remote support. Luckilly he was quite merry, so I didn't get any sort of ear bashing. Eventually it was all sorted out and Boxing Day was thankfully a lot quieter.
Having effectively been 24/7 support for many years (working for very small IT departments in manufacturing companies) I've had a few late night calls.
One company used to have a regular problem where the password on the generic production user would prompt for a reset (we couldn't turn this off for a single user), of course they'd do this on a Friday and not bother to leave a note for the incoming Sunday night/Monday morning shift. I'd get home from the pub at around 11 and then have to connect and reset the password for them.
Worse was the a*****e of a manager who insisted on ringing at about 02:00 one morning. I saw who it was and ignored it working on the basis if it was important he'd leave a message. No, he tried ringing a further 3 times, without leaving a message and, needless to say, without getting an answer. Turned out the next day it was just a problem with a spreadsheet he was working on. It was important-ish, but wasn't going to stop overnight production or prevent the early despatches going out. I let it ride at the time but made it clear to the MD that a repeat performance would result in a formal complaint to him and to corporate.
There's some overnight conversion run, and a DBA and me (as sysadmin) are on call. A problem occurs, and at some point the DBA calls me for assistance. I tell him the things he needs to do (he has remote access, I don't), and after about an hour and a half of prodding the problem (whatever it was, can't recall) is sufficiently resolved that things start proceeding in an orderly manner again.
Monday morning I come in, to be roasted by a philosopher reschooled as IT twonk, and obviously ending up as a manager, for being unreachable while on call. Errr, what? I Being on the phone with the DBA for over an hour doesn't count as "unreachable" in my book. Apparently he was trying to reach me too, but there were no missed calls, no voice mails, no "someone's calling while you're on the phone" alerts, nothing. At all. He insists on having called the right number, and as I was clearly to blame for having an unright number, it meant end of contract. Well, good riddance anyway.
Try being on-call for a Hospital, where at 2am the stroppy fecker in the Pathology lab tells you people will die if you don't fix the IT problem immediately (when it's a 20 minute drive away).
When you get there and find out she's locked out her access cos she had the caps lock on! and when you bang on the lab door she complains you woke her up (lab/radiology staff have on-call bedrooms for overnight work)!
Being told patients will die if you don't fix it is far from unknown for me. Also been called to fix a VPN issue so a consultant could login to look at X-rays. He could practically see the bloody hospital. I was an hour away. Lazy bastard
Speaking as a stroppy fecker in a Pathology lab, I can assure you that as a non-patient facing service (ie not important) we have to sometimes stress that patients might die if they don't get their results.
As IT support for 40 other stroppy fecker in a Microbiology lab (being told we're Pathogy is intensely irritating, but a matter for another day), I can feel your pain. I regularly have to sort caps-lock password problems. The LIMS we use needs you to switch on caps-lock once you're logged in, unfortunately our users generally change their passwords after they've put caps on, then next time, they lock themselves out by forgetting they used caps lock the last time.
I'm going to thumbs-up you anyway, because I've been on both sides of that fence...
"Also been called to fix a VPN issue so a consultant could login to look at X-rays."
So my wife turns up for a 6:30 appointment to discover the consultant is running over an hour late and when she finally gets to see him he can't look at her X-rays because the subcontracted IT department have buggered off home at 5 p.m. leaving the VPN down and in the middle of a password change which means some people do, and some don't have their old passwords.
As a result of which, and staff holidays, her operation is going to be a month late. Real event.
Thank you for not a lot.
On a similar note, I caught someone with our expenses spreadsheet and a calculator. Adding up all his parking receipts by hand.
Even more annoying that if he'd bothered to use the other tab (that said Calculated), it even cut out the obviously difficult business of pressing the Sum button as well...
It does remind me of the time I thought I was really clever to have our backup software print out details of the overnight jobs it had run. Unfortunately I didn't realise that "Detailed results" meant a list of Every, Single, File, it had backed up. As the printer was the main one for the office, it had a paper drawer that would take at least four to five packs of paper (it was about 40cm deep). Fortunately the output tray could only handle about 10cm of paper, so when I got in the next day, I could cancel the print job and chuck all the wasted paper before my boss got in.
tl/dr don't try to print multi-megabyte text files if you can avoid it.
You know you've got a problem when your colleagues name a unit of measurement after one of your users, for the sheer size of their printouts.
This particular unit (the "Leslie") was equivalent to a four-inch pile of fan-fold line printer paper, and I used to pity the poor postgrads who had to haul this stuff away and read it.
One of my 'favorites' was getting a 3AM call from our Helpdesk. I slept through it but played back the voicemail soon after: "mumble..mumble..mumble..mumble--PLEASE TAKE CARE OF IT!" After playing the message several times while the fog cleared from my head, I finally figured out they were calling about a problem with a speaker phone in a conference room at our headquarters. At 3AM. No, there wasn't anyone in the building having a meeting then. Perhaps the best part was later discovering that they'd called my desk phone first. Um, no, I'm not going to be at my desk at 3AM local time.
In my new gig our ticket system actually shows the local time *of the person who raised it* in the main ticket interface
I don't know why I've never seen this before (granted, I've never supported anything US based before, only europe and then most of the timezones are pretty close to 'local' local time.
Yep, my on-call pager would often go off waking me from slumber...one night it read, "Dr Max call AE now! Emg code 2.". Well as a DBA working in a bank I was pretty sure it was a wrong number, so I just rolled back over to sleep hoping that someone managed to find Dr Max and he made to AE before something nasty happened!
I made an early call to Program support, way back in my MainFrame ops days, after a new program fell over. Was told, it'll be okay and the oncall went back to sleep.
0900 comes round and same oncall is barging into the ops dept, asking what he had told me to do?
Needless to say, he got a bollocking after we had to restore the entire system back, to the early hours and lose all the factories and warehouses had input since, to correct his mistake.
Got on site one morning to find a SCO box pretty well jammed solid.
Some job had gone wrong overnight and was still trying to write a humungous log file. The box had been set up without partitioning the storage much if at all so the file was in the root partition along with more or less everything else.
Simply deleting the file wouldn't work because the program was still running; until it closed its handle on the file the disk space wouldn't be released. Deleting any files that could be spared didn't help much because the space released would be filled by the elephant in the room almost immediately.
It turned out that a SCO box with a full file system ran very v.e.r.y v..e..r..y slowly, probably because the file was buffering a load of stuff and choking most of the memory so everything else was trying to run in about 4K left over and thrashing. That made running ps to find the PID to kill more or less impossible.
And the box was a couple of hundred miles away at the end of a slow modem link - not that the speed of the line had much influence.
Eventually it got arm-wrestled into submission. I like to install Unix systems with multiple partitions, especially keeping directories that might grow, such as /usr/spool or /var separate.
Simply deleting the file wouldn't work because the program was still running; until it closed its handle on the file the disk space wouldn't be released. Deleting any files that could be spared didn't help much because the space released would be filled by the elephant in the room almost immediately.
In which case the old favourite on those situations, cp /dev/null offending file, probably wouldn't have helped either.
Eventually it got arm-wrestled into submission. I like to install Unix systems with multiple partitions, especially keeping directories that might grow, such as /usr/spool or /var separate.
Always. If nothing else, at least /var should be separate. It really annoys me that many flavours these days try to "helpfully" make installation "easy" and install everything into single partition, which is just disaster waiting to happen.
Don't know if it's work on something from the mainframe days, but the old Centronics parallel ports could be tricked by poking a wire in the right holes so that the source's 'ready' is fed straight back into 'acknowledge.' The source sees a printer attached, and keeps on sending. The data is lost, of course, but it's handy if you just want to test some program that produces printer output without wasting paper.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
