Typical '80s IT: Good idea leads to additional duties, without extra training or pay, and a nuked payroll system Banish those Monday blues with a recollection of IT-related foolishness in The Register's Who, Me? feature. Today's tale comes from a reader Regomised as "Tom" and concerns events that took place back in the early '80s, when The Empire Strikes Back still ruled the roost and Raiders of The Lost Ark was creeping into cinemas. …
'80s??? It happens *all the time*, at least here (a university in South Forestisburningstan). OK, in some aspects we're still in the 80's, but even so...
Our IT department is outsourced and they have a very strict contract -- basic maintenance, install/configure stuff, can do departments' and events' web pages as long as they are in PHP.
Any request besides those is returned with a "not in our contract" message. So whoever suggests anything new IT-wise gets a pat in their backs and is put in charge of "ways to implement it". Without funding, of course!
You can ask for special exceptions on the outsourced IT contract, but the process is so byzantine that it is much easier to do it yourself or just give up.
"It happens *all the time*"
Oh, yes.
Any idea one has that is considered a good idea is:
A, appropriated by a boss who claims they thought of it
B, now an expected part of one's job
C, with no increase in respect or salary
And:
D, utterly one's fault if it all goes tits in the air (said boss will suffer convenient amnesia regarding their previous claims)
I've found it's simply much more peaceful to make and refine plans in my head, whilst doing exactly nothing.
You forgot E:
"After 2 years of running the system that your boss claims to run, that you took on with no training, contract or salary increase, and that many people in the company know you bear responsibility for... you now have an absolutely bullet-proof position in the company and, if you're a decent negotiator, a better pay packet. You probably also keep your boss up at night with the constant worry that he'll be found out."
I've been there a couple of times (btw, I'm not a good negotiator, so my salary isn't great).
But one company offered to relocate my job to a different country when I told them I was moving to be with my fiancee regardless of what employment opportunities I had.
"basic maintenance, install/configure stuff, can do departments' and events' web pages as long as they are in PHP."
I hope "basic maintenance" covers backups, testing restores and doing actual restores when the need arises. Otherwise, you know where you are and you'd better start looking for the paddle.
"You can ask for special exceptions on the outsourced IT contract, but the process is so byzantine that it is much easier to do it yourself or just give up."
Be thankful that the contract holder is missing a few tricks there. At a previous employer (decades ago)
1) 'projects' extra to contract quoted for at high rates
2) any alternative implementations that required any access to network or access to data held in databases &c subject to mandatory 'impact analysis' that costs $large-amount per day as part of contract
Result was lots of 'shadow IT' involving spreadsheets on shared drives, purchase of commercial systems with stand alone Internet access and so on...
To be fair though, backups were fine.
".... OK, in some aspects we're still in the 80's, ...."
---
On the left side of the pond in Toronto, an old IT friend of mine who is supposed to be retired from COBOL programming is STILL making a mint off of Canadian and U.S. financial institutions who are STILL running 1960's era COBOL code that does nightly customer business account reconciliations and financial instruments trading payouts. While the code now runs on 2015-era IBM Z-series mainframes upgraded from their original IBM 360-series minis/mainframe hardware, the original COBOL code was so well written and is so bulletproof, the banks STILL use it after 60 years and pays my friend a princely fee all year long to maintain and update it.
In fact, this COBOL code now processes and sends financial data to JAVA based web front ends at client machines and at teller desktop machines. The IBM COBOL compiler is so bulletproof, the bank now sees it as a critical component in their entire web services system and they see no issue with running it FOR ANOTHER 50 years!
And I am hearing more and more that those 60's to 80's era COBOL programmers are being brought back by multiple Canadian and USA banking institutions to train newbie programmers because that language is so bulletproof when it comes to back-office services for banking databases and backend financial services processing. It's one of the simplest programming languages to learn AND it is self-documenting. It is almost impossible to write unreadable spaghetti code in it. For backend bulletproof database financial processing systems the IBM COBOL compiler has no equal!
Now I now WHY my friend is now an actual millionaire! Who knew that knowing 60's to 80's era COBOL code can STILL make you SERIOUS MONEY! (i.e. $200,000 US+ per year!) ???
I'm kinda envious at his income from COBOL .... !!!
V
Working in the mid 1990's with a customer who was running a retail operation on amongst other things a VAX VMS Cluster (Best we do not mention their Ultrix DECsystem monstrosity and AS/400 bastion of how much can I spend with IBM). The VAX was two VAX 3000 series with DSSI disks in a RAID configuration 0+1, or in DEC words shadowed. The time had come to upgrade the VMS to the next major release and the plan we submitted was to have a plan (Which this customer was always adverse to as they thought this just increased the costs), take backups, work the upgrade, test and then release back to the customer.
But the customer had other ideas, that had a shadow, they could split it and hey presto a ready made backup .....- Customer :: Why don't you take down one half of the cluster, break the shadowing (RAID 0+1), upgrade and then switch over the systems, then bring the old VAX back up with a merge using the disk running the new version of VMS.
Now, as a technical challenge this is interesting, but has many opportunities for something to go wrong, either process based (i.e. getting commands wrong etc when breaking the shadow disks etc), but the customer would only take this approach.
Initially everything went well, the upgrade applied to the VAX not now in the cluster, old VMS VAX shutdown and then brought back into the new cluster running new version of VMS - There is a moment when you create a new shadow disk in VMS that was always a bit of a fingers crossed moment when it did the initial creation of the shadow pair, 99 times out of 100 no issues but of course this time an issue, there was some issue on the shadow disk creation and we had a fail, but also took down both disks, disks themselves OK, but some gremlin caused a software fail. The issue is that we only have the regular backup for restore, so all the changes in the system prior to upgrade where effectively lost and we had lost the upgrade and all associated effort.
We have covered ourselves with respect to the customer in that we had formally written that this was not an approach we recommended as we could not guarantee the data or upgrade - But as ever lots of noise from customer that this was our fault as it should have worked blah blah blah.
But lesson was learnt for 1-2 years that nothing beats have a project plan that also ensures you take backups before committing to a change and that RAID 0+1 (VMS Shadowing) is great but should not be used as a short cut for VMS upgrades in the future.
If the customer was so sure it should have worked, the customer could have done the himself.
There are times when you should say "no" to a contract, and when a customer is clearly cutting corners and not doing things properly is one of those times because, contract clauses be damned, it will always be your fault if something goes wrong.
And something always does.
From analogous experience you then not only get flack from the customer, but you also get "friendly fire" from your own sales droids and manglement for missing out on some dollars.
Even when it's categorically proven from experience that the net income of (profit from contract - expenditure to try the work and then clean up the mess) is negative.
Been there, done that, told the sales droids that if they ever sell something like that ever again (especially after committing me and my team without even consulting me about it) then they're the ones who will support it and I will refuse to have anything to do with it...
Been there, done that, told the sales droids that if they ever sell something like that ever again (especially after committing me and my team without even consulting me about it) then they're the ones who will support it and I will refuse to have anything to do with it...
In the end you will have to do it anyway, so that is an empty threat. However, telling them that all additional expenses will be clawed back completely from their bonuses and salary does have a tendency to get their attention, especially if you put it in writing and add it as an addendum to their contract.
I walked away once.
I was contracted to clean up after they fired (for cause) their current IT head. I took a look at what they had and his undocumented fingerprints all over everything. (He thought I was to be his new assistant.) I told the CEO that they were going to have to shut down the office for a few days, while everything was reinstalled. I was told flatly, NO. I explained why the NEEDED to be done. NO. I said, I will not be responsible for what happens, and left. I called the contracting agency I was working through, and explained the situation, and what needed to be done. They agreed with me. Asked me to write it up for them, which I did. I got paid.
I heard the company had a big data breach about a year later. Not sure if related or not.
Sometimes walking away is the only option.
I happen to LOOOOOOOVE VAX VMS and VAX Clusters (we're STILL running a VAX-9000 super for specialty financial instruments remote 128-kbit ISDN modem based financial operations after 30 years!) but I do understand the issues with disk shadowing and what THEY at DEC called mirroring which was a stop-gap solution to a common late 1980's Early 1990's backup problem of having disks that were just too small to hold all they data we were making daily.
We got around all that horrendous unusable backup crap by running some high-speed optical serial cables to multiple SUN workstations which had these GIANT Sony tape drives attached which THEN did the backups in a parallel manner. The Sony tape drives were basically a customized BETACAM SP tape machine (at $65,000 US EACH!) that had a digital encoder which wrote bits and bytes into the video signal portion of a video recording deck!
Worked GREAT for over 10 years until Exabyte tapes came out along with Sun SuperSPARC servers and the VAX 9000s were then moved into a financial trading role!
V
40 years ago I joined fresh from University, and was in the "build group" where we took the developers source, compiled it and made it available to test.
(Part of my job was to take the listings, and file them in the shelves - 6 ft high - 4 ft wide!) This was a one disk DOS/VS operation system. We had the live system disk, and the build disk. The theory being you build into the build disk, and switch with the live disk.
My second week - I followed the instructions but managed to build into the live system. This meant that test were without a system for about 3 days. My boss protected me, and said dont do it again. They updated the instructions to make it fool proof.
The next week, I event more carefully followed the instructions, and managed to do exactly the same thing, so test were without a system for another 3 days.
At the incident review, the team said the wording could have been read two ways - and mine was the wrong way!
I got moved to a different project where I could do less damage. The grad who replaced me did the same as I did about a month later. This time they changed the process so the production disk was read only.
Yep, back in the time of floppy discs used to make 3 backups of our work files on a Friday. Straight copying in them days and only took a few minutes. Rotating 3 sets of discs so that the b/u would be over an earlier one. One day I copied the bloody things the wrong way. I think I got distracted. Luckily it was the second copy. So I very carefully and with everything crossed, restored from the first one and started again.
Oh btw. Just this very day I restored an image to my home PC which had had a hissy fit for some reason. I'd been running System Restore, which fell over and took out a fully working version of Windows, instead of restoring things that needed to be restored. (My Thunderbird set up. I'd been trying to go back a version and it mucked everything up. It's like they plant depth charges to stop you doing that). The Win10 version of a BSOD. (The one that reassuringly tells you it can't start Windows so it will switch off the computer instead). Grabbed my Macrium boot disk and went to the most recent of the various images I keep stored in a partition on a second internal HDD for just that purpose*. So at least I know that works.
Now I think I should make another boot disc too. Can't be too careful.
------------------------------------------------------------------------------------------------------------------------
*Also on a USB external HDD. And on another external USB HDD that I swap with every couple of weeks.
Omitting some details to protect the innocent and guilty alike...
A facility I once worked at had a large machine that used a great deal of steam. There was a PM done by my department that involved disconnecting a controller (#1) to verify that the backup controller (#2) took over, then reconnecting #1 and disconnecting #2, to verify it switched back properly. New worker followed the instructions to the letter, and the switchovers happened flawlessly, as usual. Worker then finished his paperwork and walked away (as expected).
Shortly thereafter, the operator heard a peculiar noise, recognized it, and got everyone clear of the machine - which promptly dumped a huge amount of steam right where someone had been standing. Thanks to the experienced operator, no one was hurt.
Turns out the controllers forget their setpoint when they get disconnected, and so when they are reconnected they default to zero, definitely the wrong value here. Previous workers had always reset the setpoint, but the new guy didn't know to do so, as it wasn't in the procedure.
Good boss, though - as it wasn't documented, the new guy was held totally faultless. (And the procedure was updated.)
New HP removable disk caddies (gen9 and 10) have a big red light on the eject button, which is clearly an error light of some sort. So when I had to go swap a disk (well, SSD) in a RAID1 array, I assumed that the red light indicated the failed disk.
It turns out that the red light actually means "Do NOT remove this disk".
Oops.
Computer commands are often ambiguous.
"Copy X Y" (or variatons thereof) for example. Could mean "copy X to Y" or "copy X from Y"
No matter how carefully I have checked that I put the correct device name in the command line, there's always a stressful moment as I hit the return key on a format command ...
"Computer commands are often ambiguous."
Not just computer commands. My first vacation job was at British Aerospace, in the Systems Reliability section, collating statistics on failures in aircraft. (Everything is collated from light bulb failure to total loss of aircraft.)
Well, one incident report explained that the service manual for a fuel valve stated:
"Remove the valve. If it is worn, replace it."
So the engineer replaced it, right back in where it came from.
The aircraft crashed.
Manual amended to state:
"Remove the valve. If it is worn, replace it with a new valve."
I remember reading something similar about railway maintenance. I think it was related to the brake. There was a choice to perform task one or task two and after that there was a final stage to do in both cases. The indentation of the printed manual was such that the final task looked to belong only to case two. Hence accident.
I don't remember the details, but that came about in a discussion on the advantages of a structured text editing (LaTeX) vs. a WYSIWYG.
When writing Technical manuals, we had a rule that "Replace" meant fit a new one, and "Refit" meant put the old one back in. We had this (and other similar rules) explained in the Foreword of every manual we produced, so if anyone cocked it up, we could point to the Foreword and say "RTFM" very loudly.
At the incident review, the team said the wording could have been read two ways - and mine was the wrong way!
Given a 50:50 chance or choice, in such situations you'll always lose...
That's a statistical law even stronger than million-to-one chances coming up nine times out of ten.
That's because unseen USB ports are always in a quantum superposition of states.
The first attempt to install the connector collapses the wave function to the orientation of the cable, but too late to allow the connector to work. Then you flip it and try the other orientation which of course does not work, then you flip it again and it works.
...still nightmares about that phase when DAT tapes could only be read by exactly the same unit that created them. It made it so difficult to call them useful for DR, apart from one or two rather more short-term and hard-discovered events. Maybe that was true of all DAT units, I don't know, as DAT was not the way (aha aha) I liked it (aha aha)
Ahhh - not so fond memories. Where I worked the demarcation of duties was very strict. I had a piece of code that basically did:
1) Write a header
2) Write the data
3) hand over to a DBA piece of code to write a database backup
4) Write a trailer
My steps were 1,2, and 4 - and I did these specifying a no-rewind device (except for step 4 where I'd rewind the tape to unload it). Unfortunately, one of the DBAs responsible for step 3 didn't realise the difference between a rewind and no rewind device and changed it one day (without telling anyone) - and guess which one he picked.
Roll on 4 weeks later, I needed to do a recovery, and all I get is the trailer file....
I came out squeaky clean, as I had pointed out the problems with doing it this way in the first place, and it wasn't any change I'd done....
8mm tapes and VHS also had this issue.
Logical. However, for some reason my experience with EXB-8505 was that they were rock solid and I never had issues. Even XL was pretty reliable and think I only ever had one or two slight issues. I did run cleaning tapes pretty religiously.
Oh my, that brings back memories. I was working as a consultant in a major insurance company, on an on-call basis.
One day, I get called to go modify something in the mail template that defines what every mailbox is supposed to look like and what features it is supposed to have. So I pack my laptop and off I go. When I'm settled at my desk and after the meeting with the IT manager, with all the technical details I need in mind, I log into my local account and ask the system to start up the Designer on the mail template.
The Designer was a no show.
Not that the Designer had a problem, it was the template that was not accessible. It's design had been locked.
After a brief but intense moment of WTF! and deep soul-searching, I reassured myself that I would never have been stupid enough to lock the design of the most important template the customer had, so I went back to the IT manager and reported the problem. His matter-of-fact reply was simple : get the backup copy.
Like every responsible IT shop in any major company, backups were made incremental every day, full every week-end and end-of-month. So finding a good backup should be simple, right ?
Well, in a word, no. I basically spent a day with the systems team, going back every further in time to try and find a copy that hadn't been locked. When we had gone over the two months of backup that were stored locally, my new friend turned to me and said "Okay, this is all I've got here. Do you want me to go to the archives and fish out the storage tapes of the previous months ?". I could clearly see that that was not a prospect that he particularly relished, and I had already spent too much time on this issue, so I declined with thanks and left him relieved to be able to finally take care of his normal duties.
But I still had a problem : I had a template to rebuild. Or find a copy of, somewhere.
I will spare you the details, but let me just say that I finally did find a valid copy of the template on a server which, ironically, it never should have been put. I was able to make the requested changes, copy the unlocked template to production servers, and keep a local copy in my local profile - just in case someone else got the same stupid idea.
And that's how a 30-minute job was invoiced 10 hours and paid in full without any discussion.
Nobody ever told me who had locked the design.
"Not that the Designer had a problem, it was the template that was not accessible. It's design had been locked"
Are those templates not just some config file?
Could you not have duplicated or copied the locked design into a new unlocked design, edited that and then, once the customer was happy, lock that new design?
did you ever go back and do any further amendments or do you think they got someone else in?
No, the template is not a text file, it is a database. The design was locked, meaning that it was no longer readable, so it was not possible to copy it into anything.
And yes, I am still working with that customer. The IT director trusts me and there really isn't any other consultant with 25 of years experience in that particular field that they can call on.
Could you not find a mailbox with an unlocked design and created a document-free template copy from that? Ok, not possible if they were all created from the design-locked template.
I remember being told that, with that particular software (you've mentioned your specialism before), there was supposedly a particular location in the database file header that you could flip in a hex editor from 1 to 0 that would unlock the design... never found it.
I ran across that particular tidbit as well, and I tried it. It doesn't actually work, but there's a caveat : if someone with access to the design locked the database, then updated the design without locking it again, then flipping that bit would indeed give you access to the unlocked overwrite.
But if the database was managed properly, then there is no unlocked code to read and the manipulation is useless.
You see, when you lock a Lotus Notes database, you compile the code and get rid of the source. When you work in an unlocked database, you have the source with the compiled code.
this is the flaw with backups, they back things up as is at a point in time. With more and more things getting saved and then backed up but rarely accessed you get this issue more and more. Its not apparent there is an issue and once your backup retention period has expired all you'll have is backups of of the issue!
"I finally did find a valid copy of the template on a server which, ironically, it never should have been put."
The best place to look. If it shouldn't be on there it's the place where it's least likely to come to any harm.
Always test backups, as our ArcServe software said it had performed backups correctly but when needed after a server went ill, would fall over in a heap with anything older than 3 days working. Ended up rebooting the ill server when it blue screened and just hoping I could copy everything across and restore anything found corrupt from the older backup.
Turned out to be rust on the IBM Servers Backplane, only affected that one server to.
"Always test backups"
I was distantly involved in an update in a business who went from a bespoke DOS system to using MS Office (Win95?), with their stock in the... what was it called, Access?
They were religious about backing up. Every day they inserted a floppy disc (colour coded for the day of the week) and ran the backup batch file that had been written to copy the important files to disc.
Fast forward a year or two, a harddisc failure took the machine out. No worries, they got a new harddisc, reinstalled Windows, installed the Office suite.
It was about then that they discovered that their backup was essentially useless. In order to keep the screen "tidy", echo was turned off and all output sent to null. The script failed to copy several megabytes of data to a single disc, instead only copying across some of the index and meta files.
Customer data? Gone. Stock inventory? Gone. Traceability of who was renting what? Gone. Details of payments made and owed? Gone. Scans of all sorts of things? Gone.
The company basically had to start from scratch using the small amount of paperwork they had (they believed in shredding, too). I think they hobbled along for a few months before throwing in the towel.
I once, as an external consultant, visited a client whose backups were kept on tape, on top of the server, in the top floor server room, which got hot over the weekends as the air conditioning was turned off. It was a 'secure' site, so the combination lock on the door was how to gain entry, only they'd lost the key used to open and change it when the previous incumbents left a few years previously.
They also stored the staff record files in a filing cabinet, whose key was kept in a desk drawer, which was locked and the drawer key 'hidden' somewhere outside office hours, despite having HMG cabinets with Manifoil locks. (At least they didn't keep it in the server room.)
The management were strangely muted on receiving my report, and did not invite me for a drink afterwards :o(
We had what we thought was a robust process of visually checking the backups had run every day, by looking at the filename (which contained a timestamp) and the size of the file. We also did test restores every week, of a random three out of about 100 databases.
However, when the inevitable shit happened, and the database server crashed due to a power outage, we found that one of the databases which hadn't been written to for ages, was running entirely from the cached copy in memory, and the actual file on disk was corrupt. Our backups had been dutifully backing up the corrupted copy for over a month, with no outward sign.
A good database backup backs up the actual database, not what's on disk. If the database is running there's always stuff in memory that hasn't been flushed to disk yet, at least there is if it's anyway busy. The database S/W should have its own backup process to take care of that.
One of my customers called up to say that he had accidentally dropped the parts table and could I recover it for him, please ... I asked "Where is the backup?" and his response was "What is that?"
3 days it took me to rebuild the parts table and that meant they could not sell anything in that time.
At the end of this, he said it was my fault that he didn't have a backup because he was the customer and we should have told him that he needed to backup the data ...
Anon because he is still a customer.
I'm no spring chicken myself but was still at school in the early eighties. My IT career took off in the late nineties and my claim to destructive fame was to newfs the /usr rather than the /data disk. /dev/dsk/c0t0d0s4 rather than /dev/dsk/c1t0d0s4.
What stories will this column have in twenty years by today's crop of newbies?
What stories will this column have in twenty years by today's crop of newbies?
"This one time my boss asked me to do something so I called the offshore team that actually does the work, and they told me the following story about they eventually solved it"
In an environment where everyone was running Solaris on Sun workstations and we only had a few ( like 3 or 4 ) Pc's running windows 95.
We need to create backups of the data ( mainly sourcecode) on the PC , so a Samba share is opened on the big file server.
One day we discover that all files coming from the PC are stored as ROOT. What the hell ? IT panics , changes root passwords. no solution. New incoming files still appear as root..
What has happened is that, on the windows95 pc someone had just created a user called 'root' with password 'root'. When connecting to the samba volume that machine asked for credentials and went : oh you are root ? ok then ....
With a few mouse clicks i could move entire directories , including the operating system itself .
Took a while to get that hole plugged.
PDP-11/23 with 2 disks, 1 removable (user disk), 1 fixed (system disk); no other storage devices
Stand-alone backup process:
replace user disk with backup disk A
copy system disk onto A (overwrite A)
replace A with user disk
copy user disk onto system disk (overwrite system disk)
replace user disk with backup disk B
copy system disk onto B (overwrite B)
replace B with A
copy A onto system disk (overwrite system disk)
replace A with user disk
What could go wrong ?
I worked for a major IT company, and one of my friends there had a nice new 'Laptop computer. He had an application on the Laptop which was necessary for his work, and eventually as these things do, it stopped working. The company had recently split suppliers for maintenance of all IT resources, so said friend called the 'Help' desk.
010 IT Support politely explained that their contract was only to support applications on desktop PCs, he needed 'Laptop' support.
020 'Laptop' support politely explained their contract only supported the laptops' hardware and OS, he needed IT support.
030 GOTO 010
"Is there anyone who has never had the bowel-loosening sensation that arises from the discovery that a backup was not all you hoped it might be?"
As far as I remember, I have never had that experience.
Yep, that's right, and it's taken massive quantities of alcohol, combined with PTSD induced memory repression to get there.
The flat ones were surprisingly durable if you took care of them, the 3.5 aberrations? They were so faulty that in the last few years I was forced to use them I had to backup the same files in three different floppy disks and do them as Rar compressed files with a recovery file included.
Some batches or brands of disks were decidedly dodgy. Can just take one bad disk to ruin someone's day.
There was also the problem of some people just not looking after them. There were occasions I'd been called to help with a problem floppy disk (3 1/2 or 5 1/4), only to find it covered in score marks like an ice-skater had gone round them in circles, obvious signs of grit or debris on the surface that had been dragged around by the head. Where's your backup disk? ... next to their ash-tray with a coffee cup perched on the top of it. Sigh.
Reminds me of the time I used to work for a private school, the stdents were supposed to keep their floppy disks (3.5" variant) in a special protective holder - the school shop even sold these but inevitably these got lost, damaged or even just forgotten about because they were 'inconvenient'.
As such it meant that very often the disks just got slung into bags and the metal shield would start to bend. Eventually these got to such a state that the disks would go into a drive but then the shield wuld spring out and prevent the disk from being removed.
This didn't stop the students as they had to either get to the next lesson or go out because it was breaktime - the disks were generally forceably removed leaving the shield in the drive (plus they didn't want to leave any evidence of teh fact they hadn't been using a protective cover as that would have had to be explained to the head). Guess what, next lesson that computer couldn't be used and guess who got a call to fix said computer (along with a complaint that little so-and-so couldn't save their lessons work).
Got very adept at removing said shields with 2 expansion blankers to the extent that I could even do it with the original disk in there - several culprits were actually charged for the price of a replacement drive as a result.
I'd like to be all nostalgic and say they were happier times but they weren't, the school was a git to work for, the teachers treated all anciliary staff as if they were something that had come in on the sole of their shoes and I couldn't wait to get out of there.
I had a colleague who told me that 3.5s would never take over from 5.25s "because they weren't reliable enough" - I pointed out that at least people wouldn't staple them to their covering letters - Then I showed him what I thought the real killer of 5.25s was likely to be: I put a 3.5 in my shirt pocket...
The worst nightmare I ever ran into regarding backups was one which I had not been responsible for doing (which covers 98% of backup problems, I imagine, as to which you good fellows can attest).
I got dispatched to the home office of a restaurant chain here in the Southeast with complaints of "It won't work and we tried everything." I was suspicious that I was walking into a huge mess just from talking to the lady for 2 minutes on the phone but my boss had every confidence that I could dispatch with their problem in no time at all.
So I arrive. Their "server" was an aged 486 (which, admittedly, was perfectly capable of being a server back then) stuck under the lady's desk upstairs and their "backups" were done "automatically" by the system via writing to a DVD drive. All the quote marks indicate things that were not that they were named, naturally.
The problems: They told me it ran Windows something-or-another but it was Unix, it was crashed so hard the logs were not readable (HDD had only tipped 96% over, not dead as in really dead yet) so I had no idea where to even start with a fix, and thinking maybe this backup, which was run "every day" she told me, could help me with at least a start, after replacing the drive
They were re-writeable, there was 7 of them, rotated daily. I pulled the DVDs and put them one at a time in my laptop's drive and each was _completely_ blank. So I had to be the bearer of that most horrible news "Your data is not only gone, it's gone for good." Some other local tech vendor had charged them a princely sum for this useless "backup system".
Bright side, the owner, who had shown up during all this placed the blame 100% on them and not me (I was shocked at that part) and even though they didn't use us for their IT needs later, we did handle her and her husband's systems at home which paid better than that nightmare at the home office ever would have.
solving RS232 problems(oh god noooo)....
Unit gets broken into, and some of the PCs get swiped.....boss is all in a flap, junior boss is in a flap, admin are in a flap, sales are in a flap with variations on "OMG my data my data" all around.
Boss cant believe my calmness and generally relaxed attitude...... "well they did'nt steal the production PCs" (covered in oil, fingerprints, coffee stains and unmentionables) "They're the most important part of your company since they contain over 10 yrs of various programs saved at a rate of about 3 a day"
Their Boss believes in backing everything thing up off site now ..
I once worked for a printing company who decided that the mainframe backups should be kept in a fire proof safe at the other end of the large site.
A second hand fire proof safe was purchased and installed in the warehouse on a wooden pallet (so it could be easily moved if required) surrounded by millions of copies of the coming years calendars and other paper and cardboard items.
I could never work out if the IT Manager simply didn't take backup storage seriously or he wasn't right in the head. As far I recall there was never any test restores either so option two might be correct in his case.
sounds the kind of mentality we had from a former head of operations at a lab I worked at. We were going to build a new DC, there were two locations in the running for it, (really only one location was suitable).
1. Next to the old server room, perfect, had access to risers for power, cooling and networking, etc
2. Next to a messocosum lab that contained about half a dozen very large tanks (several thousand l of seawater in each), a generally very wet and salty atmosphere.
6 DC installers were asked to tender only two even did a design for site 2 and both said DON'T put it there.
Guess which site this clown preferred! But the twat still persisted. Only due to planning issues around noise with the AC compressor units was the location ruled out! (there location would have been close to a residential area so a full noise survey would be needed) The prick also suggested we could use the roof space of the building, until we pointed out it was quite open to the outside in places full of ducting, wire's etc, filthy dirty and the lift didn't go up that far!
The bloke was one of the most odious c**ts I've ever met in my life
The main problem with payroll problems is that it's the staff that get most disadvantaged by them. I remember as the newest person there getting pulled out of a work Christmas dinner to talk to a client with a payroll problem. Got back just as everybody else was finishing eating, and still had to make a 250 mile round trip on Boxing Day, when I was supposed to be off and spending time with my family, in order to sort out whatever had gone wrong. The client told us, in no uncertain terms, that if they didn't pay their several hundred minimum-wage care home staff on time, they wouldn't still have most of them the next day... Not sure I ever got thanked, or recompensed, by work for that, though the customer was fairly happy I recall.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
