I think we've all worked with a Guillermo, and I think a lot of us have wished we never had worked with them in the end.
Backup tech felt the need – the need for speed. And pastries and Tomb Raider
Wait, what? Is it that time again? Time for Who, Me? in which we invite readers to share stories of the less-brilliant moments of their workplace lives, and how they were caught out – or narrowly escaped. This week we hear once again from a past contributor we Regomized as "Guillermo" when he told us about a lesson he'd …
COMMENTS
-
-
Monday 27th February 2023 10:25 GMT Anonymous Coward
...wished we never had worked with them in the end.
Indeed ...
... relax and play Tomb Raider on his PlayStation Portable?
I once had the missfortune to have to deal with a intern of roughly the same characteristics.
Who put my job on the line more than once.
Regomized as "Guillermo"?
A definitely more fitting monker would be IR *.
* Irresponsible Asshole
-
Monday 27th February 2023 13:18 GMT Evil Auditor
Indeed. Mine was called Goofy and that was his name. At least it was what I called him. A decade after our paths separated, I had the displeasure to meet Goofy again, hoping that in the meantime he had matured. He hadn't.
The good thing is, Goofy ended up in an organisation where he fits in perfectly.
-
-
This post has been deleted by its author
-
-
-
-
Monday 27th February 2023 16:42 GMT Elongated Muskrat
Re: He, who has never cut a corner...
The risky bit here, as far as I can tell, would be to format those discs in another drive. If my memory serves me correctly, some of the DVD writers were a bit temperamental, and I don't think it's unheard of to have discs that, once written on one drive, could only be read by that same drive. Errors could creep in and not be readily picked up, for example reading first or last sectors, unless doing a full verify after writing. If that corner has been cut too (and it probably was because it would be the most time-consuming part, often done at 1x speed), then you could end up with a stack of coasters and not know it until you needed the data on them.
-
Monday 27th February 2023 18:27 GMT spuck
Re: He, who has never cut a corner...
Hmm... I'm legitimately curious about this.
This rings true for magnetic media (floppy drives and the like), but my understanding of writable optical media is that there is a factory-imprinted track that the heads have to follow. I'm trying to envision how a misalignment or difference between drives would cause a disk to work completely in one drive, but not another?
-
Monday 27th February 2023 19:27 GMT Terry 6
Re: He, who has never cut a corner...
I dunno. But in the early days of CD drives I certainly had CDs that would work perfectly in one machine, but be f---ing useless in another identical PC that a colleague was waiting at. Had it changed by the days of the DVD? by then we had a network, so I've never had to find out.
-
Tuesday 28th February 2023 11:00 GMT Elongated Muskrat
Re: He, who has never cut a corner...
I've only heard of it anecdotally, but I guess it would have something to do with the laser focus being slightly off, and thus the spots of dye that get sublimated (or however rewritable drives work) are in a slightly different position; I guess this may drift across the width of the disc as well, due to variations in stepper motors. If that "track" is imprinted by formatting the disc (not sure if this is how the DVD-RAM discs worked?) then you'd want to do it on the same drive to be sure.
edit - Having looked up how these things work, they are more like an optical hard disk than a DVD. They have "sector markers" imprinted on in the factory that provide start/stop marks, but the sectors themselves are written by formatting them, and the data is held by melting the surface and recrystallising it in one of two phases with different reflectivity. Unlike other writable / re-writeable optical media, they have concentric tracks, not one spiral track, which presumably would be more sensitive to things like laser alignment and stepper motor variation. They also use dual lasers with different power levels to write the two different phases, so focus and alignment would be important, especially if it varies between drives.
-
-
-
Monday 27th February 2023 20:59 GMT doublelayer
Re: He, who has never cut a corner...
I might have tried something like this as well, although I'd have asked for permission to test it rather than doing it in production. I can get annoyed by unnecessarily slow procedures and attempt to improve them, but the most important thing is verifying that my improvement didn't break anything and actually improved the situation before that comes to replace the old procedure. Our paths separate when it was tried in production and not verified, because this doesn't confirm that it's safe to do next time so it's useless.
-
-
-
Monday 27th February 2023 11:14 GMT dwodmots
The whole prcedure seems dumb, so I get why he tried to streamline the process. Manually burning 10 DVDs and then only checking the result for errors a week later seems like a desaster waiting to happen no matter how closely the operator followed the manual. I still remember how error prone burning data to an optical medium was. From the labeling requirement I assume that they were writing a split archive on those DVDs. They were banking on burning 10 DVDs in a row with no error and then only verifying the result a week later?
This could not have been earlier than the middle of 2006, when the first Tomb Raider for PSP was released, which means USB harddrives were readily available and easily large enough to hold 10 DVDs worth of data. Just write the whole archive on one of them in one go, no errors from lasering holes in a silver disk, no split archives, no reading all 10 DVDs one after the other for a restore.
It's Guillermo's fault for modifying the procedure by himself, but it was still a garbage tier procedure.
-
-
-
Monday 27th February 2023 13:23 GMT ChrisC
I take it you've not spent much time around production line/other production process-related PCs... If they're running a particularly business-critical bit of software, or needing to be interfaced with a business-critical bit of hardware, which requires a specific version of the OS, then that's what'll be maintained for as long as humanely possible. Because the alternative may well involve having to spend more money than anyone below CEO level is able to authorise on replacing said software/hardware.
Not that we know for sure that this particular PC was or wasn't USB-enabled mind - there may well have been another reason why DVDs were being used in favour of removable drives for the backup process.
-
Monday 27th February 2023 15:40 GMT Anonymous Coward
No thumbdrives
I'm currently supporting a group of computers that have USB ports, but only support keyboard and mouse, no thumbdrives. They're simply that old. They're running the manufacturing that is a substantial fraction of the global company's revenue. Yes, we're working on upgrading, but, as ChrisC put it, the funding requires near-CEO-level approval. Not the first time I've seen seriously-obsolete hardware running critical equipment; the Pentium 2 machines in 2007 come to mind...
-
Tuesday 28th February 2023 16:13 GMT JT_3K
Re: No thumbdrives
I can top that. The owner bought a (very specialist industry specific) machine in 2022 from a competitor that shut down. It was a messy environment and he proudly proclaimed that instead of spending £600k to get his new "[ITEM] Line" in, he'd secured the machine from the liquidators of the old company for £26k.
Muggins here turned out to know more than the "specialist company" that was brought in to install the machine about 286(? - maybe 386) DOS 5.0 networking stacks and came down to get it online, speaking to an XP VM (in an isolated, locked-down to hell and firewall protected VLAN) in SMB 1.0 so it would suck down exported CAD instruction files. It was only once it was running happily they all admitted that the place it'd come from hadn't been able to get it on the network and it'd never done it, being used with 3.5'' floppies
-
Tuesday 28th February 2023 20:01 GMT Jou (Mxyzptlk)
Re: No thumbdrives
IF it is running DOS, Windows 3.x (not NT) or Windows 95 without USB: There are generic drivers to support them.
A bit finniky for DOS, you may find various versions of those drivers on the net. You might be lucky, cause if it works you can make a good image!
For Windows 95/98 (with USB supplement, depending on the version) there are several generic USB-Storage drivers.
You could use the LAN way too, if possible. As emergency: Netcat and ftp clients exist for DOS as well, bypassing the SMB1 issue. You can even use rsync for DOS!
-
Saturday 4th March 2023 19:02 GMT Ignazio
Seconded. I helped format a computer in 2002 with windows nt, both a few years old by then. Reason was it held an ISA interface card for a hydraulic press sensor of some description. Value of the computer, about fifty euros, value of the attached machinery about one hundred thousands, replacing the isa with pci or usb probably 10 grand. So nt and hopes of the psu not burning out it was. Still is, for all I know.
-
-
Monday 27th February 2023 15:49 GMT Elongated Muskrat
CD / DVD writers that used caddies are pretty ancient tech. This might have been around 2006, but that doesn't mean the "server" (and I use the term advisedly) dates from 2006. In "factory environments", those machines probably have some sort of custom control board in them and will be kept running for as long as humanly possible, because replacing the machine might be cheap, but replacing the control board within them almost certainly is not. And you probably are talking "replace" not "move" because that control board will probably use an ISA interface, or something equally horrible, which modern MOBOs (at the time) didn't support. So - get new board from manufacturer - probably a couple of grand. Plus a new license. "What?" I hear you say, "a new license?" Yeah, a new licence, because the old board had a licence key specific to it, and the new board needs a different one. That'll be £150 K please, or some similarly ridiculously high number. Suddenly the old machine in the skip is starting to sound more appealing, if we can fix it...
This is the reality of how a lot of businesses operate, especially ones that operate custom or bespoke factory processes with their equally exotic control systems.
These days, of course, the whole lot can probably run on one Raspberry Pi Pico.
-
-
Tuesday 28th February 2023 12:21 GMT Elongated Muskrat
It supports I2C, if you need to emulate an old-fashioned parallel or serial port through that I believe it's doable. It also has plenty of GPIOs if you can't use I2C for some reason.
Granted, if you need high data-throughput that needs to go faster than the Pico's clock, then a pico might not be the solution for that...
-
-
Monday 27th February 2023 17:38 GMT Anonymous Coward
I have 2 like that.
The first uses a an ISA SCSI board, proprietary output, that connects to an external control board using 3 SCSI ribbon cables. The equipment is so old that they don't even make a replacement for it. To upgrade the Windows XP SP2 (yes SP2, it breaks if you install SP3) we have to buy an entire new system. What does this system do? It prints labels on Newspapers to mailing. The cost of a new system? Upwards of US$500K.
The second is an old Mac OS9 (running in OS8 compatibility mode) connecting to a proprietary SCSI card that acts like a printer interface for a Newspaper-Plating system.
We keep 2 backup systems for each, with the exact same hardware specs because losing either results in potential Millions of US dollars to replace.
The press itself? It is older than I am, and I'm ancient.
-
Tuesday 28th February 2023 13:03 GMT Elongated Muskrat
The stuff I used to have to work with, which controlled high-speed roll-fed industrial printers, ran on Win NT via a proprietary board (probably the same, SCSI over ISA), and also wouldn't run unless there was a physical dongle on the serial port to provide the license key. This was back in the mid '00s and there was no sign of it being upgraded any time soon (the desktop machines we did the print layout on were all XP IIRC). That may well have necessitated replacing the printer itself, which was about 6 ft tall, 4 ft wide, not including the 1+ tonne roll of paper feeding into and out of it, and ran on 3-phase 440V.
You never know, replacement costs for kit might even have been what finished off that business (it apparently folded about a year after I left). I suspect it was the horrendous mismanagement, though.
-
-
-
-
Monday 27th February 2023 18:34 GMT Jou (Mxyzptlk)
Re: Back in those days...
Windows NT had networking! But there is another way:
You you put in your DOS-Disk with either loads IPX/SPX to backup the image to a Novel server, or use TCP/IP to backup the image to a Windows server. Whether you use Ghost or Powerquest is up to you, sometimes one works better than the other. Even with a 10 MBit card it would transfer with up to 1 MByte per second in reality being much faster than a 4x-ROM burner which does 600 KB/s max. Let alone saving the time for switching the CDs. And yes: DOS could do TCP/IP and connect a windows server. And it still can connect Server 2022, though you have to dumb down that server quite a lot. But you can use NetBUI at well, if you want to.
On top: You had a disaster recovery capable backup: Plug in replacement HDD, boot from disk and restore the machine from the server.
Waaait a minute, the say "DVD" not "CD"! If it was THAT new you could boot from a CD with your backupprogram + use at least a 100 MBit card (3x905b for example). Same result: faster, better, stronger. Gigabit should have been possible!
-
Monday 27th February 2023 21:33 GMT david 12
Windows supported USB in Win95 (not well, but it was there)
And in 96-97, I deliberately specified USB hardware because of the well-known fact that USB was supported in Win95.
But by the time we moved to Win2K, we still hadn't got USB support working in Win95 or Win98.
By 2010 USB was possible, but even then we didn't consider it useful. It was so flakey we preferred RS232 where we had a choice.
-
Wednesday 1st March 2023 10:33 GMT Roopee
PCs?
And who said anything about PCs??
In 2001 I worked as a contractor on a brand new Unix system that used writable DVDs to archive data (considered cutting-edge at the time) - I was never privy to the server room, although it was all visible through a glass wall adjoining the developer office, but I’d be surprised if anything USB was used anywhere, and it would certainly have still been running 5 years later...
-
-
-
Monday 27th February 2023 13:15 GMT ChrisC
Umm no, the article notes that the procedure involved verifying each disc as it was written...
What does seem dumb however is that the article mentions having two DVD burners and three caddies, yet as written it sounds like the procedure only called for the use of a single burner to do all the work - formatting, writing and validation - which would be a clear waste of available resources. So making use of the seemingly redundant burner to format discs whilst the first burner is busy doing the write/validate work was an entirely reasonable optimisation of the process.
Where the optimisation fell apart was in failing to label and re-spindle each disc at the successful completion of the validate step - had this been done then he wouldn't have been left with that pile of unlabelled discs needing to be hurriedly scooped up. Given that he'd have had to pause his gameplaying long enough to reload the burner with the next disc anyway, taking the extra few seconds to label/respindle each disc then wouldn't have caused any real additional inconvenience, and would have overall been a time-saver compared with his plan of going back through the pile disc by disc at the end to work out which was which and label/respindle at that point.
-
Monday 27th February 2023 14:38 GMT Anonymous Coward
@dwodmots
From the article:
"Anyway, the first time through Guillermo did the backup with the boss watching like a hawk. He followed the manual "to the dot" – formatting each disc, writing each disc, verifying each disc, removing each disc and labelling each disc before repeating the process. All up it took ten discs and several hours.".
They didn't wait a week to do same.
-
Monday 27th February 2023 14:58 GMT C R Mudgeon
"Manually burning 10 DVDs and then only checking the result for errors a week later"
No. They verified each backup both at the time and *again* a week later. (See the description of Guillermo's first, by-the-book run-through.) So it wasn't, at least in that respect, a "garbage" procedure; it sounds to me like a particularly diligent one. (It may well have had other problems -- the length of the manual hints at the possibility -- but on the evidence we have, failure to verify backups wasn't among them.)
"USB harddrives were readily available and easily large enough to hold 10 DVDs worth of data"
They had about 40 weeks' worth of DVDs in the pool. That many HDDs would have been pretty expensive, I imagine, and also quite physically bulky to store and manage.
Formatting DVDs in parallel seems like a smart improvement to the procedure, but it would have been best to clear it with the boss first -- maybe there were reasons for doing it the way they did. (Given the then state of the technology, potential drive compatibility issues come to mind -- but that's just a wild guess.)
It seems to me that if Guillermo was too "busy" to label the DVDs, he might also have failed to verify them. But if so, that wouldn't have been the procedure's fault; rather, it would push his story even further into "Who, Me?" territory.
-
Monday 27th February 2023 18:02 GMT chivo243
The initial "real procedure" did check the integrity "verifying each disc,"
Anyway, the first time through Guillermo did the backup with the boss watching like a hawk. He followed the manual "to the dot" – formatting each disc, writing each disc, verifying each disc, removing each disc and labelling each disc before repeating the process. All up it took ten discs and several hours.
Our friend Bill, skipped that procedure in his abbreviated version, the boss wanted to verify later anyway, but was conveniently distracted...
-
-
Monday 27th February 2023 11:45 GMT Anonymous Coward
Some excellent training by the boss there
I can't help but re-write this episode in my head from the boss' point of view. First the new guy is forced to follow the manual to the letter to learn the right way to do it (and even check the written procedure is correct). Then the new guy is left to do it unsupervised. Clearly it's a tedious job, and they are going to cut corners, so after a few weeks the boss springs the surprise inspection ("I walked in and the lazy sod had a PSP out on the desk! Honestly!"). A few days of letting them sweat, and a sudden surprise phone-call later, and new guy, suitably chastened, does the backups properly from then on.
Nice demonstration of the boss' people-management skills if you ask me.
-
Monday 27th February 2023 21:57 GMT Anonymous Coward
Re: Some excellent training by the boss there
... excellent training by the boss there.
I beg to differ.
The new guy was not forced, he was given precise instructions to do a specific job and a manual to follow.
A correctly made back-up is always important, whether it was tedious or not was absolutely irrelevant.
Proper training by the boss would have been:
- Have someone walk him through the procedure (dry run), manual in hand.
- Make sure he understands how to do it, all of it.
- Explain and make sure he understands the importance of a proper backup vis-a-vis the consequences of a failed one
- Supervise the operation as he goes about it for the first time and correct any mistakes.
Then and only then come back a couple of days later (not weeks) to check on the result of his labours.
Having a PSP on his desk is another matter altogether, one (not at all funny) which requires a stern talking to along with a warning.
He was the guy in charge of the back-ups, not the one in charge of making the morning tea.
Excellent training indeed ...
-
-
Monday 27th February 2023 13:27 GMT ChrisC
Re: Most tasks can be condensed onto 1 page of A4
By the sounds of it, the entire description of the process could have done with some optimisation here - that initial talk of a multi-page manual, multiple burners/caddies, and 400 blank DVDs on hand made it sound like this was going to be an insanely onerous backup process, whereas once you actually read the remainder of the article it's clear that it was just a bog-standard backup requiring far less actual effort (even if following the procedure to the letter) than the description hinted at...
-
Monday 27th February 2023 18:35 GMT spuck
Re: Most tasks can be condensed onto 1 page of A4
It also sounds like this is a procedure that was built as a result of an extreme failure that the company swore up and down they would never repeat again, no matter the cost.
Someone wrote a very detailed and explicit procedure, probably because whatever they had in place before failed them in the worst way, at the worst possible time. Also sounds like the boss was there to either there to witness and be burned by the original failure, or had been the recipient of enough beatings to understand he didn't want another.
-
-
-
Monday 27th February 2023 16:59 GMT LogicGate
Re: Most tasks can be condensed onto 1 page of A4
I have written procedures that take more than 60 pages and have had users complementing me on how much easier they are to follow than other procedures found in the industry.
Mind that these are procedures that are bi-lingual, illustrated, and they go down to detailing the type and size of tool used in each step. It helps that these are one-off (non repeating) procedures, but I would definately not trust a procedural description for the same job that has been compressed to 1 page.
A procedure is a tool, and it isa matter of selecting the right tool for the job.
-
-
Monday 27th February 2023 13:34 GMT Anonymous Coward
Re: Most tasks can be condensed onto 1 page of A4
I have worked as an Auditor for many years in different organizations. There are very many reasons processes are longer than they should be and while YOU might not be one of those reasons, believe it or not, there are others.
If it weren't for the confidential nature of my job, I could entertain with a few who .. me?s myself - from an auditor's point of view.
Hmm maybe ElReg should host an alternative column titled: "You're not going to believe what they did!"
Kidding of course.
-
Monday 27th February 2023 17:17 GMT An_Old_Dog
Re: Most tasks can be condensed onto 1 page of A4
reasons processes are longer than they should be
1. The author of the procedure is a bad writer.
2. The procedure is written at a very detailed level, and presumes the reader knows next to nothing about the subject matter. I've read military electronic repair manuals like this.
-
Monday 27th February 2023 17:56 GMT EarthDog
Re: Most tasks can be condensed onto 1 page of A4
and possibly puts the procedure in context and what to do if something starts to go wrong. E.g. faced with a problem you basic Windows user will want to warm start the machine. This could be a bad thing. Instead you get a warning "DO NOT REBOOT MACHINE!" a trouble shooting procedure to follow and who to call if said troubleshooting fails.
-
Monday 27th February 2023 18:35 GMT Anonymous Coward
Re: Most tasks can be condensed onto 1 page of A4
"...and presumes the reader knows next to nothing about the subject matter..."
I've written procedures like this. Rev 1 said something like "print the document".* During training, the tech was very whiny, and wanted me to spell everything out. I said I could do so, but that means if I said to click "file", then "print", etc, then the tech *always* had to print that way. Hitting ctrl-P would also work, but wouldn't match the procedure, so they'd fail an audit. I preferred to give the minimal spec that a qualified user could execute, giving the user the necessasary freedom in details wherever possible.
*(actual example was more complicated and domIn-specific, but the same idea)
-
Wednesday 1st March 2023 10:56 GMT Roopee
Re: Most tasks can be condensed onto 1 page of A4
Agreed - too much unnecessary information is clutter, getting in the way of seeing the wood for the trees - a good procedure should specify prerequisites, including assumed knowledge and tools required.
i.e if you haven’t got the necessary knowledge or tools, you shouldn’t be doing the procedure...
-
-
Monday 27th February 2023 14:53 GMT Anonymous Coward
Re: Most tasks can be condensed onto 1 page of A4
I once had Oracle database installation "instructions" that were page after page of:
Click B
Click 2
Click next
Click 3
Click 1
Click next
Click C
Click next
No screen shots, no names of the options or even the screens. Something like 10 pages of this.
-
This post has been deleted by its author
-
-
Tuesday 28th February 2023 15:29 GMT Down not across
Re: Most tasks can be condensed onto 1 page of A4
Reponse files are finicky. If you really want to tear your hair out try the dbca response and template files. To make it extra fun most error messages will be misleading and/or confusing.
Having said that, I do agree that time would've been better spent by crafting appropriate response file rather than tedious manual (especially a manual where you're nearly guaranteed to at some point get lost as to which point you were at.
-
-
-
-
-
Monday 27th February 2023 14:29 GMT Annihilator
Re: Formatting DVDs?
DVD-RAM was a weird one. It definitely needed pre-formatting (have a look at the wiki page to see the “pattern” that identified a disc) and it was slooooow, but handily it presented itself like a normal hard drive.
I only came across them a few times (one was supplied with my early burners that supported all 5 formats - remember the -R +R -RW +RW format wars?). They were useful but from memory they don’t work particularly well in regular DVD drives.
-
-
Tuesday 28th February 2023 09:20 GMT Cian_
Re: Formatting DVDs?
DVD-RAM discs could/should be readable in any drive. I used them fairly extensively in ~2006 as I had both a TV DVD+VHS recorder that used them, giving something more like DVR functionality as it could delete shows from the disc; and a laptop that properly supported them.
Most modern drives do actually support at least reading them still - out of caddy of course; but most later DVD-RAM drives were caddyless anyway. I have used this quite recently to copy a VHS to DVD-RAM and then pull the files on my PC; and use Handbrake to to convert to something sensible.
-
Tuesday 28th February 2023 09:45 GMT Annihilator
Re: Formatting DVDs?
Yeah very expensive, to the point I only ever used the one that came free (must say, I found it recently and 20 years later it was perfectly serviceable). They *should* have been compatible with every drive, but often their reflectiveness was slightly weaker and so some drives didn't cope very well. Similar to early CDRW discs wouldn't always play accurately in players (cars were often the most temperamental).
They also remained very slow. I never saw a faster disc than 2-3x speed.
-
-
Monday 27th February 2023 18:24 GMT John Brown (no body)
Re: Formatting DVDs?
Maybe they were DVD+R discs? Also, ISTR some methods of "formatting" or preparing write-once optical disks such that you could still add to them later and/or "overwrite" files. I would guess that the backup process in the article may have used a proprietary backup solution that required something special on the discs before being used as part of a backup set. The caddy bit seems to date it a good while back in optical disc terms unless they were not actually DVDs as stated in the article. There were various more specialised optical discs in caddies that didn't go mainstream. In fact, I don't remember any standard DVDs using caddies. They pretty much disappeared right at the very early days when CDs were still running at single or double speed. I still have an old IBM SCIS 5.25" CD-ROM single speed drive that uses a caddy left over from my Amiga days. It was obsolete when I acquired it :-)
-
Tuesday 28th February 2023 17:35 GMT Peter Gathercole
Re: Formatting DVDs?
To incrementally write a one-time writeable DVD (either +R or -R), it was necessary to write it as a multi-session disk.
Each time you added a something new, it put it into a new extent, and then invalidated the old disk index, and wrote a new one. The amount of space for the indexes was finite, so there was a limited number of additions you could make to a disk. And each time you added something, the total space that could be used on the disk went down.
In this case, they're talking about DVD-RAM, which is a very different beast. The disks did indeed come with a pre-recorded timing track, and were effectively 'hard' sectored.
-
-
Monday 27th February 2023 15:50 GMT Marty McFly
Helluva RTO & RPO scheme there!
Willing to lose up to a week's worth of data, and spend hours doing so. All to avoid paying for a real enterprise backup software solution. Seems like Backup Exec & Netbackup were the top backup solutions back in the Tomb Raider days.
I would not want to be the one owning this decision should a real disaster actually occur. Kind of a resume generating event.
-
Monday 27th February 2023 18:06 GMT Grogan
As Guillermo, I would have considered it my duty to point out what a ridiculous, foolish, error prone (both human and physical) expensive, labour intensive backup strategy that was.
DVD RW media is a horrible thing to use for backups, too. Unreliable, AND good luck reading some of them on a different drive.