Reply to post: Almost 30 years in IT

Almost 30 years in IT

I've been to lessee now, 22 DRP exercises.

Other than two particularly outrageous DB backups (VeritasNB/Rman) that used I think 6 or 8 drives on BU and only 2 on restore -- (if you've been there you'll get it) I can't think of a case where we had *everything* coming back from tape fail. I've seen Day1 runs involve nuking and rebuilding the NB master more than once, but this was inevitably a case of the Docs needing updating.. (guess why we did the exercises?)

I've just dealt with executing a move from HSM to permanent dirt cheap disk for an archive solution. It wasn't what I suggested at the beginning but it will suffice. Some of our HSM tapes had been around for 15 years. Of ummm between 2200 and 2500 odd tapes, some 38 of them did not contain the data that was originally written to them. Of those 38, 24 were overwritten in a physical server migration about 8 years ago (eye-dee Ten T errors). The remaining 14 had bitwise errors in block headers -- sometimes right at the beginning of the tape, sometimes way the heck into the tape, of *those* 14 I was able to recover 3 of them with manually correcting the one bitwise error, and on 4 more I was able to skip the dead file by changing the pointer to the next block. On the balance of the tapes I would have had to spend more than 7 days walking the data to attempt a recovery.

Tape's 3 biggest issues are

1) environmental affect. One of the reasons we likely had the bitwise error issue in so many tapes was that in the time frame when these were written we had raised floor tiles that were shown to be shedding microscopic EM active particles to the air when they were pulled and placed. These particles were found in all areas of the DC, including on tape surfaces. We had a *LOT* of tape read errors for a year or two until all work was completed. We just made more copies.

2) Hardware lifecycle. In the 13 years I was in OS/Platform support we went from LTO to LTO5. Most of the drives claimed at least one generation of backward support, but to be honest we've often found it advantageous to keep the drives == tapes, mostly for reliability. And rolled through *countless* updates to the backup software itself.

3) One of the projects that was wrapping up when I started in Platform/OS support was a 4 year exercise of migrating from WORM disks to tapes. I know, precisely, where in the primary hardware warehouse our two remaining WORM drives are stored. I *seriously* doubt that anyone else does. Same can and will apply to our LTOx drives some time in the future.

If you've got a tape solution that has never restored data for you, your *BACKUPS* are broken and you need *serious* assistance with your hardware and software. Someone with experience is recommended. Not someone off the street with a CS degree. I can admit that I've seen horrifying reasons why a backup/restore will not work, from misconfigured NICs (MTU settings at source/switch/path/switch/dest) to assumptions being the mother of all fuckups (overnight DB tables complete drop and reload during an RMAN full!!!), to just plain not understanding (backup over the application network causing app to 'Go Down' in the middle of the busy period) basic best practices, to only grabbing two tapes out of an 8 tape backup and expecting the restore to work.....

Personally? I *DESPISE* tape. Hate it with a passion. But I know my tools and can get it working when needed. It will *always* be the cheapest, most effective way to hold data for long term archival storage. It *WILL* have problems beyond X years in every case, physics being physics, and Drives will be an issue Y years out. Migrating the data with the physical tape lifecycle HAS TO BE WRITTEN INTO THE PROJECT. LTO4/5 tapes when we first saw them had 10 year lifecycles. My life could have been much easier lately....

/rant

Sorry - - just finished writing up the audit responses.