back to article Housekeeping and kernel upgrades do not always make for happy bedfellows

You can't hurry Linux kernel upgrades, as The Supremes never sang and a Register reader discovers in today's episode of Who, Me? Our story, from a reader Regomised as "Aapt", takes us back to the days when the DEC Alpha reigned, er, supreme over computer labs and anyone with even a passing knowledge of Linux could fall all too …

  1. A.P. Veening Silver badge

    Lucky escape without any real problems (except for probably an UUR experience).

    1. Joe W Silver badge

      "let's just get rid of the stupid hidden files, shan't we?"

      # rm -rf .*

      yes, the promt was # (me doing this in the /root folder as root)

      not $

      oops.

      1. Paul Crawford Silver badge

        Once did something similar with:

        chmod -R <settings> .*

        In my home directory. Thankfully not as root/sudo because I discovered that '..' matches the wildcard and it went up a level and tried to descend in to the other user's home directories.

        But for the grace of $DEITY go I...

        1. This post has been deleted by its author

        2. Rufus McDufus

          Yes, I did this once too. Early version of Solaris I think back in the mid 90s. I did it as root. The damage was impressive.

          1. yetanotheraoc Silver badge

            Says it all

            "The damage was impressive." - Could be applicable in so many walks of life. I love the passive tense too, so perfect for a "who me?" moment.

      2. This post has been deleted by its author

      3. ilithium

        What does a Cyberman say?

        I remember doing that too, back in't day. Spent all day building a FreeBSD server for our colo.

        Set it all up, got it working, really tired. Did "sudo rm -rf .*".

        Was amazed to find everything [that was in memory] still running.

        Or that Friday at 4pm after a really long week... closing down my workstation, in MySQL Workbench (remember that?). Accidentally click "DELETE DATABASE". Too tired and not paying attention, I clicked YES on "ARE YOU SURE?" and "ARE YOU REALLY SURE?".

        Got half way to the door before the boss pipes up: "Where's that relaunch for the big client I just won gone?"

        Oops. MySQL binary log to the rescue. And adrenaline.

  2. KittenHuffer Silver badge
    Alert

    The secret to intelligent tinkering ....

    .... is to keep all the bits!

    For computers this means making sure that you've got backups of the bits you're just about to screw with!

    1. Mongrel

      Re: The secret to intelligent tinkering ....

      I think it takes 'an event' to make most people think seriously about the question "What's the worst that can happen?" before they hit the button.

      1. Dave K

        Re: The secret to intelligent tinkering ....

        Yep, it's the curse of the young and over-confident sysadmin. Nothing like a trashed file system or major outage to bring someone down to earth with a bump. I was fortunate when much younger in that I witnessed someone else at my first place of work cock things up and bring down half the network. Seeing the panic unfold around the place as people tried to coax an entire rack of servers back to life taught me an important lesson about backups and testing before doing anything to a live environment...

        1. cookieMonster Silver badge
          Facepalm

          Re: The secret to intelligent tinkering ....

          I was nerd as a child and had learned all about these kind of fubars as a young lad, by the time I actually started working in the field I had MANY years of experiencing “wtf have I just done??”

          Never did it on the job, but I was an expert in reinstalling Linux and backup strategies by the time I was 18 :-/

      2. big_D Silver badge

        Re: The secret to intelligent tinkering ....

        I was "lucky", I saw it by other people first.

        I tend to have the problem that I have so many copies of the bits, I have to shuffle through and find the "latest" copy.

        I currently have Carbonine for continual backup, a cloud drive for sync (sync is not backup!), rsync from SSD to spinning rust, rsync from spinning rust to NAS and manual backups from NAS to an external drive.

      3. Shadow Systems

        Re: The secret to intelligent tinkering ....

        At Mongrel, re: "I think it takes 'an event' to make most people think seriously about the question "What's the worst that can happen?" before they hit the button."

        You just made me hear Dexter's sister asking "What does THIS button do?", Dexter shouting "Don't push that button!", the button being pushed, a female computer voice saying "Thank you for pressing the self destruct button. Goodbye.", and Gir squealing happily "FINE-LEE! WHEEeeeeee!"

        Please enjoy a pint while I try to stop laughing. =-D

        1. Doctor Syntax Silver badge

          Re: The secret to intelligent tinkering ....

          Draw all visitors' attention to axe prominently labelled "For use in emergencies."

          When inevitably asked "Is that for breaking out?" tell them it's for removing any digit that presses a button it's not authorised to press.

        2. TheWeetabix

          Re: The secret to intelligent tinkering ....

          With some of the very youngest sysadmins, I'm given to understand it's usually a lever, not a button that is the wrong one. Some kind of empirical research.

          I'll show myself out now.

      4. Rufus McDufus

        Re: The secret to intelligent tinkering ....

        I often think the sign of a real expert is not just avoiding the cock-up, but also knowing what to do if the very worse happens.

        1. G.Y.

          Re: The secret to intelligent tinkering ....

          " a clever man gets out of trouble which, if he were wise, he wouldn't have gotten into"

          1. Terry 6 Silver badge

            Re: The secret to intelligent tinkering ....

            Which is fair enough, 'cept beyond what I learnt at High School ( the only decent thing I learnt at North Manchester and it wasn't even part of the formal curriculum! ) and a limited amount of training from time to time, like most school staff I was mostly self-taught. So mostly I found out how to do stuff by trial and error, (I wonder what would happen if........).

            For me it's a good way to learn.

          2. TheWeetabix

            Re: The secret to intelligent tinkering ....

            "experience is the knowledge you needed just a moment earlier than you learned it."

      5. TDog

        Re: The secret to intelligent tinkering ....

        The most terrifying event is discovering the answer to the question, "What's the worst that can happen now?". Twice.

      6. TheWeetabix

        Re: The secret to intelligent tinkering ....

        "An"? You must be smarter than me sir, it's frequently taken me a couple of events before the nail gets driven fully into the skull.

    2. John Brown (no body) Silver badge
      Joke

      Re: The secret to intelligent tinkering ....

      ".... is to keep all the bits!"

      That's what the bitbucket is for! The only hard part is getting the bits back in place in the right order :-)

  3. GlenP Silver badge
    Alert

    The early Apricot PCs weren't IBM compatible, one of the differences being they used A: for the HD.

    Doing some work on a customer's machine after I'd been using the compatible machine back at the office.

    Insert floppy

    A:

    DEL *.*

    S**t!

    ^C

    Search for Norton Utilities disk, UNDELETE...

    Fortunately the time between typing DEL,,, and ^C wasn't too long and I knew enough about the structure of the system to be able to figure out the first character of the filenames after UNDELETE had done its best,

    That's when I learnt always do DIR before DEL

    1. don't you hate it when you lose your account

      Thanks for the memories

      Had a similar experience with the apricot kit, think I had a mental block from that job due to the pain (was also my first experience of MS Basic)

    2. John Brown (no body) Silver badge

      A: should have put you in the root directory IIRC so there should be little more than config.sys, autoexec.bat and command.com there. The rest should be directories which DEL *.* would not touch, especially back then. But not everyone was that tidy in storing their files.

      Having said that, my first experience with MS-DOS involved ONE PC that used A: for floppy and C: for HDD (also not IBM compatible), as well as a fleet of Apricots, only some of which had HDDs fitted.

      1. jake Silver badge

        "A: should have put you in the root directory IIRC"

        Under DOS, switching drives puts you into the currently open directory on that drive, not necessarily the root directory.

  4. Ian Entwistle

    One thing i instill in all the juniors I mentor is always always be in control of your own destiny when making a change on production systems, never rely on the infra backups being fit for purpose, never assume anything you haven't seen with your own eyes or done with your own fingers is there, even then be sure that you have copied what you think you have. Being able at very worst to just be able to put everything back how it was and walk away for another day is the key to a happy techie.

    1. ColinPa

      Delete is written rename

      I learned that delete is a two stage operation

      1) rename the file

      2) next week after the change has been running for a week, delete the file.

      1. Ian Entwistle

        Re: Delete is written rename

        :) but what is that poor techie 5 years down the line going to do if you haven't left that ancient config file lying around that has the password in for the DB that everyone has forgotten. ;)

        1. Loyal Commenter Silver badge

          Re: Delete is written rename

          If you have passwords (or any type of credentials) in config files, you are doing it wrong.

          1. Ian Entwistle

            Re: Delete is written rename

            i know that, you know that, tell me it doesn't happen in reality... :)

          2. John 110

            Re: Delete is written rename

            I agree, passwords should be on post-its under the keyboard. (NEVER stuck on the monitor! Especially a Dell, as when they invariably fail, hardware support will take your stickies with the old monitor...)

          3. brotherelf

            Re: Delete is written rename

            My boss agrees. All his cruddy perl scripts have the passwords right in the code, hardcoded next to the database name.

      2. Doctor Syntax Silver badge

        Re: Delete is written rename

        That's why GUIs have waste-bins.

        1. Anonymous South African Coward Bronze badge

          Re: Delete is written rename

          Or Recycle bins.

          The only exception to this was the Shredder in OS/2. It did its job too well.

          1. Zippy´s Sausage Factory
            Unhappy

            Re: Delete is written rename

            Now I've come over all nostalgic thinking about OS2.

            Wonder if my install CD still works? Can I get a VirtualBox instance going, I wonder?

            Hmm... time to find out, methinks...

            1. jake Silver badge

              Re: Delete is written rename

              You can still purchase a brand new, shiny license to run OS/2, complete with support for (some) modern hardware. And telephone support. I've used two different versions in various places over the last couple years ... Serenity Systems has sold eComStation since 2001, and Arca Noae LLC has sold ArcaOS since 2017. Both with IBM's blessings. Wiki for more (and links). Recommended.

        2. big_D Silver badge
          Facepalm

          Re: Delete is written rename

          Except it is usually the "the files are too big for the recycle bin" files that are permanently deleted that are the ones you probably need!

          1. Doctor Syntax Silver badge

            Re: Delete is written rename

            I've never hit that one. Maybe I'm using the wrong right GUI.

        3. Jou (Mxyzptlk) Silver badge

          Re: Delete is written rename

          > That's why GUIs have waste-bins.

          Which don't work on network drives in many OS-es. A big THANKS to shadowcopy/snapshots and other server side "recycle bin" implementations nowadays - IF YOu ACtiVATeD THEM !!!11!11oneeleven!!!

      3. Jou (Mxyzptlk) Silver badge

        Re: Delete is written rename

        Same for registry changes.

        In config files: Duplicate the line to edit, put a comment in front of your "copy" with an iso8601 date marker.

    2. Terry 6 Silver badge

      Yup. In my days of being the only (vaguely) IT knowledgeable person in my or the other related teaching service my rule was always that a belt and braces was a starting point. I'd make sure the data we were using was always backed up, but, before I did anything hairy I'd do a separate back up, copy it and then check the copy to make sure it was OK.

      And then I'd do something similar to any software I was replacing. I always assumed the effort I was making was commensurate with the harm that could be done if I f***ed up.

      This is the benefit of being mostly self-taught- and many of the bits I wasn't self-taught had been taught to me by people who were self-taught, all of us going back at least to 386s with single floppy drives and no HDD- (Thank you John Browne- now I assume a long retired headteacher).

      I'd learnt my earliest computer skills as a school kid in the 70s- testing the school's -rudimentary by modern standards- machine to destruction. So I knew the effort required when things went awry.

      1. John Brown (no body) Silver badge

        "all of us going back at least to 386s with single floppy drives and no HDD- (Thank you John Browne- now I assume a long retired headteacher)."

        First, I'm not THAT John Brown :-)

        Second, 386's without HDDs? That's pretty unusual. an original 8088/86, even some 286's, but by the time 386's came out it was pretty rare to see one with no HDD in it.

        1. Terry 6 Silver badge

          School ones had floppy drives, no hdd at first. It's too long ago for me to remember when we got machines with HDDs. But I think they arrived with our first few 486s.

          1. jake Silver badge

            I honestly don't remember 286s without hard drives being common. Floppy-only systems were (mostly) an 8086/88 thing.

            The 286 based IBM PC AT (mid '84) came with a standard 20 Meg HDD. Clone manufacturers had to keep up or fall behind. Remember having to partition aftermarket 40 meg drives into two partitions because DOS couldn't use the whole 40 megs? That was the world of 286s, and some 8086/88s that were retrofitted. Compaq's first 386 computers also had the 32 meg limit ... until their own DOS 3.31 came out a little over a year later. The rest of the planet had to wait for DOS 4.0 ...

  5. ColinPa

    DOS/VS build problem

    One of my first jobs 40 + years ago was in build. I built a major application on DOS/VS running under VM on IBM 360.

    Disk space was short and we only had 2 disks/systems - the "current system" and the build system.

    I was given an envelope with a short list of instructions which started "delete the old build". So off I went.

    About half an hour later we had some developers came round to say there were some files missing... I was working on the live system - not the build system. Whoops. I got a bollocking and the instructions were amended to "1) change to the disk to the value written on the board".

    Next Monday I started the next build.... and half an hour later a developer came round and said there were some files missing. I got another bollocking because someone had forgotten to update the board!

    The next week I got moved to work on VS1 where I could do no damage. All important disks were read only.

  6. Doctor Syntax Silver badge

    But why clear out /usr anyway? Even if the files were going to be updated the old ones would have been replaced with the new ones. It''s not as if they'd have been left hanging around.

    1. jake Silver badge

      Depending on how far the new kernel deviates from the old kernel, you might very well have new commands with new names that supersede old commands. Installing the new will not get rid of the old, which can raise all kinds of merry hell.

      This was especially true in the early days of Linux, when people were still in shell-shock after the UNIX Wars and weren't quite certain which direction Linux was going to move off in.

      1. Electronics'R'Us
        Devil

        New and updated commands

        Oh the joy.

        I was working with a mixture of SPARC and UltraSPARC (Solaris) machines and some SGI Indys (IRIX) in the late 90s. A lot of the 'standard' commands (such as at) had totally different command line options.

        Even on the same machine where /usr had not changed in a long time but the OS had undergone a number of releases, differences could exist between apparently identical commands in /bin and /usr/bin

        Confusion reigns!

        Happy days.

        1. Mike Dunderdale

          Re: New and updated commands

          And also when some of your developers had optimised by rebuilding the commands against other libraries than the system standard places... all different given versions of irix on the on the Indys, the O2s and the Optanes...

        2. Down not across

          Re: New and updated commands

          Not to mention on Solaris you had the be aware of the path (/usr/bin vs /usr/ucb) as you had both SysV and BSD variants for some commands.

      2. Doctor Syntax Silver badge

        The time to remove them is after they become obsolete, not before, and on a case by case basis.

    2. Jonathan Richards 1

      Move it don't kill it

      As someone points out above, you can always shift something sideways (given sufficient room). My approach these days is to do something like

      $ mv /opt/coolprogram /opt/was_coolprogram

      The was_ prefix is sufficiently unusual that I can always find such "sidings" (backup is too strong a word) and remove them when I need the space, or I'm sure they're obsolete.

      However, I'm not sure that would have worked on a running system with /usr. There's a lot of pretty essential stuff in that hierarchy.

      1. John Brown (no body) Silver badge

        Re: Move it don't kill it

        "There's a lot of pretty essential stuff in that hierarchy."

        ...including, depending on the system, /usr/home

      2. Jou (Mxyzptlk) Silver badge

        Re: Move it don't kill it

        I use a similar scheme... I append "delete at <six month in the future>" to such directories.

  7. UCAP Silver badge

    I feel for "Aapt"'s pain

    I did something fairly similar once on a Sun 3 workstation (which really dates this event for those in the know) running SunOS. We had an considerable accumulation of files in /tmp that where starting to cause issues (SunOS insited on putting /tmp on a separate partition that never seemed to be large enough), so one morning, nice and early before anyone started work, I decided to have a clean up. Having logged in as root, I then entered the dread command "rm -rf / tmp/*" - yes I had accidentality inserted a space after the first slash.

    A rapid control-c proved futile - too much was gone. I had just enough left to perform a backup of the important files, then spent the next couple of hours reinstalling the OS from tape.

    Lesson learnt the hard way.

    1. Pascal Monett Silver badge

      Those are the lessons you remember best.

    2. wub

      Re: I feel for "Aapt"'s pain

      "...putting /tmp on a separate partition that never seemed to be large enough..."

      While I was still too new to Linux to be confident about things like planning partitions, I ran into several authorities that suggesting several partitions: / /boot /usr /var /tmp each on its own partition. Even though I was working with fairly small disks, even for the day, I followed that advice. Yes, it is a pain to compile and install complex packages when /tmp is constrained. I certainly don't do it that way now.

      One day, I was surprised to see a number of comments online from folks running the same SQL server as I was who were suddenly suffering from "disk full" conditions. It turned out that a bug in a recent update of the SQL server was causing thousands of error messages a second to be written into the logs. I looked, and discovered that my /var partition had indeed filled up, but the system wasn't too unhappy since the other partitions were fine. For the most part, everything was running smoothly.

      I was impressed when I realized that those authorities whose advice I had followed seemed to have had some sort of clue about the way things work, and fail, after all. I wondered how long it might have taken me to notice. Yes, I was not as log-oriented then as I am now - another thing I learned along the way.

      1. Doctor Syntax Silver badge

        Re: I feel for "Aapt"'s pain

        "Yes, it is a pain to compile and install complex packages when /tmp is constrained. "

        This is why you keep spare space available and use LVM or whatever is appropriate for the OS to adapt to circumstances.

        1. John Brown (no body) Silver badge

          Re: I feel for "Aapt"'s pain

          To some extent, but things like LVM were not available back then. Having said that, I started my Unix life with FreeBSD and the received wisdom was separate /, /uisr, /var and /tmp back then too. Over time and OS upgrades and larger apps being installed, /tmp was no longer large enough to build firefox or OpenOffice. My workaround was to create a link to /usr/tmp from /tmp, temporarily until I could find the time to backup. wipe the HDD and build new and bigger /var and /tmp partitions.

      2. KarMann Silver badge
        Linux

        Re: I feel for "Aapt"'s pain

        One thing that still mystifies me in hindsight is how my first RHL install ever managed to work. I got really carried away with the separate partitions. I mean, really carried away. Like, separate /etc/ and /lib/ partitions carried away. To this day, I'm not entirely sure how it ever managed to function. I'm guessing that mkinitrd or whatever we used back then was smart enough to include the relevant /etc/ bits and necessary /lib/ libraries in the initrd (which was on a floppy then, of course).

        Surprisingly, I never really came to regret it in any painful way. The closest I ever had to suffering for it was when I finally did the big upgrade to Windows 95 (it was dual-boot, obvs), and Windows mucked up the Linux parts of the partition table, because why would it care about other OSs? That was my big intro to really getting into the guts of the system, in a sink-or-swim way, since I had to use the Windows to go get the relevant docs, and edit the partition tables by hand to point them at the right places, lots of fun the way the MBR partitions are chained, let me tell you, and using dd to put them in just the right place. (I don't remember just how I was running binary editors & dd at the time; was a live floppy a thing then? I think it predated live CDs.)

        1. John Brown (no body) Silver badge

          Re: I feel for "Aapt"'s pain

          Under MS-DOS (and by extension, Win98-98 (ME never existed!!), you could read and write sectors with debug. It was a remarkably powerful and usefull command, not really available in *nix AFAIK (or I never need to look for one! - Likewise, SnoopDOS on AmigaOS was great for seeing just what files, disks, devices etc a program was trying to access, again, seemingly never made it to other platforms or OS)

          1. jake Silver badge

            Re: I feel for "Aapt"'s pain

            Yes, there are disk editors for *nix. I don't recommend trying them out just for random fucking around, they are dangerous tools and probably shouldn't be used by your average Ubuntu "sysadmin" who thinks that RTFM and understanding it is for old-farts..

            Most (all??) versions of *nix since before BSD have had tools to keep an eye on which files, disks, devices etc a program was accessing or attempting to access.

    3. DrBobK
      Headmaster

      Re: I feel for "Aapt"'s pain

      I had a Sun 3/60 many years ago. One thing I think I remember about SunOS 3.2 and 3.5, but my memory might be faulty, was that they had a partition scheme in which stuff you really shouldn't mess with was mounted read-only. I think it saved me, an academic not a proper sysadmin, on more than one occasion. This was around the same time that one of my colleagues decided to remove all the unused files in /dev on the departmental server (another Sun box) - enough remained in memory to rescue the system from the command line. In the land of the blind the one-eyed man (me) is King.

  8. J.G.Harston Silver badge

    Rassen frassen

    AUTOEXEC.BAT sets TMP=C:\DOS

    sometime later, something does DELETE /R/Y %TMP% .....

    1. jake Silver badge

      Re: Rassen frassen

      That's why on DOS one of the first things I did was set TMP=D:\TMP and TEMP to D:\TEMP ... most folks probably still don't know it, but Microsoft uses TEMP for user temp files, and TMP for development temp files. Pointing them at separate directories can save headaches occasionally.

    2. Jou (Mxyzptlk) Silver badge

      Re: Rassen frassen

      This is the reason why my "clean up temp" scripts have a fallback, here the windows example:

      del /s /q %temp%\..\temp\*.*

  9. AlexG_UK

    I started my working career in a specialist software dev house where all the developers had SPARCstations of various flavours. The configs, while generous at the time, where tried by the work we were doing .. particularly disk and memory. One of the other devs, need to free up space on his hard drive and started deleting a stack of files he never 'used' from /, /etc, /bin, /usr, /dev, /mnt and so on.

    All was fine until the next reboot when we it didn't. Our sysadmin spent many happy hours in PROM mode rebuilding just enough of the files with echo "xxxx" > 'filename' until he could minimally boot it and restore from back up. Happy days :)

  10. David Robinson 1

    Mistakes are how we learn

    I've done "rm -rf /usr" before. Fortunately it was on a personal Linux machine. For whatever reason I'd made a copy of /usr and then came time to remove it. Of course muscle memory kicked in and put the '/' before 'usr'.

    Back to the title of my post. I got into Linux in the late 1990s. Back then, you learnt by scouring Usenet groups and using this up and coming search engine called 'Google'. (I wondered whatever happened to them?) You'd try something, break your system and work out how to put the pieces back together. Over the years I've noticed a trend for newer users not to experiment but want the answer spoon-fed to them.

    1. A.P. Veening Silver badge

      Re: Mistakes are how we learn

      And that is what happened to Google, it leads to spoon feeding YouTube videos.

      1. Tabor

        Re: Mistakes are how we learn

        I add "-youtube" to all my searches nowadays. I started doing that after Google "helpfully" pointed me to YouTube videos when I was looking up some Cisco CLI references...

        1. stiine Silver badge

          Re: Mistakes are how we learn

          I add site:experts-exchange.com to my first search. This way I know if somone with a budget has had the problem before.

          As an aside, has anyone esle noticed that within the last year or so, that at least one group of scammers have downloaded every error message that can be generated by RHEL and use them as meta tags on their scam pages? There are some error messages that ONLY show up in sites with names like btrfucnyihum.wordpress.com...

    2. nintendoeats

      Re: Mistakes are how we learn

      I ask you to consider; what would happen if I told my boss that I spent my day "experimenting with Linux trying to bring it back up"? If sysadminnery is not your job, it is a thing that gets in the way of your job.

  11. Anonymous South African Coward Bronze badge

    We have fond memories from the last century of an overpromoted DBA insisting that using DEL *.* on a running SQL Server would be fine since SQL locked .MDF files and .LDF files. Oh how we laughed when we discovered that backups hadn't been taken first.

    I think the time has come that I can now safely say that I borked an Exchange solid - by deleting all the journal log files, but I mistakeny ASSumed the Exchange database files will be locked.... I deleted those files as well... and no backup in sight....

    ....was a very interesting week. Company was not amused. This was about 10 years ago.

    Learnt my lesson then. Always, always have a backup (and know what you're doing)....

  12. jake Silver badge

    Just over a billion years ago ...

    ... as the Internet measures time (call it roughly 1984), I received a brand new Sun 2/160. It was a dual pedestal beast, with all of 8 Megs of RAM and a pair of 380 Meg CDC SMD drives. Roughly 65 grand worth.

    I decanted it from the boxes-on-pallets, plugged all the cables in, and fired the thing up. Into a beautiful new GUI on the Sony Trinitron monitor, just as advertised. Logged in as root, on purpose as there were no other accounts as yet(!!), using the default password(!!!!) ... and poked around. All was well, near as I could tell.

    The plan was to repartition the disks to better suit our needs and then reinstall the OS. So I made absolutely certain I had the correct tapes, and did the one thing I had never done as a sysadmin ... closed the GUI, and from the # prompt ran rm -rf / intentionally. I was curious to see how long it would take to lose it's tiny little mind. It trundled away to itself for a few minutes, but seemingly was still working fine, enough of vmunix and the shell were in RAM and the swap partition to keep doing simple stuff. I was quite surprised, but that wasn't really what I was there for ...

    So I shut her down, went and got a cuppa coffee, reached for the tape and went to fire up the machine ... only to discover it didn't ship with a tape drive, despite one being listed on the packing list. It had a lovely beezel that LOOKED like it might be a tape drive, but the space behind it was empty. Oops. So there I was, stuck with 65K worth of dead Sun hardware that I was supposed to demo for the Brass at 4PM.

    Fortunately the 1980s Sun had Clues about customer service. One call, and their field service rep had the SCSI tape drive, the requisite cables, and a couple of VMEbus cards, (E)EPROMS and spare OS tapes "just in case" on my desk in under forty minutes. She even hung out and made certain that the system worked properly after we took it apart to install the bits that needed installing, and then partitioned it and re-installed the OS.

    I made the 4 o'clock deadline ... and bought the Rep the first of many well deserved dinners.

    Silly Con Valley was a very small place back then ... Sometimes I miss it.

    1. Doctor Syntax Silver badge

      Re: Just over a billion years ago ...

      "I was curious to see how long it would take to lose it's tiny little mind"

      We were due to return a loan machine. I'd dropped the database but in a similar fit of curiosity decided to try to ensure everything was overwritten by doing the dread recursive cat that we used to be warned about, starting with a file of ajs[odiJS[IFOJO'[SA style rubbish. I discovered it took far too long. I had to hit the switch so we could hand it back.

    2. Phil O'Sophical Silver badge

      Re: Just over a billion years ago ...

      DEC customer service was as good in those days as well, there's a price to be paid for the hugely cheaper computing we take for granted now.

    3. ilithium

      Re: Just over a billion years ago ...

      What a great story - thanks! :)

    4. Paul Crawford Silver badge

      Re: Just over a billion years ago ...

      We also got some Sun hardware around 1991 and it was good. not cheap, but good and came with proper support and knowledgable folk at the end of the phone or email.

      Later we got some UltraSPARC machines and one of them continued as web/email server without a hiccup for about a decade after it should have been retired. Its only real problem was the non-replaceable real-time clock module's battery had gone after ~10 years. And that also held the CMOS config! Doh! So to boot it you have to type in the partition info (that our guy had put on a sticky note on its box just in case)

      1. l8gravely

        Re: Just over a billion years ago ...

        Who else remembers the early Sun Ultra 140s and 170s came with a crap CPU fan that had to be replaced before they stopped working. Luckily you could prioritize them by the sound of the fan whining if you only had a few spares kicking around. And luckily they also had good over-temp protection in the BIOS so they would just shutdown if they fan bit the bullet.

        It was a major supply snafu at first, they (Sun) doled out spares like scrooge for a while. Sometimes you could get them working again with some compressed air, but usually the bearings had locked up so hard it was obvious that nothing could be done.

        Fun days...

    5. KarMann Silver badge
      Coat

      Re: Just over a billion years ago ...

      Kids these days! Why, back in my day, we'd gotten our first Sun about 4.5 billion years ago!

  13. Paul Cooper

    Limited space and facilities

    Many moons ago, I was "sysadmin" for a mini-computer system - a CMC reality (this was the late 70s). In those days, system upgrades were performed by the manufacturer, so that wasn't an issue.

    However, we were handling large databases (by 1978 standards - tens of thousands of records, I think) and writing software to manipulate these databases in ad-hoc ways almost every day. But what a lot of people here are forgetting is that in those days, system resources were tiny, disc (yes, we had a disc) space was extremely limited and backup was a 1600bpi reel-reel magnetic tape. Indeed disc space was so limited that you often had to save a file to tape to be able to load another one. However, several databases had to be on-line - we ran an online query system for a few clients (no internet - this was via modems and hunting groups). Remember, too, that in those days even the concept of a file or a record varied from machine to machine - the ubiquitous Unix style bit-stream model of a file wasn't by any means universal. The CMC, for example, regarded a file as a collection of records whose location was allocated according to a hashing function derived from key fields - a record was a collection of fields. So even if you had a backup, it wasn't an image of what was on the disc, it was the output of a command to convert the disc version to a sequential format.

    Anyway, the point I'm working round to is that very often we had no choice but to work on a live and running database - there wasn't space or resources to do anything else. And equally, the safeguard was to ensure you were working on a very well-defined small set of records when carrying out tests so that you knew what you were going to have to fix if it went pear-shaped. But of course, it was far more error-prone than anything a modern DBA or sysadmin would allow. today! And equally, of course, we were often manipulating data input by bored keyboard operators, so incorrectly formatted data were fairly common (we did apply validation on entry, but again, it was limited by the resources available). Very often, we'd run a command to reformat a field according to new customer requirements and find that it fell over when it hit a condition that we hadn't anticipated because an operator had put in something the client had told us didn't happen!

    Whatever the rights and wrongs of what we were doing, it didn't half teach you to bench-check your code VERY carefully!

    1. keithpeter Silver badge
      Windows

      Re: Limited space and facilities

      Google suggests Microdata minicomputer running a version of the Pick OS? Seems like a very non-mainstream system from today's perspective, so you all did well with it!

  14. DarkwavePunk

    The joys of recursive file operations.

    Back in the mid '90s when I'd just started out in IT I was mucking about on a production Sun system as root as all such tales must start.

    I'd downloaded a tarball of some software and unpacked it into a tmp directory. The software had an etc/ directory. For whatever reason I wanted to do chmod -R 755 etc/

    As you can guess I typed chmod -R 755 /etc/ instead. It was not a happy bunny to say the least. Fortunately there was a backup server that was mostly a mirror of live.

    It did however mean I spent rather a lot of time manually changing the permissions of everything under /etc/ back to how they should be from a terminal console using the backup as a reference.

  15. Anonymous Coward
    Anonymous Coward

    I've seen a sysadmin do this on a database filesystem many years ago.

    Mid week panic call. The software I was supporting at the time would not start. It was logged as a fault with our software. Turns out the sysadmin 'saw some large files and decided to clean them up'. They were the database files. Their last backup was at the weekend. Poor woman was nearly in tears, but there wasn't much I could do to help.

    Also seen someone do the rm -rf / not once but twice! Still makes a mess of the operating system.

    Same person did a chown/chmod from /. That took a while to sort out too.....

  16. Paul Crawford Silver badge

    On a machine I was going to re-image anyway I decided to try out the rm -rf /* sort of thing as root just to find out how far it would go.

    To my surprise it was very effective, eventually the desktop menu text/etc vanished as the fonts were removed and finally the machine went quiet. Booted in to the live CD to see what remained and only a few system directories were there, presumably as they had open files that somehow meant the directory itself could not be removed. But all files had gone, including 'rm' itself!

    I was impressed and sobered at the same time!

  17. My other car WAS an IAV Stryker

    "You can't hurry Linux kernel upgrades", or love

    ...and you can't cheat death.

    If that Supremes reference was not meant as tribute to the late Ms. Wilson (died 8 Feb), it should have been.

    RIP, Mary.

  18. Børge Nøst

    Versioning - still not an option

    I have been waiting for 25-30 years for a versioning filesystem that saves me from myself, but alas - I seem to have to keep waiting...

    1. Steve Kerr
      Mushroom

      Re: Versioning - still not an option

      Oddly, a friend of mine had written one as a never delete filesystem so it's possible to get stuff back.

      He brought it over for destruction testing, I destroyed it to which he went "oh"

      Fun times.

      1. Doctor Syntax Silver badge

        Re: Versioning - still not an option

        "a friend of mine had written one as a never delete filesystem so it's possible to get stuff back."

        Assuming you do get it to work you then meet the problem where data absolutely has to be deleted for security data protection reasons.

    2. Anonymous Coward
      Anonymous Coward

      Re: Versioning - still not an option

      Thats fine until you decide to clean up your folders with a "purge all but the last 3 copies of each file" command, and then discover you really need to know what was different in the -4th version of one specific file - speaking hypothetically you understand.

      For all their irritations there is a lot to be said for (properly managed) file repositories.

    3. Phil O'Sophical Silver badge

      Re: Versioning - still not an option

      a versioning filesystem

      Like the one that VMS has had since the 1970s? Even the standard CDROM file system allows for it, but no-one seems to use it.

      1. someone_stole_my_username

        Re: Versioning - still not an option

        I can tell from personal experience, that even the VMS versioned filesystem did not protect you from yourself, if you were determined enough!

        It was far too easy to type delete *.*;*

    4. Jou (Mxyzptlk) Silver badge

      Re: Versioning - still not an option

      Most OS-es of today offer snapshots, and depending on which OS an option to do them on regular intervals, and on some OS-es with a definition "how much space to waste on snapshots", and on some OS-es with an easy to use GUI to handle ALL those options at once.

      I know Windows does that since Server 2003 R2 (and Windows XP with SP1 or SP2). Other OS-es vary. Saved me countless times - and I started to rely on it. You can ever browse them easily from CMD shell, if you know how. Adn the default of "7:00" and "12:00" as daily snapshot times is indeed a good choice too.

      Sadly they removed the shadowcopy-GUI entrypoint from Windows 10 lately (VSSUIRUN.EXE specifically), but not on Server 2019. You can still call it with Windows+R sysdm.cpl on newer windows installs and "steal" the shadowcopy-tasks from an existing Server 2012 to 2019...

      1. Jou (Mxyzptlk) Silver badge

        Re: Versioning - still not an option

        PS: you can steal that handy UI from a Server 2019. VSSUI.DLL, VSSUIRUN.EXE and de-DE\VSSUI.dll.mui (or en-GB for most reg people). Copy them to the same place on your Win10 OS.

    5. Anonymous Coward
      Anonymous Coward

      Re: Versioning - still not an option

      openvms' files-11 filesystem ???

  19. Camilla Smythe

    Bloody Experts

    Yah Yah Yah. All of your horror shows about rm -rf and you don't know about -rm rf. Thick idiots.

    1. DarkwavePunk

      Re: Bloody Experts

      I think a lot of these things are recounting silly things done in youth with little experience or devilish typos rather than being "thick idiots".

      I remember one poor sod deleting /proc/kcore after doing a find when disk space was low. He'd seen core dumps from software before so thought it was a good candidate for deletion. Obvious outcome is obvious, but the reasoning was to some degree sound.

      We all learn.

      1. Robert Carnegie Silver badge

        Re: Bloody Experts

        I deleted /unix once, it didn't seem likely to be an actual important file. But apparently it rather was.

        1. ilithium

          Re: Bloody Experts

          Just do cat /dev/null and stare into the void...

    2. ilithium

      Re: Bloody Experts

      Totes amuse. There's always one...

  20. Anonymous Coward
    Anonymous Coward

    Me and the Home Office.....

    ME

    Upgrading the OS on an AS/400....needed a recent backup tape. In fact ALL the backup tapes were corrupt......no one had ever tested a restore. Ooops!!!

    HOME OFFICE (recently)

    DELETE FROM <table one>

    DELETE FROM <table two>

    ...and so on.....so much easier that adding that irritating WHERE clause!!!!!

    1. A.P. Veening Silver badge

      Re: Me and the Home Office.....

      ...and so on.....so much easier that adding that irritating WHERE clause!!!!!

      Easier? Yes. Quicker? No, on an AS/400 you will get a request for confirmation if you forget or deliberately leave out the WHERE clause. If you know what you are doing, it is usually quicker to delete the files (DROP the tables) with all indices and recreate them empty.

  21. Xalran

    It's in that kind of situation....

    That a colleague got himself into last century with the dreaded rm -rf * in /opt. ( on an HP-UX system )

    Luckily for him we had a backup. unlickly for him the backup script that made the tape was bugged and

    when we tried to restore /opt it restored /home.

    Another lucky point for him : he did that on our internal testbed, not on a live system.

    After several hours of frantically learning the intricacities of the mt command we managed to restore the /opt where it was meant to go.

    After that day he ( and I ) never used rm -rf * unless we were in the directory of the content we wanted to delete ( and had checked that fact with a pwd ).

  22. Anonymous Coward
    Anonymous Coward

    a colleague of mine was doing some updates to the nodes in a NUTANIX cluster, so update node wait for it to complete and reboot and added back in to the cluster etc move to next node, unfortunately, he got a bit ahead of himself and kicked off another node whilst the one he was working on before was still wasn't up, redundancy factor only allowed for one node to be out at any one time. Cue full failover of the 500 odd VM's to the other cluster at the other DC!

  23. Roger Kynaston

    been there done that

    Mine was

    # newfs /dev/rdsk/c0t1d0s4 rather than c1t1d0s4. I was able to watch my. /usr disappear and the server fall in a heap.

  24. Nate Amsden

    don't understand

    As someone who manually compiled and installed many kernels from 2.0 until late in the 2.2 series(fond memories of 2.2.19 for some reason), I never ever even one time had to delete any files as a result of a kernel update (outside of MAYBE the /boot partition if it was low on space). I had to check the article again to make sure it was referring to Linux and it seems to be. Since with 2.4.x onwards(basically when they stopped doing the "stable" and "development" kernels) I rely on the distro packages for kernel updates again, no files need be deleted for such updates.

    Could this story somehow be referring to an OS upgrade rather than a KERNEL upgrade? Of course in linux land these can and typically are often completely unrelated(even in 2021, as much as the linux kernel is terrible about binary compatibility with it's own drivers many people know that whether you run kernel 3.x or 4.x or 5.x provided it supports your hardware there isn't much difference for typical workloads). But even with an OS upgrade I don't see a need to delete (m)any files. I also spent hundreds of hours back in the 90s compiling things like gnome, kde, X11, even libc5 and glibc, among dozens of other packages I can't remember anymore.

    maybe this was a thing before 2.0 kernels (my intro to linux was slackware 3.0 with 2.0.something I believe back in 1996), but I suspect it was not.

    this whole story just doesn't make any sense.

    1. ilithium

      Re: don't understand

      I see your point but this _is_ El Reg, after all. And even if it's technically wrong, it's still amusing.

  25. bill 27

    It's OK.

    Honest...it'll keep on chugging until it needs something not in memory. IIRC is was the network that gave up the ghost first, not that I've ever run that command as root, why fiddle with just /usr go for /, just to see what happened.

  26. LovesTha

    Active Directory - not TM

    Due to a bad IDE we had a script that recursively deleted all common executable files. It lived in the project dir, a quick double click and the next build might work.

    Unless you were working on a network share and windows treats c:\Windows as the active directory to run the script in.

    That was a late night repairing the bosses desktop.

    1. Jou (Mxyzptlk) Silver badge

      Re: Active Directory - not TM

      That's why you don't work as local admin by default - NOWADAYS...

  27. e-horace

    It wasn't just "rm" you had to be careful with...

    I remember as a fledgling sysadmin being sent with a colleague (hello Huw) to a week long DEC Ultrix Systems course at Shire Hall, Reading (now that building really was an exercise in spacial awareness!). We had been introduced to "setld", the Ultrix package manager and as part of the whole learning, I recall removing one teensy small package and wondering why it was taking so long to finish. Until it dawned on me that by mistake I had selected to delete one of the core operating system sets... [isn't it strange how for some, those moments make the room go terribly warm and for others, somewhat chilly].

    Not enough of the OS survived for it to be repairable but fortunately we were using RIS servers to provide the packages rather local media, so a swift reinstall followed. But it is safe to say that the remainder of that session 'till afternoon tea was somewhat hazy.

    Actually, more fortunate was the fact that my mistake was made on the training ground as opposed to in the heat of battle.

  28. yetanotheraoc Silver badge

    Aapt get?

    Missed opportunity. :-(

  29. Jou (Mxyzptlk) Silver badge

    MS-Basic 3.5 on Commodore +4

    What,

    SCRATCH "*FILENAME"

    is not equal to

    SCRATCH "FILENAME*"

    ?

    That is when I learned that Commodore MS- Basic 3.5 and 7.0 ignore everything after the * wildcard. Unscratch PRG for rescue.

    https://www.c64-wiki.com/wiki/SCRATCH

  30. TheWeetabix

    /etc/skel/bash.rc

    alias "rm -rf \/"="echo 'Are you fonking daft?'"

    alias "rm -rf ."="echo 'Use full paths you massive shit.'"

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like