back to article Undergrad thought he had mastered Unix in weeks. Then he discovered rm -rf

Another Monday and what a fine one it is here in the lair of Who, Me? – the reader contributed column in which your fellow Reg-admirers admit to the moments they messed up the tech they were supposed to tame. This week, meet a reader we’ll Regomize as “Miles” who told us he cut his computing teeth on the Commodore C64 and …

  1. Tim 11

    what does ~* do?

    Can anyone explain that? I could imagine someone might invent a shell where ~* expands into a list of everyone's home directory, but I've never heard of that

    1. may_i Silver badge

      Re: what does ~* do?

      Doesn't make sense does it? ~* is nothing, ~/* is everything in the current user's home directory.

      I can't think of any variations where that might be everything in everyone's home directory.

      1. Michael H.F. Wilkinson Silver badge

        Re: what does ~* do?

        tcsh quirk, maybe. Haven't used anything other than bash for donkeys' years, so wouldn't know

        1. YetAnotherACUser

          Re: what does ~* do?

          rm -fr ~/..

        2. geoff61

          Re: what does ~* do?

          If it's a tcsh quirk, it has been changed since the events recounted here:

          $ tcsh -c 'echo ~*'

          Unknown user: *.

          1. munnoch Silver badge

            Re: what does ~* do?

            ~X would expand to the home directory of user X.

            Whilst I was a big fan of tcsh in my early unix days I was unaware that ~* expanded to all users home directories. Seems a very odd thing to implement and struggling to think of a good use for it.

            And of course emacs can be taught to look after its backup files all on its own.

            1. Ken Hagan Gold badge

              Re: what does ~* do?

              Thank you. That explains everything except ... what the blazes did anyone think that feature would be useful for?

              I mean "~" is certainly nice and I can almost imagine a case for "~user" but "~*" just seems crazy. Why on earth should joe user even gave access to a complete list of user accounts, let alone a shortcut way of referring to their private directories?

              But that's not your problem. Thanks again for the explanation coz there's no way I'd have guessed the existence of such a feature.

              1. Karl Vegar

                Re: what does ~* do?

                Just guessing here

                * is a wildcard, so making ~* not work might be more work than leaving it in

                Not to mention I can see a few cases where it might be handy for admin (backup, clearing out all userdata...)

                I'd guess designers didn't foresee users having access to more than their own home directory, so leaving it in shouldn't cause any issues (theory meet reality)

        3. katrinab Silver badge
          Meh

          Re: what does ~* do?

          I use tcsh as my default shell in FreeBSD. Bash is only available in ports, and I don't usually install it.

          It returns "Unknown user: *."

      2. OllieJones

        Re: what does ~* do?

        The emacs backup file for kernel.c is named ~kernel.c, with a leading tilde. So that formulation removes the emacs backup files.

        1. katrinab Silver badge

          Re: what does ~* do?

          You would need to do \~* for that

          Certainly on tcsh/FreeBSD you have to escape the ~

          1. doublelayer Silver badge

            Re: what does ~* do?

            It could be one of those situations where there is one more or one fewer level of escaping than you expected. I've certainly experienced it as I learned things, things which I would have to escape when I was typing them on the shell, but now that a program I wrote was doing it, the escaping was done for me. Usually, that led to one more level than I needed and the commands just didn't work, but it can go the opposite way where you assume it will do that and it doesn't. Or it could have been a path thing, where a script was running in a working directory different to where the user thought it was, which could easily turn into something too close to root if there was a "cd .." somewhere in there.

      3. John Doe 6

        Re: what does ~* do?

        In some quite old UNIX systems root didn't have a separate home directory...

        executing rm -r ~* would be something like rm -r /*

        This is from the HP manual "HP-UX 11i Version 1 Installation and Update Guide" under "Cold Install" ...

        "Consider creating a root home directory that is not /. This keeps the user root dot files out of the / directory. Make sure it is on the root volume by not locating it under the mount point of the file system. A good name to use is /homeroot (not /home/root) as /home is often a mount point. This is especially important if you are using LVM or VxVM and /home is a seperate volme."

        1. Anonymous Coward
          Anonymous Coward

          Re: what does ~* do?

          "Make sure it is on the root volume by not locating it under the mount point of the file system." is such a horribly written and ambiguous sentence. Or maybe the writer has issues telling "the" and "a" apart.

    2. Alan J. Wylie

      Re: what does ~* do?

      Quoting the original article: Emacs backup files

      emacs appends ~ to the filename when creating backup files.

      $ cd .emacs.d

      $ ls *~

      bbdb~ gnus.el~ viper~ vm.el~

    3. Ian Johnston Silver badge

      Re: what does ~* do?

      Was there not an issue at one time with shells which thought that ".." was part of a directory. so deleting "~*" included "../*"? Or perhaps I am dreaming.

      1. katrinab Silver badge
        Meh

        Re: what does ~* do?

        From the description of what happened it seems it is deleting

        ~katrina

        ~miles

        ~everyoneelse

        In other words every home directory.

        But trying it out on a sand-box virtual machine running FreeBSD, with tcsh as the default shell:

        root@kagamine:/usr/home/katrina # rm -rf ~*

        Unknown user: *.

        I did a snapshot before trying it. I don't need to restore anything.

    4. Arthur the cat Silver badge

      Re: what does ~* do?

      arthur@arthur[4]▶ echo ~*

      Unknown user: *.

  2. This post has been deleted by its author

    1. Korev Silver badge
      Flame

      Re: Colour Me Shocked...

      BBC Micro for the win!

      1. Yet Another Anonymous coward Silver badge

        Re: Colour Me Shocked...

        >BBC Micro for the win!

        Bet you had a Volvo and a proper leather satchel

        1. This post has been deleted by its author

      2. This post has been deleted by its author

        1. Terry 6 Silver badge

          Re: Colour Me Shocked...

          Ah, but having done my tooth cutting (excluding the early teething at school with punch cards and stuff) on the Acorn Electron with all the bits of add-on stuff I could afford to buy from Watford Electronics, I have to say that nothing in the computer world has ever seemed as much fun since then.

          1. This post has been deleted by its author

            1. Terry 6 Silver badge

              Re: Colour Me Shocked...

              Electron. BBC micro was beyond my budget as a newly qualified teacher. Elks were selling cheaply by then, and then I could add stuff when I had the pennies.

              1. Noram

                Re: Colour Me Shocked...

                I remember the my dad's purchase of the Electrong, and the Plus1 (I think it was) for an accounts package on a rom cart, alongside a serial printer that drove him and his friend who worked for ICL nuts trying to get working.

                1. Chris Evans

                  Re: Colour Me Shocked...

                  A Plus1 added a parallel printer port to an Electron as well as two ROM cartridge slots and an analogue port. There may have been a Serial port cartridge but it certainly wasn't common. n.b. I've taken delivery of an Electron & Plus1 today!

    2. Anonymous Coward
      Anonymous Coward

      Re: Colour Me Shocked...

      ...aiming left...

    3. Huw L-D

      Re: Colour Me Shocked...

      Laughs in ZX80...

  3. Korev Silver badge

    It's a bit like this old BOFH

  4. SVD_NL Silver badge

    Bold move

    Most, if not all of us, have had to learn a lesson about rm -rf and/or backups. But intentionally deleting automatic backups is probably the boldest move i've heard of yet!

    Also good thing the sysadmin had proper backups, otherwise he would've been the one in on call/who me? retelling this story!

    1. Korev Silver badge
      Pint

      Re: Bold move

      I had a data scientist call me in a panic as he had an "urgent" report and he'd lost his code. I pottered over and found that it was under version control which made me think it'd be easy - turns out he'd not pushed a commit in six months...

      I showed him how to access the nightly snapshots and was suitably rewarded -->

    2. Dinanziame Silver badge
      Boffin

      Re: Bold move

      But intentionally deleting automatic backups is probably the boldest move i've heard of yet!

      I don't think these really count as backups, but whenever you modify and save a file in emacs, it saves a copy of the previous version in a file of the same name with a "~" suffix. So you end up with a lot of files called input.txt~, code.cc~, index.html~ and the like, which tend to clutter your directory. It's not a big deal, but it tends to be forgotten with time, so I see now that I still have in my home directory a file called blo~ created in June 2023.

      I don't really rely on this feature, though it must have happened a couple of times in the past 29 years that I was useful. But at some point I was sufficiently annoyed that I created an alias to delete all of them more safely than typing "rm *~" with three paranoid checks that I don't have a space in the middle.

      1. Gene Cash Silver badge

        Re: Bold move

        Unless you do something like

        (add-to-list 'backup-directory-alist (cons "." "~/.emacs-backups"))

        which makes it put backups in that dir, and the filename has a mangled path prepended.

        My .emacs also says:

        (setq version-control t)

        (setq kept-old-versions 0)

        (setq kept-new-versions 25)

        (setq delete-old-versions t)

        (setq vc-make-backup-files t)

        which makes it keep up to 25 numbered versions of the files.

        1. Yet Another Anonymous coward Silver badge

          Re: Bold move

          Don't emacs users all use VMS anyway ?

          1. Anonymous Coward
            Anonymous Coward

            Re: Bold move

            I wish they actually did.

    3. Ian Johnston Silver badge

      Re: Bold move

      But intentionally deleting automatic backups is probably the boldest move i've heard of yet!

      DCL (under VMS) had the "PURGE" command which removed all but the most recent version of each file. But then it could do that because VMS had file version numbers, lack of which in Linux still pisses me off, thirty years after I stopped using VMS. Come on, Linus, how hard can it be?

      Also VMS prompted for "Username" and "Password:" at login, which neatly aligned the two inputs. "login:" and "Password:" don't align and aren't consistently capitalised. Bloody Linux/Unix.

      1. doublelayer Silver badge

        Re: Bold move

        "But then it could do that because VMS had file version numbers, lack of which in Linux still pisses me off, thirty years after I stopped using VMS. Come on, Linus, how hard can it be?"

        That's not a kernel responsibility and a lot of people wouldn't want it. If you want it, that should go in the filesystem, and there are a bunch of filesystems available with features similar or possibly identical to what you're looking for. Filesystems with automatic snapshots of various kinds are available for that, and they all have the cost of a lot more disk usage, both space taken up and writes to the hardware, to manage it.

        1. rcxb Silver badge

          Re: Bold move

          File versioning has nothing at all to do with file system snapshots.

          "Each file version is stored as a separate file"

          https://wiki.vmssoftware.com/File_version

          They're more like Unix hidden files, just with a special naming scheme and a few utilities that understand the scheme.

          Instead of and /etc/ full of postfix.conf-June21threeCpR type files, we could have just postfix.conf, with (hidden) versions ;1 ;2 and ;3 you could access as needed.

          1. doublelayer Silver badge

            Re: Bold move

            They're not implemented in the same way, but they exist for the same reason, making it easy to return to earlier versions, and they have the same costs, lots more disk space used. Importantly to this discussion, they have to be in the same place if you're doing it in the best way, the filesystem. Now theoretically that last bit is not true. You could implement file versions somewhere other than the filesystem, and the kernel would be the second best place for it to go in the same way that, if you didn't want to put a car in the garage, right outside your front door might be the second best place for it to go.

            While you could implement a lot of this with a bunch of special file names, trying to do it without the filesystem being aware of it is likely to cause lots of problems. You wouldn't do it at the application level because there are way too many tools that won't do it at all or properly. However, to do it at the kernel level would require patching so many different things to decide what they're supposed to do. For example, when you use rm and it calls unlink, is that supposed to delete all the versions of the file or just the latest one. If rm was executed knowingly, they probably don't want that file anymore because if they're planning to use the last backup, they would have executed the command to return that one instead, and if they executed it by mistake, they would want the latest version, not the second-latest. So what is correct functionality for unlink now? This is why the filesystem should manage it, which means you don't need Linus to do this for you. There are some versioning filesystems already. Some of them have problems. It is and should be up to the user to select one that works for what they want.

        2. Ian Johnston Silver badge

          Re: Bold move

          That's not a kernel responsibility

          If everything is a file in Linux then I would suggest, with respect, that support for versioning has to be in the kernel, even if the implementation is in the file system. How would /dev/tty1;1 and /dev/tty1;2 differ, I wonder?

          1. doublelayer Silver badge

            Re: Bold move

            That is yet another reason why it shouldn't be in the kernel. "Everything is a file" doesn't mean that everything should be treated the same way in a filesystem. Versions for device descriptors won't do anything. Either you have a lot of aliases to the same thing in the same state or you have a lot of nonfunctional things. Nobody needs or wants either of those. For the same reason, you really do not want versioning on anything in /proc or everything will be very confusing. We get around that by not putting the contents of /dev or /proc in the normal filesystem.

            The kernel specifies how the filesystem will get instructions. You can implement whatever behavior you need there without having to impose it on every other filesystem in existence, but if you put a feature in the kernel, it will have an effect on every filesystem someone might choose to use, most of those effects being various forms of broken. If your reasoning is that you want a versioned filesystem overlayed over a different one, you can have that. Nothing says you can't have a versioned filesystem that happens to write a structure also interpretable as FAT32. In fact, I've written that. Well not a versioning one, but a hybrid filesystem which could be interpreted as a normal one by devices that didn't support it and as something more advanced by one running my code. It did what I needed, but it had the weakness of needing refreshing after being connected to something that only treated it as a dumb one. By all means write that if you want. Just put it where it should be.

    4. doublelayer Silver badge

      Re: Bold move

      People delete old backups all the time for various reasons. I've disabled a text editor's automatic backup procedure because it created a lot of clutter and frequently didn't clean it up later. There was a cost in that I had to manually track things that I might want to return to and I had to rewrite them if I failed to do so, but I haven't suffered too much by having to do it. I've also had lots of scripts whose purpose is to clear up old backups after new ones are created to save on disk space. There are rules for how many backups there should be and which old ones should be skipped in the deletion process. I have a few programs that intentionally write temporary data, which is sort of a backup in that it lets you resume a process from halfway through, on a ramdisk so that if the computer goes down, that temporary data is cleared, because I intentionally chose to require me to start from scratch rather than have potentially wrong temporary data create a flawed product. Automatic backup removal is pretty common.

    5. Gordon 11

      Re: Bold move

      Most, if not all of us, have had to learn a lesson about rm -rf...

      Mine was when leaving the Shift Key down too long for the * when typing rm -rf ~/*.o.

      Fortunately I did have backups.....

  5. Anonymous Coward
    Anonymous Coward

    Ah....backups....

    Every night, the AS/400 operator picked a new tape, stuffed it into the tape slot, and overnight the AS/400 performed a system backup.

    This went on for years......

    Then one weekend the AS/400 was replaced with a bigger, better model. The upgrade crew need to do some restoring, so they reached for the latest backup tape.

    Ha! Corrupt! ...and so were all the other tapes in the cycle!

    No one had ever tested a restore!!!!

    ....and I still wonder how many systems have backups which have never been tested. Someone out there might be able to tell me.

    1. lglethal Silver badge
      Trollface

      Re: Ah....backups....

      I still wonder how many systems have backups which have never been tested.

      Any company run by Beancounters, so every publicly listed company in effect.

      "What? You want to spend money testing that the Backups actually work? Dont be silly, of course they work. We don't waste money on such frivolous things! Now if you don't mind, the CEO has decided we need to move the Logo on the corporate letterhead to the other side of the page. So I have to organise a budget to get a new design for the corporate letterhead made, get an external printers to start printing the new letterheads in bulk, and hire another firm to dispose of all of our unused current stock. So please go away, I'm busy."

      Cynical, Moi?

      1. Manolo
        FAIL

        Re: Ah....backups....

        Our government spent millions on renaming the Ministry of Safety and Justice to Ministry of Justice and Safety (or the other way round, can't remember, to disgusted to look it up).

        Nothing as easy as spending other people's money.

        1. collinsl Silver badge

          Re: Ah....backups....

          Here in the UK back in 2005 the Department for Trade and Industry was going to be renamed to the Department for Productivity, Energy, and Industry, which would have been written as DfPEnI - unhelpful wags started adding the S to the end of that, other unhelpful wags re-ordered it to DIPE (Dippy).

          The re-branding got to the point where some workers had unscrewed the original name from the side of the department's building and were about to screw the new name up, when the minister responsible, Alan Johnston, spoke to the Prime Minister, Tony Blair, and got the old name put back.

      2. PriestOfTheTemplesOfSyrinx

        Re: Ah....backups....

        We are the Priests of the Temples of Syrinx

        Our minds control the systems all!

        We guide the hands that hold the backups,

        Ensuring nothing dares to fall.

        Oh, foolish minds with your strange requests,

        "To test the backups?" you dare to plea?

        Our beancounter gods laugh at such waste,

        For they see profit's destiny.

        "What folly to squander our sacred funds,

        On testing systems—how absurd!

        Our logos must shift! Our image renewed!

        For this, the CEO has conferred."

        We must design anew the scrolls,

        That bear the mighty corporate crest!

        Discard the old, in silence bury,

        While our budgets are thusly blessed.

        So speak no more of trivial aims,

        Your testing dreams shall turn to dust.

        The Priests proclaim what wisdom deems,

        To maximize the shareholder's trust!

        We are the Priests of the Temples of Syrinx,

        Your doubts must bow, your words recant!

        Profit and pride rule all we build,

        No place for cynics, or your chant.

        1. Huw L-D

          Re: Ah....backups....

          Are you suggesting that your backups are a Rush job?

    2. jake Silver badge

      Re: Ah....backups....

      If the backup hasn't been properly tested, it's not a backup. At best, it might be a copy of whatever you thought you were backing up.

    3. Korev Silver badge
      Facepalm

      Re: Ah....backups....

      I managed to screw up my OPNSense firewall at the weekend, luckily I had configuration backups saved which meant I didn't have to create the VLANs and firewall rules from zero.

      The only problem was that they had an old PiHole IP address and it took me ages to figure out why I couldn't access the Internet (see icon)

      1. katrinab Silver badge

        Re: Ah....backups....

        You can do all your PiHole related stuff directly in OpnSense. It uses the exact same filter lists, so effectiveness is exactly the same.

        1. Korev Silver badge

          Re: Ah....backups....

          I did that for a bit, but found it to be really slow to do reports. The reports in the Pihole are much nicer too.

          Thanks for the suggestion though.

    4. Gerhard den Hollander

      Re: Ah....backups....

      As a friend of mine used to say

      ``Restore ? Restore ? Noone said anything about restore. Alll you wanted was backups.""

      1. Jonathan Richards 1 Silver badge

        Re: Ah....backups....

        > No-one said anything about restore.

        Similarly, I was firmly told that I needed to ask for permission to undertake some course of action I wanted to take (can't remember the details). When challenged later, I said "Well, I did ask for permission. No-one said anything about waiting for it to be granted."

      2. Anonymous Coward
        Anonymous Coward

        Re: Ah....backups....

        In that case, I will suggest using a free and *very* fast device for said backups. /dev/null should work.

        1. Richard 12 Silver badge
          Coat

          Re: Ah....backups....

          It's really big, too.

          Been using it for years and never had to change the tape!

    5. Wally Dug
      Megaphone

      Re: Ah....backups....

      Well, speaking as an old f4rt and someone who is heavily into backups, surely if someone is responsible for changing the tapes (or nowadays checking on the backup storage utilisation or data crystals or whatever the storage du jour is), then they should take the initiative and responsibility to also wonder, "What would happen if we needed to recover some data? Or a system?"

      Build regular recovery testing into the job and make it official. And when it becomes official, it will simply slide into your monthly tasks along with your other checks.

      It doesn't have to be exhaustive and put in the procedures - yes, write procedures! have a checklist! have a record of what has been recovered and when! - a disclaimer that these are just quick recovery tests and no guarantees can be given to recover all data possible and that there is no gaurantee that the contents won't be corrupt without proper testing. With regular recovery testing of the various objects (e.g. files, databases, individual e-mails, AD objects, complete VMs, etc.) that you back up, then when the time comes you will either know exactly what to do and quickly get the system back or it will fail spectacularly. In which case, point to all the previously successful recovery tests and the disclaimer and say "I did my best, but without management buy-in to test properly, there's nothing else I can do."

      1. Doctor Syntax Silver badge

        Re: Ah....backups....

        A test restore can be invaluable for refining your backup creation. The first time we tried that in a DR centre we realised that /etc was so late in the backup sequence it took hours of restoration before we had anything we could even log into to check the files which had been restored. A simple change there made a big difference to the success of the second test.

        1. Yet Another Anonymous coward Silver badge

          Re: Ah....backups....

          Even a successful restore isn't necessarily a good test:

          Back in the far off Unix days our DAT drive was /dev/nrst0

          Due to a small typo in the backup script it wrote to /dev/nsrt0

          The backup script ran as root so this wasn't a problem, and the root filesystem was large enough

          Reading back the backup from /dev/nsrt0 tested correctly and everyone was quite happy with how fast the process ran

          Then one inquisitive user questioned how the backup "succeeded" in reading back without loading the tape ?

    6. Ken Hagan Gold badge

      Re: Ah....backups....

      "....and I still wonder how many systems have backups which have never been tested. Someone out there might be able to tell me."

      I'd put money on Thames Water for one.

    7. rcxb Silver badge

      Re: Ah....backups....

      ....and I still wonder how many systems have backups which have never been tested. Someone out there might be able to tell me.

      With the advent of LTO, all tapes are verified when they are written, so none are properly "corrupt".

      Verifying the contents are what you want is another matter... Unless you work for a company with programmers like I've got to put up with.

      * Despite repeated training on Unix file ownership and permissions, all files and folders are set: chmod 777

      * Critical files even in prod get deleted all the time. None of them have a clue who's program got bugged and clobbered the prod files... just restore from backup and get on with life.

      * When did the important folder on the dev system get deleted? "Oh, I dunno, last time I used it was 3 years ago."

    8. Anonymous Coward
      Anonymous Coward

      Re: Ah....backups....

      Back when I was responsible for the care and feeding of an ageing AS/400, I got to test our backups a couple of times per year, thanks to failing hard drives. Call IBM, arrange to meet the engineer with a new hard drive, grab the latest backups from the fire safe to get them prepped, and then IPL from 1/2" tape.

      I could just about catch a nap during each 1/2" tape. I think they took just under 1 hour per tape. The rewinding noise would snap me out of a doze. We had 4 1/2" tapes for the system, then the rest of the programs and data were on 8mm tape (just one tape for the balance of the restore, I think).

    9. Anonymous Coward
      Anonymous Coward

      Re: Ah....backups....

      It's also the kind of testing done on your backups. If you just test that all the data is there you may be surprised later on if you have to actually restore *everything* from backups as it could take a long time, with people unable to work for days. You might need to rely on your backups to actually work out how effective it is, and convince beancounters to upgrade.

      1. Anonymous Coward
        Anonymous Coward

        Re: Ah....backups....

        I have a backup that takes 26 hours, and yes, I know that I should split it. I don't know how long a full restore would take, but I have tested random restores and DO know that I can successfully restore from it.

  6. Anonymous Coward
    Anonymous Coward

    Tim's Cube

    That he alleged to have used to create the WWW Is on display at CERN. Quite a good family day out, if you're in Geneva and have some time to spare. We didn't get to see the tunnels when we visited, because they weren't open that day, but we did get to see the control room. (Catch the tram out from the city centre, and when you get there, make sure you get yourself booked on to a tour before they fill up for the day.)

    Also worth visiting is the Diamond Light Source near oxford, which does open days, and is (kind of) the same thing on a much smaller scale.

    1. Korev Silver badge
      Boffin

      Re: Tim's Cube

      Not far from CERN is the Musée Bolo in the EPFL Campus at Lausanne. It's free to enter and has lots of nice toys ranging from home computers to bits of massive supercomputers.

    2. FIA Silver badge

      Re: Tim's Cube

      What's all this 'Alleged' shit?

      From the article:

      One of Jobs’ creations was the Cube, a workstation-class machine that legend has it was used by Tim Berners-Lee to create a little thing you have heard of called The World Wide Web

      "Legend has it"???

      No, these things are facts. (You even linked to the computer in question).

      Can we stop with all the hyperbole. Seems Adam Curtis' 'Hypernormalisation' was depressingly spot on.

    3. PB90210 Silver badge

      Re: Tim's Cube

      So what is the 'Tim Berners-Lee's computer' in the Sci Mus in London?

      (haven't been for a few years, but one is on display in the computing area)

      1. Richard 12 Silver badge

        Re: Tim's Cube

        He's probably used and even owned more than one.

        1. Anonymous Coward
          Anonymous Coward

          Re: Tim's Cube

          Haven't we all...

          Personally, I'm currently using 3 laptops, and there are three dead laptops and a dead desktop under my desk...

  7. Bebu sa Ware
    Windows

    You know the answer in your bones :)

    "....and I still wonder how many systems have backups which have never been tested. Someone out there might be able to tell me."

    A lot, lot more systems than you would like to might imagine. :(

    Working in an institutions where "fat fingers" are the rule rather than the exception usually means the backups are frequently "tested."

    Archives weren't routinely tested even during thekr nominal retention period. Since migration of older archives to current media and tech wasn't a thing most archives ceased to have the means to be read before anyone could determine whether they were unreadable.

    You can get lucky though - I have had a complete restoration of archives from a DDS2 (DAT) tape ~15 years after they were made (the drive pulled out of the junk pile.)

    Hint: don't use vendor backup/dump utiities (eg vdump, vxdump or even xfs_dump) for archives as there is a good chance you won' t have a system (tru64, hpux, sgi) to be able to read them. Gnu tar (or cpio) has served me well here. You cannot expect to have good fortune of vmsbackup where someone once created a compatible tool. (I once made one chap's day recovering his files from a very old vax vms dump tape. I didn't even laugh when he asked whether we had a vms system with a tape drive. :)

    1. F. Frederick Skitty Silver badge

      Re: You know the answer in your bones :)

      Talking of proprietary backup formats, one of my first jobs dealt with data that was generated by the European Union. It arrived on half inch magnetic tape reels created on Groupe Bull hardware (the French had somehow fixed it so all EC systems had to use their kit). The data was a backup of various databases of things like treaties, laws and questions asked in the EU parliament (the questions only - no answers - so I'm not sure what the value of it was).

      The records were encoded with an ISO format. Except the Groupe Bull operators had "tweaked" it. We weren't allowed to have the documentation describing the changes they'd made, nor were we allowed to speak to the operators - all requests went through a liaison person who was totally non-technical and seemed to spend all of his time travelling between Brussels and Luxembourg. It took two years for us to completely work out the deviations from the standard.

      Fun times.

  8. Xalran

    We all learned the same way...

    The power of the rm -rf command.

    You're not a real unix/linux sysadmin until you have issued it in the wrong place and had to feel how powerful that command is.

    1. Doctor Syntax Silver badge

      Re: We all learned the same way...

      Big hint: mv can have the same effect and, yes, I discovered that the same way.

      1. disgruntled yank Silver badge

        Re: We all learned the same way...: mv and move

        Forty-odd years ago, Data General took the odd decision to reverse the sense of the arguments of the move command in its CLI. In the RDOS CLI, the arguments were evaluated sh style, with the destination directory at the end of the list. The AOS/VS CLI had the destination directory as the first argument. A fellow support guy, who hadn't worked on an RDOS machine in quite a while, dictated the command AOS/VS fashion to a customer, helpfully wiping out the recovery floppy. Neither party was happy with this.

        (Or so I remember the commands: it has been almost thirty years since I dealt with an Eclipse.)

        1. Doctor Syntax Silver badge

          Re: We all learned the same way...: mv and move

          So that's what happened to Data General.

        2. The other JJ

          Re: We all learned the same way...: mv and move

          ISTR from my brief exposure to them that IBM's AIX flavour of *nix for the RS/6000 had a few contradictory command line switches.

        3. Tim99 Silver badge

          Re: We all learned the same way...: mv and move

          I started with RDOS on a large scientific instrument that had a foreground/background Nova for data acquisition and processing. Initially it had a "loan" Phoenix fixed 5MB and a similar removable drive. I kept filling it. A few months later the "real system" was delivered, a huge (19" rack mounted) 25MB Winchester with an 8" 1.2MB floppy drive. A DG salesman came out to see why we had bought such a large system. As I recall, a low level format followed by a high level format and installing RDOS took most of the morning.

          A few years later its Tektronix terminal was replaced by an IBM AT as a terminal emulator with its own 30MB HDD. I could copy the data over from the DG and only had to archive the important stuff from that onto cheaper 5.25" 1.2MB disks.

          A later instrument used the desktop DG20, which had a tape drive. RDOS might have been "idiosyncratic", but it got the job done. Worryingly, I too can still remember many of its commands, but I'm not sure that I can remember what I cooked for dinner two nights ago.

          That instrument was replaced with a competitor's which, of course, used an MS-DOS computer.

    2. Tim99 Silver badge
      Coat

      Re: We all learned the same way...

      A really long time ago I did it as root console in usr...

      These days it would be a little harder - ssh into an admin account then - cd /home; sudo rm -fr (?).

  9. Just Enough

    Slap on the wrist?

    Not sure why "Miles" got a slap on the wrist. At worst all he should have been able to do is empty his own home directory, and have no-one to blame but himself. But the error that allowed deletion of everyone else's files is entirely the sysadmin's responsibility.

    1. Korev Silver badge

      Re: Slap on the wrist?

      Exactly this.

    2. tip pc Silver badge
      Holmes

      Re: Slap on the wrist?

      this all happened in the 90's, last century in new speak.

      the admin didn't know he needed to tighten things to prevent incidents like this.

      once it became apparent, the admin applied the appropriate controls.

      1. Just Enough

        Re: Slap on the wrist?

        Why were the "appropriate controls" in existence if not needed?

        User permission management wasn't some new concept introduced in the 90s.

        1. Phones Sheridan Silver badge

          Re: Slap on the wrist?

          “ User permission management wasn't some new concept introduced in the 90s.”

          It pretty much was. My uni in 1990 was exactly the same. Network of Suns, and what we discovered as students was pretty much no security at all. It took Sun about a year to lock the network down enough to stop us students from wreaking havoc, but remain useable for us as software engineering grads, because they had no real world experience of an entire network connected to the internet (JANet) either with each machine having a dedicated public IP directly connected to a router. I don’t think I even heard the word firewall for at least 5 more years.

          1. Peter Gathercole Silver badge

            Re: Slap on the wrist? @Phones

            I would be really surprised of this. I've been managing UNIX systems from many manufacturers (including Sun) for 40+ years, and in that time exactly no UNIX systems I administered ever created users home directories with group write, let alone world write permissions fresh off the tape.

            I will admit that it was possible to change the tools that created home directories to set wide permissions, but fresh off the tape, it would never happen.

            UNIX has had a well understood permission system and distinct users with their own home directories since at least Edition 6 (I can't go back any further, but I did use and administer Edition 6 UNIX systems). One thing I will admit. User management was more than a little ill-defined before the adoption of /home as the location of one's home directory (I've come across people using /u, /users. /u01 etc over the years, but still permissions were set correctly unless the admin. was a total numpty).

            If the permissions were too permissive, that would be entirely down to the system admin, and little to do with the supplier.

          2. DaveLS

            Re: Slap on the wrist?

            User permission management on Unix and other commonly-used operating systems (e.g., VMS) was around long before the 1990s, and didn't allow group writes by default unless local managers changed it. My recollection goes back to the early '80s for VMS and the mid '80s for BSD 4.2 Unix.

            Unix system vendors, including Sun, had plenty of experience of internet-connected systems in the US and elsewhere, years before JANET even switched to IP in the 1990s.

          3. Spamfast
            FAIL

            Re: Slap on the wrist?

            As with the other replies, I was using VT100-accessed System V'ish microcomputers & minicomputers in the Imperial Technology computer department in 1983 (although I was in the physics department) and nobody had even read access to others' home directories, let alone write. That's been UNIX standard since T,K&R created it to the best of my knowledge.

            The sysadmin in question or one of his predecessors would have had to change the user home directory creation stuff and/or default umask explicitly to make home directories world-write.

            I can see it happening if 'we're all friends here' for cooperative work but shared directories with sticky bits were available long before Sun/POSIX ACLs became common.

            I'd have arranged a user lynch mob, sorry, posse to chase down the sysadm responsible.

        2. Doctor Syntax Silver badge

          Re: Slap on the wrist?

          "User permission management wasn't some new concept introduced in the 90s."

          Of course not, but it all depended on what permissions were set. Write for group? or for all?

      2. Snowy Silver badge
        Headmaster

        Re: Slap on the wrist?

        It is also the last millennium :O

  10. Graham Lee
    FAIL

    Also learned UNIX on NeXTs

    My first "real" job was as a university sysadmin, it was in a Physics department, and the computers were NeXTStations, with a couple of cubes, and a couple of OPENSTEP PCs (and then a massive Solaris server because NeXT had already disappeared by then). My first "real" mistake was getting the arguments to tar the wrong way around, and writing an empty hard disk to our master image tape instead of writing the master image to an empty hard disk. Luckily, there was a backup of the tape…

  11. herman Silver badge
    Devil

    rm

    The Remark command.

    1. lglethal Silver badge
      Trollface

      Re: rm

      "rm -rf" that just means "Remark, Read Further", doesnt it?

      1. Sudosu Bronze badge

        Re: rm

        I always thought -rf meant Really F&#$ed

    2. mirachu Bronze badge

      Re: rm

      rm -rf, i.e. read mail really fast

  12. herman Silver badge

    RTFM

    Don’t do anything on a production system until you can read the UNIX man pages and actually understand them.

    1. Doctor Syntax Silver badge

      Re: RTFM

      You didn't mention remembering them.

    2. jake Silver badge

      Re: RTFM

      "Don’t do anything on a production system until you can read the UNIX man pages and actually understand them."

      You're a couple steps ahead of yourself ... instead, "Don’t give a user root access to a production system until the user has demonstrated they can read the UNIX man pages and actually understand them, AND demonstrate a full, working knowledge of the corporate (school, government, etc.) software package and how it is used within that particular corporation (school, government, etc.)."

      1. Anonymous Coward
        Anonymous Coward

        Re: RTFM

        That would have taken all of the fun out of life. I was introduced to Unix on an IBM RS-6000 but really learned Unix by repeatedly destroying a Honeywell XPS-100, and its Bull replacement... and rebuilding them from tape many times.

        Later, I spent many days building CentOS 5 vms because some dumb bastard at Red Hat decided that bright red was an excellent colour choice for their logo to be displayed during installation. All I knew was that some red text, meaning an error had occurred, flashed by during installation...the first 32 times I performed the install. I vowed to never forgive that bastard, and I never will.

    3. Anonymous Coward
      Anonymous Coward

      Re: RTFM

      You're giving man pages too much credit. Many of them are useless. Some are worse than useless.

  13. Pete 2 Silver badge

    Prisoners here, of our own device

    > learning how to clean things up in Unix was a good place to start.

    Similar to a rather panicked call we got from a client.

    The person in question must have been feeling bored. He was looking through /dev and saw hundreds of files he wasn't familiar with. Since many had zero size, he decided they were obviously useless, so he deleted them. After all if they are 0 bytes, it isn't as if they held any vital information.

    The first thing we told him was NOT to log out! After that we spent a fun (not) hour rebuilding all the devices that had disappeared.

    Surprisingly, after all that his machine worked as intended. Nobody knows what he told all the users he was responsible for. But no doubt he kept the praise for restoring normality all to himself.

    1. Mike_R
      Windows

      Re: Prisoners here, of our own device

      Similar episode on VMS (this was before OpenVMS renaming)

      On a remote system (lawyer's office as I recall) with one user responsible for changing tapes, and unfortunately with system access.

      User noticed there were identical files in SYS$COMMON and SYS$SYSTEM (IIRC the directories names correctly, it was three decades ago)

      So obviously one set was redundant, and could be deleted, and was.

      Unfortunately the files were the the same files, linked to two different directories.

      Great opportunity to test a system restore (or re-install - memory fails me...see icon)

  14. OllieJones

    I did rm -rf / once

    I did rm -rf / once, by mistake. (I wanted to do rm -rf .)

    It was on a workstation product I was helping to develop and test, so no big loss except of install time.

    It was really interesting how long the OS kept running after I did that. UNIX and its descendants don't delete open files, so the thing kept right on working, if I remember right, until I tried to start another shell.

    1. MrBanana

      Re: I did rm -rf / once

      It is a rite of passage. Either yourself with your head in your hands, or watching some hapless colleague furiously thumping Ctrl-anything on the keyboard to no avail. It can be surprising just how long a system will struggle, and yet still stay up - not to do anything useful though. At best you may have a root shell somewhere that will accept "exec halt" - no forking allowed - to put it out of its misery. Then it is the dance of the system backup tapes, or "why are there no backup tapes" if you're having a really bad day.

    2. Gene Cash Silver badge

      Re: I did rm -rf / once

      Linux doesn't do that. The moment libc6 goes away, you start getting lovely messages like: -bash: /bin/ls: cannot execute: required file not found

      Don't ask how I know...

      1. Pete 2 Silver badge

        Re: I did rm -rf / once

        > bash: /bin/ls: cannot execute

        In that same situation, real hackers use bash's built-in echo *

        Don't ask how I know ;)

      2. doublelayer Silver badge

        Re: I did rm -rf / once

        A lot of the time, /bin/ls is going to get deleted before glibc does. So either way, that command isn't going to be available. A few things will still work. Fortunately, my run of this was not a result of an accident. I had a machine I used for an experiment and I was going to wipe it anyway, so I took the opportunity to run the command, let it finish, and see what could still be done.

  15. Doctor Syntax Silver badge

    “I could sit in comfort writing papers using LaTeX in emacs,”

    Not my idea of comfort.

    1. TRT

      Would your idea of comfort be sitting in latex writing papers in emacs?

      1. KittenHuffer Silver badge
        Gimp

        You forgot the icon best suited (pun intended) for 'sitting in latex'!

    2. Sudosu Bronze badge

      Maybe they meant wearing latex?

  16. bobkn

    Homed in, not honed. Dammit.

  17. wsm

    After restoring the files?

    Permissions should have been the first thing to be modified, if not the entire approach to staffing and training anyone with access to any part of the systems.

  18. JugheadJones

    recursive chown

    I gave access to a trusted developer once and he accidentally did a recursive chown -R from top level. He forgot he was sudo'ed to root and owned everything from / instead. I tried but this again was as bad as a rm -rf /, and would take too log to fix, although most thing are owned by root:root, there are some nobody,tty etc there as well (solaris days). Rebuild required. this was prior to zfs , if that had been around, you could easily rollback to a snapshot in single user mode

  19. martinusher Silver badge

    Been There, Done That

    It was a Xenix system in the early 90s. It was used for development and was running short of disk space. The easiest way to free up space is to remove things like program list files. "rm -rf *.lst", work of a moment. Unfortunately the keyboard was crap (it was only used occasionally) so what I typed was "rm -rf * .lst" -- note the space. I noticed the problem immediately and stopped the operation but not before I'd deleted the system files etc.

    The good news is that the system was still up and running and it was about six weeks before a power glitch caused to reboot and fail. The lessons for me, the computer 'expert', were....

    -- Don't do stuff at the top of the directory tree, especially when you're 'root'.

    -- Watch what you're typing; bash has different rules than the DOS type shells we all know and love

    -- File extension in 'ix' are merely part of the filename.

    Anyway, lessons learned.

    1. Anonymous Coward
      Anonymous Coward

      Re: Been There, Done That

      "the DOS type shells we all know and love"

      Buh? I take Windows command line's slow creep towards *sh-ness as an absolute win.

      1. Anonymous Coward
        Anonymous Coward

        Re: Been There, Done That

        Can their next update introduce sleep.exe again, please? Everything slows down when you do it the old DOS way, with a null for loop...

  20. Boris the Cockroach Silver badge
    Unhappy

    Who remembers .dll hell?

    Yupp the dreaful memories when programs installed their own versions of core operating libraries... much to the delight of the admins/users.

    So you delete the offending file and reboot......... and bugger

    Now you're for it

    (Luckily for me the admin had a DOS floppy boot disk, and the knowledge to get the correct replacement .dll from the win95 cab files , and move it to whatever /windows/systems folder it lived in)

  21. Martin Howe
    Joke

    Never mind the WWW, a NeXT system was used to make something far more useful: DOOM :)

  22. DS999 Silver badge
    Facepalm

    Sounds like this is the sysadmin's fault not his

    Either he had a uid 0 account or /home was group writable. Either is an enormous sin for a sysadmin, even back in the days of NeXT.

  23. bill 27

    Just for fun...

    Last week I had a computer I was going to give away/toss in the trash. So I reproduced a situation, from several decades ago, just to see if it would do the same thing.

    As root I:

    cd /

    rm -vfr *

    I'm nosy I like to see what's happening...but basically it just locked up after a while. The time before at least it said something at the end...I got a kernel panic.

  24. Luiz Abdala
    Joke

    I don't know, I like windows using *.TMP files now.

    For some odd reason, Windows not deleting temp stuff, and an unknown amount of years later making a tool to delete temp files, feels much better than placing a flavor of rm -rf on a script.

    Making files with a *.tmp extension feels like a premium feature now.

    I wish I could put the Joke Icon, the Windows User icon, and the Coat Icon, at the same time.

  25. Stuart Castle Silver badge

    A friend of mine used to do tech support for a small budget software company who did cheaply made games for a couple of quid a go. Bit like Mastertronic, but much smalller budgets. In fact, they did a football game whose main advertised feature was digitised sound (in fairness, this was a big thing in the late 80s/early 90s). Their digitised sound was the MD and his PA standing at one end of the office, and yelling various cries, including "Palace" into a Mic that was set up at the other end of the office, Bit of a bummer if neither of the teams playing on screen was Crystal Palace, but the cries were picked randomly, so that did happen.

    One day, my friend got a panicked call from the PA saying the MD's machine had stopped working. When he got there, DOS was working fine, but Windows wouldn't start (it couldn't find win.com). When my friend looked, the Windows folder had vanished.

    It turns out that in an attempt to keep his computer tidy, the MD kept all the Windows applications in the Windows folder.

    He either didn't consider that Windows would need a lot of system files, or thought Windows would protect them. Windows 3.1 offered NO protection of any sort, so when he dutifully went through the Windows directory deleting all files and sub directories in DOS, he was able to do so relatively easily. He was just a bit surprised when Windows failed to start and was actually quite angry when my friend had to re-install Windows and every application on it.. Not an easy or quick job when everything was on floppies. Yes, they were cheap enough they had individual PCs, so didn't have a nice file server somewhere hosting all this stuff.

  26. AbeSapian

    Does Anyone Remember Xenix?

    I didn't need a powerful command to destroy a system - just the OS and a little help from the office power.

    A company I worked for put me on a project in which they decided that the user workstations would be X86 boxes running Xenix. I found out, one day, how fragile Xenix is as an operating system. One of the workstations in my test group lost power, Needless to say, it didn't shut down clean. When I started the box up again, the Xenix boot sequence began to analyze every file on the system. I watched in horror as, one by one, the files were analyzed and deleted - leaving me with an absolutely pristine hard drive that had to be rebuilt from the ground up.

    Ah, those were the days.

  27. Matt C.

    Especially dangerous in modern systems...

    ... that use flash rather than true ROM for their BIOS.

    And systems that have knowledge of that fact, along with a full driver implementation of said flash IC, will quite happily delete their BIOS if you ask them to.

    As a friend of mine found out whilst working on an embedded system.

  28. billdehaan

    Undergrad? Pfft. Try system administrator

    In the 1980s, I contracted at $COMPANY, a defence contractor. It was like a Dilbert strip, co-written by Franz Kafka.

    They worked exclusively on closed tender cost plus contracts. For those unfamiliar, "cost plus" means the DOD/DND says "build this", $COMPANY says "we don't know how, or what it will cost", and DOD/DND says "build it anyway, we'll pay for all costs plus 15% profit". It's zero risk, and guaranteed profit.

    If the company spends 10 hours doing a job, they get paid for 11.5. If they spend 100 hours, they get paid for 115. If they buy an Ada compiler for $20,000, they get reimbursed $23,000 for it. If they buy a different Ada compiler for $100,000, they get reimbursed $115,000, etc.

    You can see the incentive structure that was formed. The company was filled with lifelong employees who considered this all perfectly natural. The company culture reflected this, and inefficiency was not only not punished, it was rewarded. Competent employees actually cost the company money. A few were required so that at least some projects shipped, but incompetent employees where the company's bread and butter.

    Enter... the system manager. Or "System Mangler", as he was commonly referred to.

    It was a VAX shop, and the SM administered it. His administration skills were already legendary. It was perfectly normal to log in and discover that the C compiler had disappeared. Or that the disk pack had only 3kb of disk space (for 40+ developers). Or that you no longer had write, and sometimes even read, access to your own files. There were two printers, one being a laser and the other a typewriter ball "high quality" printer. Files sent to one printer routinely went to the other, resulting in dozens of dead trees worth of printed gibberish.

    The SM did the weekly backup at high priority during the workday on Friday. This mean locking all files, so users could do no work until he completed the multi-hour backup. Of the swap volume. And if the backup didn't fit, he'd delete user files to free up space. When users demanded their deleted files back, the SM would explain that he deleted them before the backup, because "otherwise the backups take too much tape", and there was only a small number of installation tapes to do the backup on.

    So when we got a contract that stipulated Unix, we got an Apollo unix machine for users to telnet into. And clearly, SM was the man to administer them.

    Work proceeded smoothly for about six weeks, as the unix-capable devs just coded away quietly. But then the SM decided it was time for him to learn unix. He logged in, as root of course, and started running a "how to unix" introductory tutorial.

    It took him 17 minutes before he reached the "rm -rf /*" step of his introduction. All user files were gone. Unix was gone. The entire disk was wiped clean, as he repeatedly answered "y" to all "are you sure?" prompts. Naturally, he blamed the OS for allowing him to do it, and management was happy with that answer.

    So, when the replacement Unix installation tapes arrived two weeks later (he'd overwritten the originals with VAX backups, of course) and the Apollo had a clean install, the first thing the lead dev did was change the root admin passwords, and locked the SM out of the system.

    When the SM complained that he wouldn't be able to administer the machine, he was told that was the point.

    When the SM pointed out that meant there would be no backups, the lead dev said "we had a meeting about that. the team voted unanymously that we'd rather work without backups than let you near the system again"

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like