back to article A cautionary tale of virtual floppies and all too real credentials

With the weekend gone, like the contents of a file share after a hasty execution of a seemingly innocuous script, pause for a second before tackling the week's shenanigans with another Reg reader Who, Me? moment. Our reader, Regomised as Dave, told us of his time working in the IT department of a bank "at the end of the …

  1. DJV Silver badge

    As soon as...

    ..I saw the line "my application first deleted all files and directories (recursively)" I knew we were in for some "fun"!

  2. jake Silver badge

    Waddayamean "nowadays"?

    "and nowadays you don't even think about giving anyone access to your environment."

    Even in those days I would have opened an account with appropriate privileges for the interloper visitor to muck about in[0]. What's the point of having separate accounts if you share them willy-nilly?

    [0] With logging. Lots of logging. All of the logging. Paranoid? Me? Not yet, but I was getting there ...

    1. Anonymous Coward
      Anonymous Coward

      Re: Waddayamean "nowadays"?

      He was working at a financial institution. You couldn't just go creating user accounts willy nilly.

      Getting a user account setup in that kind of institution was a hugely bureaucratic process involving lots of forms which had to filled in and passed around and signed off by lots of people. It took about two weeks minimum.

      1. MiguelC Silver badge

        Re: Waddayamean "nowadays"?

        Not so sure about that. When started working for an investment bank in the early noughties, I recall users regretting how IT had lost the freedom to do things like in the "old times" (i.e., early to mid 90's) where they would sit alongside the developer who would hammer changes directly in the production environment and the users would test along until everyting was OK (with appropriate changes to accounting tables, etc., as needed to correct any previous fumble). So, yeah, a new user account might take 2 weeks of paperwork, but really dangerous stuff was readily available.

        1. Anonymous Coward
          Anonymous Coward

          Re: Waddayamean "nowadays"?

          There is a UK bank where the business side told IT they didn't want a new system to update reference tables, because it would be slower and require logging of changes.They quite like being able to go in and amend production data themselves, direct access to the data warehouse.

          This is in 2020.

          1. jake Silver badge

            Re: Waddayamean "nowadays"?

            I'm pretty sure you should let the authorities in on that one, before you are implicated for aiding and abetting.

          2. anothercynic Silver badge

            Re: Waddayamean "nowadays"?

            I hope the message from IT to 'the business' was "bugger off" and a report to the compliance officer!

        2. ICL1900-G3 Bronze badge

          Re: Waddayamean "nowadays"?

          Yep...I did that, in a well-know US charge card company. Insert new code in hex then insert a branch instruction to new code...usually worked. But not always.

        3. MOH

          Re: Waddayamean "nowadays"?

          So, fully Agile?

      2. MarkET

        Re: Waddayamean "nowadays"?

        Used to work for a financial software developer - even seniors weren't allowed near the production environments, both physically and 'logically'. Several layers of test / verification and re-test etc.

        1. jake Silver badge

          Re: Waddayamean "nowadays"?

          Nonetheless, people DO have root access to production systems.

          There is a big difference between "I can do that" and "I will do that".

          1. Down not across Silver badge

            Re: Waddayamean "nowadays"?

            There is a big difference between "I can do that" and "I will do that".

            Ain't there just. I excercise that on a daily basis. Plenty of "I could, but..." and also lot of my favourite "What is it that you are actually trying to accomplish?" Quite often it turns out they don't need what they're asking for at all.

            1. alanh02

              Re: Waddayamean "nowadays"?

              I also ask "Why?" Cue nervous shuffling of feet.

      3. Anonymous Coward
        Anonymous Coward

        Re: Waddayamean "nowadays"?

        lots of places new users still take weeks to action.

        A few jobs back took 3 month for my access, 3 long months basically twiddling thumbs 8 hours a day.

        so glad to do some actual work!!!!

        1. Arthur the cat Silver badge

          Re: Waddayamean "nowadays"?

          lots of places new users still take weeks to action.

          In many of the places I've worked that happened because neither the hiring manager nor HR bothered to tell IT about the new person, even if it was three months between job acceptance and arrival. The first IT knew about a new person was when their manager appeared with them in tow and said "meet X, our new hire, where is his/her machine?". "New machine" usually meant a Unix workstation in those days, so not something to easily get delivered the next day.

          1. Anonymous Coward
            Anonymous Coward

            Re: Waddayamean "nowadays"?

            That happens far too often, and is usually down to utterly incompetent HR or management and a lack of even the most basic of HR processes such as New Starters, or following them should the happen to have them.

            Occasionally it's worth being slow just to prove a point so the offenders remember to follow the processes that they often had a hand in writing.

            On the other hand, I've had compliments that new starters have everything in place on the IT front with plenty of time to spare and no drama. It's almost as if they expect these things to be painful - they shouldn't be as new staff starting and staff leaving is a standard part of any organisation.

        2. jake Silver badge

          Re: Waddayamean "nowadays"?

          Yes, lots of places are incompetent. That's why you'll see so much moaning about corporate bullshit in place like this.

      4. jake Silver badge

        Re: Waddayamean "nowadays"?

        First of all, you are ignoring the fact that the narrator of the story gave the visitor from Corporate the narrator's own root access to a live system. That is a big no-no, and has been since the 1950s, or perhaps earlier.

        Secondly, yes, getting a user account can be difficult because bureaucracy. However, a systems administrator providing a tool for a Corporate troubleshooter isn't all that difficult. The dialogue goes something like this:

        Me: This is $guest_admin from Corporate. I need you to sign this paperwork, authorizing me to create him an account with the permissions listed.

        $BOSS: Excuse me? $CHANNELS!!! Weeks & months!!! It'sHowWeDoIt[tm]. G'way, I'm busy ::shifts gaze back to porn he thinks I don't know he's viewing::

        Me: I'll send him back to corporate, then, with your name & number as the contact to explain the refusal to let him do his job.

        $BOSS: ::sputter:: Excuse me?

        Me: You heard me.

        $BOSS: Where do I sign?

        There is always a way around corporate roadblocks to getting things done. It's my job to know them ... and to have the cajones to implement them as necessary. That's what they pay me for.

        1. chas49

          Re: Waddayamean "nowadays"?

          The narrator of the story *was* the visitor (from Corporate or wherever), it was the local sysadmin who gave him the login details and access to the machine. So Dave wasn't at fault.

          Your general point still stands though.

          1. jake Silver badge

            Re: Waddayamean "nowadays"?

            Mea culpa. I was running late for my post-lunch siesta, tends to scramble the circuitry.

          2. Olivier2553

            Re: Waddayamean "nowadays"?

            Nor did Dave ever requested anything with such power, he only needed a machine with a floppy.

    2. NBCanuck

      Re: Waddayamean "nowadays"?

      You're right. Nowadays I wouldn't think about it....as in wouldn't consider giving it any though at all beyond forming the process required to say "Uh....no. Not happening".

  3. ColinPa

    Here is one command you dont want to type in

    I remember being in the machine room late one night, and I was chatting to one of the bored operators. He was showing off his knowledge, and what he could do with his all powerful userid. He said, as he typed - this is one command you never want to issue "PURGE SYSTEM ALL", then automatically pressed enter!.

    Whoops.

    Next morning when I came in there was a logon message for all users "Due to a technical fault - all spool files were lost last night".

    Over the next few weeks they implemented very granular command security, so it could not happen again.

    1. jake Silver badge

      Re: Here is one command you dont want to type in

      "then automatically pressed enter"

      Sometimes muscle memory is a bitch. I think we all have similar stories.

  4. chivo243 Silver badge
    Go

    under his usercode

    Whoops! That sysadmin who gave his creds should have been shown the door.

  5. Richard Gray 1
    Facepalm

    Networked office

    Remember the days when programs would be run from network drives to save the space on the local hard disks?

    I'll never _ing forget... As a fresh faced PFY I had a problem with my local PC office, and it was suggested that I un install and re install.

    Unfortunately with my rights it decided to completely un install office, including the networked version that everyone was using.

    Fortunately quickly fixed with the /admin install back to the network location, with a lesson well learned that could have gone a LOT worse.

  6. GlenP Silver badge

    For various reasons connecting to the file server at our Australian office is most easily accomplished via NET USE.

    I always follow this by DIR W: just in case I've connected to the wrong share!

    1. Prst. V.Jeltz Silver badge

      That wouldnt have helped the guy in the story.

      he'd have seen what he expected to see at the last folder in the mapped drives path.

      Then his software deleted from W:\ ( as opposed to W: )

      [edit]

      Although if he actually went to w: in cmd he might have noticed the full path , as noted in the story.

      1. Prst. V.Jeltz Silver badge

        ha , i said "folder" . Iv'e gone all user.

        Are we calling them folders now? or sticking stubbornly to 'directory' ?

        1. Anonymous Coward Silver badge
          Linux

          If you're typing in a shell, terminal or command prompt (or actual DOS), they're directories.

          If you click to access them they're folders.

          Simples.

          Proven by the fact that to list the contents of one you type "dir", not "fol"

          1. Loyal Commenter Silver badge

            Proven by the fact that to list the contents of one you type "dir", not "fol"

            ls

            1. phuzz Silver badge
              1. jake Silver badge

                Can I get a listing of empty directories with gci?

                1. phuzz Silver badge

                  You need to pipe the output:

                  gci -Directory | where { $_.GetFileSystemInfos().Count -eq 0 }

                  You might want to chuck a -recurse in there depending on what you need.

                  (I amuses me how many downvotes you can get just by reminding people that Powershell exists :)

            2. John Brown (no body) Silver badge

              *CAT for a disk catalogue

              (Ex BBC user here :-))

              1. Little Mouse Silver badge

                "LD" (although I forget which arguments)

                Primos - FTW!

                1. Ken Shabby

                  Which of course is for ‘List Directory’

              2. Annihilator Silver badge

                LOAD "$",8

                Ex C64 user

                1. DJV Silver badge

                  cA - ex PET user!

            3. Anonymous Coward
              Anonymous Coward

              "ls"

              I dont get it .

              16 upvotes for knowing the unix for dir and crowbarring it awkwarly into some elses joke.

              people just lurrrv the nix

              1. jake Silver badge

                It wasn't crowbared awkwardly, nor was it a joke. Rather, it was a gentle correction. It would seem that folks approve.

                1. Loyal Commenter Silver badge

                  Thanks @Jake.

                  It was actually a gentle reminder that not everyone uses dir to list the contents of a folder / directory / inode / whatever terminology you favour*.

                  My life in IT doesn't personally date back that far, but I believe the use of ls predates dir by some decades, originating in Multics (the predecessor to Unix and all modern *nixes). dir I believe originates in CP/M, and became widely used with the advent of MSDOS. Wikipedia assures me that Muiltics predates CP/M by a decade or so, and MSDOS by almost two. The history of such things is a bit more twisty-turny than this, but that is it in a nutshell.

                  The vast majority of command-line jockies certainly used to be using *nix type systems in preference to DOS-type ones (I don't believe CP/M ever really got to being a well adopted OS), so really what I was insinuating is that for those who think that those file-containing structures should be called directories because that's how the DIR command is named should perhaps broaden their horizons a little.

                  FWIW, the first OS I grew up with used CAT as its command, to CATalogue the contents of a file system. On a C90 cassette.

                  *My opinion here is that in computer science, the terminology is secondary to the actual concept. As long as others can understand what you are talking about, it doesn't matter. See also the pointless column / field debate in database nomenclature.

            4. Anonymous Coward Silver badge
              Linux

              > Proven by the fact that to list the contents of one you type "dir", not "fol"

              > ls

              ln -s /usr/bin/ls /usr/bin/dir * (not actually needed on the CentOS system I'm using here)

              dir

              1. Scoured Frisbee

                > ln -s /usr/bin/ls /usr/bin/dir

                Why are you running as root? You never know what might happen. I bet there's a "Who, Me?" somewhere with such a story.

  7. John Riddoch

    FTP config files...

    While at uni, I was doing something where I had to FTP some files to or from one of the Unix servers. After typing in the server name to the FTP client GUI, I spotted the server name already having a configuration defined. With the username "root". With a password filled in. I looked at my mate. He looked at me. I hit "connect" and promptly got logged into the server, as root. Rather naively, I tested the access by downloading /etc/passwd, deleting it and recreating it (in retrospect, this was incredibly dangerous. I didn't know Unix that well in those days...). Evidently, the shared FTP client config had been used by the lecturer at some point and he'd saved the root password for ease of use, not realising it was available to all the students. We reported it to the lecturers and it was removed from the config soon after that...

    Worth noting this was about '97 when security processes weren't as strict as they are these days. FTP as root? *shudder*

    1. Must contain letters

      Re: FTP config files...

      err, you mean there are other users?

    2. jake Silver badge

      Re: FTP config files...

      "Worth noting this was about '97 when security processes weren't as strict as they are these days."

      Excuse me? Security these days is extremely lax compared to what it was in the 1990s. Shall I start with BYOD? Open access to facebook, twitter, inistagram and whathaveyou? To say nothing of "clouds" losing corporate data all over the place ... Security? I'm not all that certain that the kids running corporations these days have even heard of it!

      Do you not read ElReg?

    3. Loyal Commenter Silver badge

      Re: FTP config files...

      FTP as root? *shudder*

      FTP. *shudder*. Let's start with the lack of encryption there shall we?

      Those were more innocent days, where nobody considered the possibility of connection eavesdropping and MitM attacks.

  8. Prst. V.Jeltz Silver badge

    I'd forgotten all about that map 'root' thing.

    It happens by default these days that the path on the mapped drive starts at the mapped location

    Seems silly that a map would show you the full path of the destination machines folder - IN the path of the mapped drive rather than 'behind' it

  9. juice Silver badge

    I've mentioned it before...

    But I've known at least one person who's accidentally ran a SQL command with the semi-colon in the wrong place.

    I.e. DELETE FROM blah; WHERE id = 5

    That required a bit of cleanup...

    1. Loyal Commenter Silver badge

      Re: I've mentioned it before...

      I know of more than one person here who has run a SQL DELETE command and forgotten to add the WHERE clause. With client data.

      This is why, when working with anything remotely like a live database, the series of steps goes something like this:

      1) BEGIN TRANSACTION

      2) SELECT * INTO Affected_table_backup_{today's date} FROM Affected_table (probably best to have backed the database up first too if possible)

      3) SELECT * FROM Affected_table WHERE {delete criteria} (to make sure you are getting the right records)

      4) DELETE FROM Affected_table WHERE {delete criteria} (by amending the SELECT statement to make sure the WHERE clause doesn't get lost)

      5) COMMIT TRANSACTION

      1. Claptrap314 Silver badge

        Re: I've mentioned it before...

        I got downright paranoid about the use of BEGIN TRANSACTION ; SELECT * INTO table_backup ... when my job was to fix data in the production database.

        One day I had a conversation with my sysadmin along the following lines:

        "If you did not have to get involved, it did not happen, correct?"

        "Yep, that's right."

        Could have had it more than once...

      2. Nick Ryan Silver badge

        Re: I've mentioned it before...

        It's just a shame that MS-SQL's interpretation of SQL transactions is so broken.

        Therefore all DELETE or UPDATE statements are thoroughly implemented as SELECT statements prior to this, along with multiple, possibly unnecessary, clauses just to be sure.

        1. Down not across Silver badge

          Re: I've mentioned it before...

          It's just a shame that MS-SQL's interpretation of SQL transactions is so broken.

          Transactions in Transact-SQL are not broken (disclaimer: I'm thinking Sybase here, but that is where MS SQL Server has evolved from). Sure you may get bitten by unchained mode being the default where you have to either set chained mode or explicitly begin a transaction.

          I actually like the flexibility of unchained mode, you just have to be aware of what you are doing as it doesn't molly coddle you like say Oracle.

        2. Loyal Commenter Silver badge

          Re: I've mentioned it before...

          It's just a shame that MS-SQL's interpretation of SQL transactions is so broken.

          Broken in what way? It certainly doesn't break ACID.

          Nested transactions can be a bit of an oddity (only the outer transaction ever gets committed, which is arguably correct), but if you are using those... why?

      3. veti Silver badge

        Re: I've mentioned it before...

        If you've got time to get through all that, it's not a "live" database.

        If I tried that at my old job, before I was halfway through making the backup, users would be on the line demanding to know why the system wasn't responding. (Because the table was locked by my transaction.)

        Use SELECT * FROM table WHERE condition to identify the data you want. Then draft another SELECT for a test sample to include both some of the records you want to delete, and some of those you don't.

        Once you're satisfied with both of those:

        BEGIN TRAN

        SELECT * INTO backup_table FROM table WHERE condition (backup only the records you want to delete)

        DELETE t FROM table t JOIN backup_table bt on t.id_column = bt.id_column (converse of previous step - delete only records that have been backed up)

        SELECT * FROM table WHERE test_sample_condition

        ROLLBACK

        Check the results returned by the test sample, and take a look at the backup_table as well. When you're satisfied with those, change the ROLLBACK to COMMIT and run it for real.

        1. Loyal Commenter Silver badge

          Re: I've mentioned it before...

          If I tried that at my old job, before I was halfway through making the backup, users would be on the line demanding to know why the system wasn't responding. (Because the table was locked by my transaction.)

          You forgot the (NOLOCK) clause; a little dangerous in itself, as this may include records from an uncommitted transaction another user has open (i.e. so called "ghost records"), but you are making a backup for the sake of safety, so including those records isn't the biggest issue if you need to access that backup to unfuck the data because you didn't follow the other steps properly.

  10. Anonymous South African Coward Silver badge

    Reminds me of the time I went for a network engineer class

    We did the whole thing - made a coax network from scratch (when coax ruled), and did a netware 3.12 install on a file server

    Everything went swimmingly well, BUT we all had superuser rights everywhere

    Then somebody inserted a Very Naughty Floppy Containing A Fun Virus into his workstation (DOS at that time) IIRC it was one which infects .COM and .EXE files, can't even remember the bastardly virus' name.

    Said virus took everything down except the Novell NLM binaries. Luckily it was a fairly new install, so we wiped everything, did a full reinstall of Novell, and set permissions accordingly. No more issues.

    1. hayzoos

      set permissions accordingly...

      Sounds like the perfect setup to learn of Netware's execute only flag. Seems like the thing to do until you find out execute only means strictly execute only. This was the era when patches were in effect diffs written to files. Remember writing to the file is verboten when execute only is set so no patchy by viruses or updates. Any other action was prohibited on an execute only file. There was a secret incantation which could make such an execute only file mortal again, I just do not remember it.

      1. jake Silver badge

        Re: set permissions accordingly...

        "a secret incantation"

        If I recall correctly, execute-only wasn't reversible. You had to copy the file, nuke the original, and then rename the copy. You needed pretty much all the permissions to do this under Netware.

  11. heyrick Silver badge

    Just a shame it took the near takedown of the bank's branch to learn them.

    If you don't come within a hiccup of filling your pants, you won't learn.

    It's the terror that drives the point home.

    1. Anonymous Coward
      Anonymous Coward

      Re: Just a shame it took the near takedown of the bank's branch to learn them.

      True. How I don't miss that feeling of your entire digestive tract compressing to the size of a golfball when the impossible happens so predictably.

  12. Eclectic Man Silver badge
    Happy

    rm - r *.*

    I've only used it for real once. There was a graphics application on Sun Workstations (remember those?) called SunAlis. It was awful. It had to be installed with root privileges, so every print job was sent as if from root, rather than a standard user (who could not remove it from the print queue as is had root authority). It caused no end of trouble, not least because when files got 'large' around 2Mb the app froze and lost it all. It was a good day when I logged in as root, went to the SunAlis directory and typed

    rm -r *.*

    and watched it all slowly disappear.

    1. Luiz Abdala
      Mushroom

      Re: rm - r *.*

      "Things that bring utter terror when used against you cause immense joy when used in your favor."

      Maybe someone famous quoted that before me... I don't know.

      1. Eclectic Man Silver badge

        Re: rm - r *.*

        The Duke of Wellington on the British soldiery:

        "I don't know what effect these men will have upon the enemy, but, by God, they frighten me. "

        Courtesy of:

        https://www.napoleonguide.com/aquotes_welli.htm

    2. spuck

      Re: rm - r *.*

      I once made a similar oopsie. Running as root (of course) I needed to delete some directories with names like ".tmp" and ".tmp2". That was the day I learned that "m -rf .*" also included "..", all the way up the directory tree.

      1. Eclectic Man Silver badge

        Re: rm - r *.*

        Umm, not sure I made this clear, mine was deliberate, not an 'oopsie'.

  13. Cynic_999 Silver badge

    Test environment?

    In my company, developers do not have any access whatsoever to any live servers or databases (apart from the server used by the development department itself of course). All new code gets put onto a test server first. Only after the technical director himself has done a final test of the code on the test server does he, personally, put it onto the live server(s). If someone trashes data on the the test server, a new test server is simply cloned from the relevant live server (although sensitive databases will be replaced with dummy databases).

    I would have thought that a bank would have a similar process.

    1. Anonymous Coward
      Anonymous Coward

      Re: Test environment?

      We do exactly that to test our stuff. Obviously.

      Worst case scenario, you forgot you were in the test database, and everything you did the whole day gets wiped along with the development database.

      Luckily I had detailed instructions and copies of everything I was meant to do that day... and Groundhog Day ensued the following morning.

    2. Anonymous Coward
      Anonymous Coward

      Re: Test environment?

      Oh yes they do, since more than 25 years. The tale was "at the end of the eighties or the early nineties"...

      Now they have DTAP environments and strict tests & procedures to get anything into Prod, and still it goes wrong from time to time. Complexity has advance as well...

    3. Eclectic Man Silver badge

      Re: Test environment?

      If only VW had done the same for their engine management software ...

  14. Andy Denton

    Not sure why all the panic.....

    ..... this was Novell Netware. A simple salvage command from the same terminal would have restored the deleted files. Got me out of a few scrapes back in the day.

  15. DBA-ONE

    Years ago, I worked with a proprietary insurance claims app that used a file-based database consisting of data (.dat) and index files (.idx). When we would have corruption at a client's site we would make a copy of the data file and rebuild the index. It was common to copy the data file to .bak. One day, a client was running low on space and instead of doing del *.bak I did del *dat. That command ran fast and purged all the data in seconds. That is the worst thing I've done. The second was in a system that was getting close, but not yet in production. As part of a refresh for an Oracle schema I would drop all objects and run my conversion and supplemental scripts. One of these scripts dropped all sequence objects. Well, I was logged in as the sys user, and when you drop these you may as well get ready to reinstall the software because you can no longer log in.

    After that, I changed how I connect to any database. For Oracle, I tweak the glogin file to display connection info and if I change user I completely close the session and start a new one. For SQL Server, I never connect even if the last connection prompt is what I want. I close that and select the desired server from the registered server list. This forces me to think about what I'm doing.

    1. Anonymous Coward
      Anonymous Coward

      All good. But some lessons have to be learnt by experience. Even if you'd read all about those precautions, it might not have changed your current working patterns, until That Moment actually happens, and you gain 20 years experience in 5 seconds.

  16. Danny 2 Silver badge

    Panopticon

    I worked in SWIFT and they are pretty unhackable. In my place they had one internet connected PC, not connected to any other internal computer and with it's drives removed and it's ports literally cut out. Plus it was on CCTV, as were most places there. And they analysed your toilet material for drugs or whatnot. And body scanners at the door to check you weren't taking or bringing CDs or drives.

    Every security scare about SWIFT is always about the banks that connect to it, it itself is pretty much the perfect panopticon.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021