back to article Fired credit union employee admits: I wiped 21GB of files from company's shared drive in retaliation

On Tuesday, a woman from Brooklyn, New York, pleaded guilty to destroying computer data at an unidentified credit union from which she had recently been fired. Juliana Barile, 35, according to charges filed by the US Attorney's Office in the Eastern District of New York [PDF], was working remotely at the credit union on a part …

  1. Boris the Cockroach Silver badge
    Facepalm

    Rather moronic

    Not to revoke access when you fire someone.

    Even more so to delete your ex-employers data and then boast about it.

    Sheesh

    1. Potemkine! Silver badge

      Re: Rather moronic

      Even more moronic is the fact not having a daily backup of the data their business relieves on.

      Requiring $10,000 to restore 21 GB shows shows this company has a clear data management problem.

      1. J__M__M

        Re: Rather moronic

        They should fire their IT person. Er, wait.

        1. Chris G

          Re: Rather moronic

          There is definitely a case for someone to answer, simply for it being possible for an ex-employee having any access at all.

          All access should have been removed at the same time as her employment ended.

          1. WanderingHaggis

            Re: Rather moronic

            Actually at least 30 minutes before she is told her services are no longer required and if she is showing signs of heading for the door or serious issues arising even earlier. Don't leave a window of opportunity.

            1. wyatt

              Re: Rather moronic

              Yep, I had to remind my people that even though the person leaving has given a date, you take away their elevated privileges now.

              Leaving someone's account available after they've left the building appears to be more common than it should be!

              1. Anonymous Coward
                Anonymous Coward

                Re: Rather moronic

                Serious question: if someone has given notice (or been given notice) and you take away their access privileges prior to that time, what work are you expecting them to do between the time they give/get notice and their last day of work? Traditionally in the US the notice period has been 2 weeks, but I know that in the UK and EU it can be literally months, especially if the employer initiates the separation. Is the expectation that the employee doesn't actually show up for work during the notice period and the pay for that length of time is severance? Because certainly I have never had a job I could do, at all, without access to my employer's computer systems.

                I appreciate the "protect the company" motive here, I'm just not sure this is always the best way to accomplish that. It might depend on who initiates the separation, any history of grievances, litigation, performance problems, etc. It might also depend on the nature of the role, the employee's reason for leaving, and a dozen other factors I haven't immediately considered. It's true that the safest thing is always to revoke access, but that's true in general too and if applied to that degree it would also be equivalent to turning off all the computers. Certainly in my time in IT we always had a list of accounts to suspend prior to layoffs or someone being fired for performance reasons, but it was typical to do the suspensions shortly before the employees were informed (i.e., same day). Employees in IT or other privileged roles who left voluntarily were usually left with their normal access until their last day, at which time it was permanently revoked. Only if there were special circumstances (e.g., going to work for a direct competitor, history of disagreements with the company) would access be revoked immediately.

                1. dave 76

                  Re: Rather moronic

                  "but I know that in the UK and EU it can be literally months, especially if the employer initiates the separation. Is the expectation that the employee doesn't actually show up for work during the notice period and the pay for that length of time is severance? Because certainly I have never had a job I could do, at all, without access to my employer's computer systems."

                  Yes it happens in the UK, people can be put on gardening leave and basically get paid to sit at home until the end of their notice period. They are not able to start another job in the meantime because they are still employed.

                  Often the company and employee will negotiate to reduce this amount of time for a cash payout.

            2. A.P. Veening Silver badge

              Re: Rather moronic

              Actually at least 30 minutes before she is told her services are no longer required and if she is showing signs of heading for the door or serious issues arising even earlier. Don't leave a window of opportunity.

              Depends a bit on the person. I've been told more than once that my services weren't required anymore (in one case twice at the same company) and I had full access up until the minute I logged off prior to leaving for the last time. At the company where I was employed twice (over ten years with a four month break after about three years) I could get QSECOFR access on the development machine in about four minutes, on the production machine it would have taken me a bit longer. I never abused that and I was informed of my contract being terminated well over a month in advance. There is something called professional pride.

              On a side note: to convert an AS/400*) into a smoking pile of trash you only need QSYSOPR access and knowledge of a couple of basic commands.

              *) or iSeries or however IBM likes to call it this month

              1. Luiz Abdala
                FAIL

                Re: Rather moronic

                rm -rf as root? Or some flavor of it? format /u ?

                Anyway, I agree that prior to firing people, their access should be revoked, and backups made.

                1. A.P. Veening Silver badge

                  Re: Rather moronic

                  rm -rf as root? Or some flavor of it? format /u ?

                  I am not going to tell how, but it will mess up the system pointers. Those with enough knowledge of the AS/400 will know enough (and probably already know about it).

      2. Gordon 10

        Re: Rather moronic

        $10,000 is what they *say* they spent. Not actually what they did spend. For example they used a PFY to do the restore/recovery but for the purposes of the lawsuit they value him using a BOFH's "consultancy" rate card.

        1. gnasher729 Silver badge

          Re: Rather moronic

          Would you blame them? The woman deserves all she gets. If they were clever and restoring everything cost less than $10,000, she is lucky. If there were no backups, I'd keep her paying to the end of her life.

          1. Martin M

            Re: Rather moronic

            What she did was moronic. But not sure I get the logic on why her fine should depend on how moronic the company is. Restoring in less than 10k is not “clever”.

            Far from restoring from actual backups, with a proper setup this should have been a case of simply restoring a NAS snapshot - 5 mins tops actual technical work, call it a half day with surrounding paperwork. This is not advanced technology - I have had it on my home server, for free, for the better part of a decade.

            But it sounds like even backups didn’t exist, so they’re spaffing money on disk recovery specialists instead.

    2. gotes

      Re: Rather moronic

      It seems to me Ms. Barile has taught their "IT department" some important lessons.

      So much customer data on a presumably general purpose "shared drive"

      No backups/few backups

      Ex employee's access not revoked

      1. Pascal Monett Silver badge

        Absolutely.

        Shame on her for deleting files.

        Shame on the IT department for not having regular backups.

        Shame on management for not insisting on proper security rules and surfing on the "hey, it's working, everything's fine" attitude.

        Yes, she is to be blamed for what she did. But the company bears a greater burden of blame for making it possible in the first place.

        1. bombastic bob Silver badge
          Facepalm

          But the company bears a greater burden of blame for making it possible in the first place

          The main point of the article, no doubt.

          They apparently had a directory labeled 'DO NOT DELETE' containing ransomware-related things. At least THAT one should've been mirrored in several places, including a USB drive, on a regular basis. Why? RANSOMWARE of course!!!

          icon, because, facepalm

          1. gotes

            Yes, that tickled me. At the very least they could have done a bodge job and added a "Deny delete" privilege on the directory if they really didn't want it to be [casually] deleted.

            1. the spectacularly refined chap

              But that kind of defeats the purpose. When I saw that it sounded to me very much like an internal honeypot as an additional line of defence. I read up on Cynet anti-ransomware and indeed that it the the case. Those files are constantly monitored and anything messing with them gets killed on sight. Wouldn't be terribly effective if access controls prevented access.

              1. Doctor Syntax Silver badge

                "Those files are constantly monitored and anything messing with them gets killed on sight."

                So why didn't that happen?

                1. sofaspud

                  Presumably she hit delete on the folder rather than mucking about with the files contained inside. Which doesn't actually 'delete' anything, it just removes the folder entry and marks the space as free (assuming this is a Windows share, anyway).

                  I imagine the ransomware-watcher is looking for activity at the file level that looks like encryption rather than folder deletion, since the former is what they want to catch and stop (ransomware wouldn't work if they just nuked you from the start, gotta have that tantalizing chance of recovery to bait the hook!)

        2. Anonymous Coward
          Anonymous Coward

          There's no need to compare or portion out blame. Each party bears 100% of the blame for its own actions and choices, 0% of the blame for any other's. They are independent.

          We can also consider the person who has insured her goods against theft but left her doors unlocked and wide open. The thief is no less guilty of a crime because he certainly knew the goods he took were not his, but the owner will not be compensated by the insurer because she enabled the theft. The two actions were independent, and each suffers alone.

          In a fair world, the ex-employee would go to prison and the credit union's managers and directors would all be fired and the clawback clauses in their contracts activated. In the world we live in, the US Supreme Court has decided that authorised access to computer systems for unauthorised purposes is not a crime, and the shareholders never ever hold their employees accountable.

  2. FozzyBear
    Flame

    Court documents indicate that the credit union had "some" of the data backed up

    Some, SOME. !

    Geez, Understand losing the deltas for that business day. But come on.

    I'm not condoning what she did . But two failures on part of the company,

    Firstly, disable access when you fire someone.

    Secondly, pay for a proper backup / DR strategy

    1. Anonymous Coward
      Anonymous Coward

      Re: Court documents indicate that the credit union had "some" of the data backed up

      It it is a Windows file server, then your first fall back is shadow copies. It is so simple to turn on, I can't understand why more companies don't use it. If you are willing to lose any work since the last shadow, then reverting a complete drive takes almost no time at all. If you don't want to lose anything then you need to mount the shadow and selectively restore with a robocopy job or similar. Even this shouldn't take a huge amount of time though.

      If this doesn't work, then go to your backups. How on Earth you go to the effort of just backing up some of your stuff is beyond me. With something like VEEAM, it is a doddle to just backup everything.

      Any company that took data security properly would also have extra layers (backup replica, off site copy, lagged copy to another datacenter, backup to tape, all of the above).

      Not having a full backup should be criminal negligence. How it can possibly cost $10,000+ to simply recover deleted files a on a single server drive is beyond me.

      1. J__M__M

        Re: Court documents indicate that the credit union had "some" of the data backed up

        I can't understand why you think they weren't using it. Our deleter had admin.

        1. Anonymous Coward
          Anonymous Coward

          Re: Court documents indicate that the credit union had "some" of the data backed up

          Where does it say in the story that she was an admin? Just says she was a part time employee. The only reference I can see to an admin is at the end of the story when it refers to someone else.

          Even if an admin though, assuming she could nuke the shadow copies, she shouldn't have been able to wipe the backups if they had any kind of sane backup solution.

      2. bpfh

        Re: Court documents indicate that the credit union had "some" of the data backed up

        4 hours downtime affecting 200 people, average salary 2000 dollars a month, that’s not far off about 10k in wasted payroll?

      3. bombastic bob Silver badge
        Devil

        Re: Court documents indicate that the credit union had "some" of the data backed up

        recovery costs ARE high when it takes expensive consultants to do it. They're probably running file un-delete cleanup utilities and doing it all by hand.

        But what you and I probably would have had at the very LEAST is a simple 'xcopy' of the shared drive (as it was last night at midnight) on the IT manager's personal computer, since its only a few gigabytes and any decent IT manager can offload some of the porn to make room for it...

      4. Brewster's Angle Grinder Silver badge

        Re: Court documents indicate that the credit union had "some" of the data backed up

        "How it can possibly cost $10,000+ to simply recover deleted files..."

        I was expecting forensic recovery piecing together sectors on the disk. But $10,000 is ~$0.50 a file (each 1 Mb) so that could be the employee time to check the output of a stock undelete opens and is intact, figure out which file it is and put it in the correct place in the heirarchy

    2. veti Silver badge

      Re: Court documents indicate that the credit union had "some" of the data backed up

      Yep. This is supposed to be a credit union. Data, and data security, is everything to them.

      If ever there were a business that really should have a rock solid backup strategy in place, this is one.

    3. bombastic bob Silver badge
      Devil

      Re: Court documents indicate that the credit union had "some" of the data backed up

      Secondly, pay for a proper backup / DR strategy

      Or something simple, like DOS-style batch files and zips of the entire share drive with the YYYYMMDD date in the file name. Probably less than 10Gb for the ZIP. A 2TB USB drive plugged into the IT guy's computer, backing up every day at 8PM with a scheduled process, and when the drive gets full, get a new drive. 10GB a day could run for 200 days before filling. Or maybe you just clean out old files once a month or something...

      (just turn off the monitor at night and leave the CPU running to do the scheduled stuff)

      But yeah you need an IT manager that knows what he's doing.

      1. Anonymous Coward
        Anonymous Coward

        Re: Court documents indicate that the credit union had "some" of the data backed up

        Why bodge something together like that? I am sure they could afford to buy a proper backup solution. Assuming they are using VMs, then something like VEEAM targeting 1 or more NAS, with cloud connect to a secondary offsite location and/or copy to tape would backup everything and be resilient to user malice or genuine disaster.

        If not virtualised (why not?) then VEEAM agent or something like Arcserve if feeling masochistic.

  3. DS999 Silver badge

    Lots of stupid to go around

    1) Fire an employee but don't immediately revoke their access.

    2) Get fired and think logging in with your credentials and deleting a bunch of important stuff won't be traced to you.

    3) Brag about deletion so if her former employer was stupid enough to have not been able to detect it was her who did it, they had a confession.

    4) Choosing the wrong person to brag to, thinking they won't report the major felony you admitted to.

    I bank at a credit union. If this was mine, I'd be looking for somewhere else to take my business!

    1. A.P. Veening Silver badge

      Re: Lots of stupid to go around

      1) Fire an employee but don't immediately revoke their access.

      2) Get fired and think logging in with your credentials and deleting a bunch of important stuff won't be traced to you.

      Between those two first items I am missing "Have a good backup strategy".

      As for taking the business elsewhere, AMEN!

      1. TimMaher Silver badge
        Facepalm

        Re: Amen

        Pity we don't know which credit union got hit.

        1. DS999 Silver badge

          Re: Amen

          She's from Brooklyn and it was heard by a New York court. So given that I don't live in New York and my credit union is not based there, I know for a fact it is not mine.

  4. johnB

    Swiss cheese

    Seems to me to be a typical swiss cheese error:

    * IT didn't delete her access at the right time - unclear why

    * Aggrieved (ex-)employee

    * Ex-employee decides to abuse IT's error / oversight

    What's interesting to me is why she was able to delete stuff. As has been said, the data IS effectively the credit union & should be more or less impossible to delete, if for no other reason than compliance with corporate reporting, taxes, etc. Surely deleting critical data shouldn't be an option for someone apparently fairly low down on the food chain?

    Alternatively, $10K is peanuts to most companies, esp in the financial sector, so maybe it was deemed an acceptable risk of doing business?

    1. bombastic bob Silver badge
      Unhappy

      Re: Swiss cheese

      banking regulators might decide to levy FINES that exceed that comparatively insignificant cost of $10k

    2. Anonymous Coward
      Anonymous Coward

      Re: Swiss cheese

      "Alternatively, $10K is peanuts to most companies, esp in the financial sector..."

      While it is true that a credit union operates in the financial sector, most of them are very small and $10k might well be a significant fraction of their profits. A credit union is a customer-owned cooperative, and in most cases those are not expected to make a profit at all. That loss comes out of their (often hundreds or low thousands) of customers' dividends or interest payments.

      There are quite a few banks in the US with a billion dollars (10^9) in assets. But with interest rates set at zero forever and endless regulatory costs, those assets generate almost no income. One should not assume that everyone operating in the financial sector is raking it in hand over fist. At least half of the US's 6000+ financial institutions are barely hanging on; they don't have trading departments making big money buying up meme stocks that pay nothing or "advising" giant corporations on their next acquisition. This kind of loss is certainly not going to sink a small credit union but it's also the last thing they need. That of course is exactly why they should have taken the absolutely free step of revoking the access of employees they terminate.

  5. bpfh

    Several levels here…

    Why was she pissed enough to wipe a drive?

    Why did she still have access - someone in helpdesk has a backlog I guess and as nothing bad has ever happened when an employee leaves it’s not a priority… I guess it will be from today!

    How long did the help desk take to restore from a shadow copy or backup? I would have guessed that having important customer data on a shared drive entails shadow copies and full backups….. no?

    Not to diminue what happened as if they had a solid BAR plan this would have sent half the company into a tizzy for an hour or two, but times that by the number of affected employees , that’s a serious chunk of change that they could claw back in a lawsuit….

    And always reach out directly to your friendly neighbourhood sysadmin to confirm account locking when you plan a termination…

    1. Anonymous Coward
      Anonymous Coward

      Re: Several levels here…

      Helpdesk should not be in charge of removing access. It should be automatically triggered from the HR system when the termination date is set on an employee, either instant or on that date, depending on what the company want.

    2. nerdbert
      FAIL

      Re: Several levels here…

      <blockquote>How long did the help desk take to restore from a shadow copy or backup? </blockquote>

      I'm guessing it actually could take quite a while to restore the state. Assume nightly backups as most of the commenters here are suggesting. That means you could lose a day's work.

      How many loan applications could get approved/rejected in a day?

      How many accounts could be opened/closed?

      How many supporting documents could be added to accounts/applications?

      And how much is the financial business done in a day related to their shared drive?

      A $10K bill for restoring that day's data is probably a drop in the bucket to what they lost by this deplorable lack of control and proper backups. In this sort of business I'd expect hourly incremental backups at the very least, even for relatively less valuable data like supporting data that could be re-requested.

      And this is even without contemplating the issues of possible data modification before deletion (corrupting the files you would try to restore), trying to audit any possible access by this employee, etc. This is an IT clusterfsck in a business where some real money is at stake. Shame on the CU for their IT practices.

      1. Doctor Syntax Silver badge

        Re: Several levels here…

        How much of that work should be on a shared drive rather than on a proper RDBMS with a proper transaction logging system running? Restore to the last checkpoint and then roll the transaction logs forward.

  6. Ochib

    $10k Cost

    They'll have spent $2k just in staff time in meetings to work out what happened, why it happened, how it happened and how to fix it.

    Then there's the server rebuild, detailed analysis of which data can and can't be recovered, the recovery itself, validating the recovery, reacquiring the data that couldn't be recovered, communications and customer management activities around that and, well, $10k feels quite cheap.

  7. bofh1961

    I did the same by accident...

    On a Saturday morning I accidentally deleted the entire data storage pool on both the main fileserver and its remote mirror. We did have full backups but it took until 0700 on the Monday morning before it was all restored. believe me, sixty hours without sleep taught me to be more careful.

    1. Plest Silver badge

      Re: I did the same by accident...

      Not knowing all the facts the only thing I can think of that was that when messing with storage, always consider suspending the DR/remote cloning processes until the work on the primary is complete so mistakes aren't replicated. Disabling such services and processes has saved me twice when I made a slight mistake that would have taken me hours to fix, it was able to fix my issue in about 20 mins, take a short walk to allow my heartrate to return to normal and actually sleep sound that night!

    2. Security nerd #21

      Re: I did the same by accident...

      Name me one Linux admin who hasn't done "rm -rf * " in the wrong directory.

      It is a right of passage, and you only do it once - followed by the swearing and then fixing it whilst hiding your tracks ...

      1. Anonymous IV
        Unhappy

        Re: I did the same by accident...

        > Name me one Linux admin who hasn't done "rm -rf * " in the wrong directory. It is a right of passage...

        Homonym failure! "Right" should have been "rite", right? Write on...

        1. disgruntled yank

          Re: I did the same by accident...

          Well, or you have the right to do it once.

      2. druck Silver badge
        Unhappy

        Re: I did the same by accident...

        Or clone disks with dd and get the source and destination the wrong way around.

        1. bombastic bob Silver badge
          Devil

          Re: I did the same by accident...

          "dd" stands for "WAIT... take a breath, re-read what you just typed, and pray to whatever dieties you believe in before pressing the 'enter' key"

          1. A.P. Veening Silver badge

            Re: I did the same by accident...

            Praying to those deities is optional, praying to Murphy on the other hand is obligatory, with re-reading immediately prior and after.

            1. Doctor Syntax Silver badge

              Re: I did the same by accident...

              Murphy says don't bother with before. Just read iafter you press return.

              And BTW it doesn't have to me an rm. It's surprising what you can do with a simple mv.

        2. captain veg Silver badge

          Re: I did the same by accident...

          I once burned a Linux ISO to my mobile phone like that. It wasn't even bootable.

          -A.

        3. Vometia has insomnia. Again. Silver badge

          Re: I did the same by accident...

          ...or repartition a disk because "I have a backup tape" and, later, reuse it as a scratch tape because "the files are still on disk". Argh. Fortunately nothing particularly important was lost, but one of those instances of learning the hard way.

        4. druck Silver badge

          Re: I did the same by accident...

          Checking the dd command line doesn't help if the issue was this was the one system out of a dozen where /dev/sda wasn't the main drive, and /dev/sdb wasn't the new drive.

      3. Martin
        Happy

        Re: I did the same by accident...

        I already mentioned this, but I never never NEVER type "rm -rf * " - not never, not nohow.

        I always go up one directory and type "rm -rf <directoryname>"

        After all, if you want to delete everything in a directory, why do you still need the directory? And it minimizes the chance of you deleting everyihing in the wrong place.

        1. Ken Moorhouse Silver badge
          Coat

          Re: I already mentioned this, but I never never NEVER type "rm -rf * " - not never, not nohow.

          You must have.

          Unless you copied and pasted what you posted (like what I have done here).

          But then, it is possible that the keyboard may never have witnessed those keystrokes and yet still be able to submit that dreaded command.

          1. Martin
            Happy

            Re: I already mentioned this, but I never never NEVER type "rm -rf * " - not never, not nohow.

            The point is conceded. A pedant after my own heart.

            Correction - I never never NEVER type "rm -rf * " at a Linux or UNIX command line prompt.

        2. the spectacularly refined chap

          Re: I did the same by accident...

          After all, if you want to delete everything in a directory, why do you still need the directory? And it minimizes the chance of you deleting everyihing in the wrong place.

          So that whatever is putting the files there can continue to do so. Log files, temp files, spool files. Favourite example would be a workgroup printer spool directory after the printer has been offline for a few hours. You can guarantee someone will have printed the same document twenty times because it didn't come out the first time.

          Better idea, have your PS1 prompt show the current directory, current machine, and whether you are a normal or privileged user

          1. Martin

            Re: I did the same by accident...

            Good point - I hadn't thought of that...!

            Though in that case, you should probably not use "rm -r" anyway...

      4. CAPS LOCK

        Hahaha, the blank space of doom...

        ...

        # rm -rf *.bak

        vs

        # rm -rf * .bak

        Smart admins create a less powerful 'del' command. Well, smarter than me anyway...

    3. bofh1961

      Re: I did the same by accident...

      It was a very hokey-cokey third-party remote disk mirroring system for NetWare I was trying to fix. The mirroring process kept hanging the primary server due to remote disk writes failing. I wasn't expecting breaking the logical mirror link in NetWare to have quite that effect though.

    4. bombastic bob Silver badge
      Devil

      Re: I did the same by accident...

      sixty hours without sleep taught me to be more careful.

      and to always have a backup!

  8. Anonymous Coward
    Anonymous Coward

    the data IS effectively the credit union

    All companies nowadays are really IT companies under the hood. Take away their IT and they are fucked.

    We learned this with the ongoing HSBC outages ages ago. Banks are IT companies that happen to do banking. And so on.

    Now with that in mind, why in the name of all that is Holy would you skimp on IT ????

    1. johnck

      Re: the data IS effectively the credit union

      "why in the name of all that is Holy would you skimp on IT ?" thats an easy one, its because IT is only seen as a cost, and cost should always be minimised so profit can be maximised.

      The fact that that the cost of IT can only be seen because of IT and the decision to limit/reduce the spend on IT is taken using data provided by IT viewed on equipment provided by IT and comunicated via IT systems is lost on people. IT is only noticed when it doesnt work or somethign goes wrong.

      In business IT is a lot like cleaners. No one notices the cleaners, or cares if they have to work 10 hour days to clean the place but only get paid for 8, but if they miss a bit, or heven forbid dont clean for a few days, people will notice and demad somethign must be done immediatly. Oh and its always someone elses fault, cleaners and IT.

      The Minimise Cost and Maximise Profit, mantra is so embedded in business that it affects everything, even non profit business

      1. Plest Silver badge
        Facepalm

        Re: the data IS effectively the credit union

        Once had an upper level PHB call the entire IT dept "The worst kind of financial black hole ever created within any company. All cost and no return!". Thanks to more savvy tech enabled PHBs, he didn't last longer than 18 months!

        Unfortunately to most biz people IT is simply seen as taking all the cash and giving nothing back, wasting money on tech doo-dads and gizmos. OK, tell you what how about I switch off all the servers and routers at 9am today OR we give all the IT people the next 48 hours off, let's see how long the company can stay afloat shall we?

        1. spireite Silver badge
          Devil

          Re: the data IS effectively the credit union

          You don't even need to be away for 48hrs.

          Reminds me of the time other elements of management at a former employer complained that the IT department was taking 30m longer for lunch than contracted.

          They completely ignored the fact that we were contracted to 9am-5pm, but routinely arrived at 830am and stayed till 6pm or beyond.

          It was raised by our manager to us that the others complained (he often was with us) and said we should cut back our lunch hour (and a half) to the contracted hour.

          In the spirit of being told that our contract said 1hr lunch, we also noted the contract said 9 thru 5 and predictably ensured we arrived @ 9, and left at dead 5.

          Time taken for the same moaning managers to wholly complain that there was no IT after 5pm ?? two weeks... if that.... first complaint - next day....

          That is the problem, IT is never appreciated

          1. Eclectic Man Silver badge

            Re: the data IS effectively the credit union

            True story.

            Office phone rings at 14:00 on a Friday afternoon:

            Worker: "Hello"

            Caller: "Hello, could I speak to the team leader please?"

            Worker: "No, mate, they're all out."

            Caller "When will they be back?"

            Worker: "Not til late, they're down the pub."

            Caller: "Do you know who I am?"

            Worker: "No."

            Caller: "I am Michael Edwards, CEO."

            Worker: "Oh. Do you know who I am?"

            Sir Michael Edwards: "No."

            Worker: "Good!" (Puts the phone down.)

            *At the time, Tory 'blue-eyed boy' and general brutal company turner-arounder.

            1. Doctor Syntax Silver badge

              Re: the data IS effectively the credit union

              Caller: "Do you know who I am?"

              Worker: "Sorry, I can't help you there. Ask whoever's near you, they might know."

              1. Eclectic Man Silver badge
                Joke

                Re: the data IS effectively the credit union

                Allegedly a true story.

                HM Queen Elizabeth the Queen Mother visits and old peoples home in England somewhere.

                HM QE t QM to an old lady resident: "Do you know who I am?"

                Resident: "No, dear, but if you ask Matron, she'll tell you."

      2. B83

        Re: the data IS effectively the credit union

        Tech Debt is the fancy phrase. Systems/applications usually have a 10 year life span in that disguise, then they have to be updated e.g. version 2, or replaced.

        Management only look at cost and thats where the killer is. Cut back this, pay off them, p1ss the workforce off and 'hey, your in the sh1t'.

        This situation should never happen but does so frequently.

        1. Davegoody

          Re: the data IS effectively the credit union

          Agree, when management stop treating IT as a cost centre and start realising it's a profit enabler, we can all sleep a bit better.....VERY sloppy management at play here that allowed this to happen. She needs to pay the consequences of her (stupid) actions, but this was 100% avoidable

      3. Doctor Syntax Silver badge

        Re: the data IS effectively the credit union

        "IT is only seen as a cost,"

        The best way to deal with that attitude would be "OK, let's shut it down for a few hours and see how much money that saves."

  9. Anonymous Coward
    Anonymous Coward

    Here and there

    Recently our FI has been sloppy about giving term notices to IT (no we don't let HR have auto fire buttons that manage access for the dozens of systems that need to be touched, because it isn't that simple) I will be sharing this post with my management team as example of WHY it is imperative we know when to disable access immediately if not the night before.

    Thanks for the story Reg.

  10. davebarnes

    What is the name of the credit union?

  11. Eclectic Man Silver badge
    Happy

    The Register - Organ of Record

    Many years ago, when I was employed, I had to do a small presentation on a security breach in the public domain. I chose one, and checked out the press reports (UK ones - The Guardian, the Times, etc.), the company web site, and "The Register".

    The regular media were fairly consistent in their descriptions of what had gone wrong (A large-ish UK based company had changed suppliers for its outsourced data processing functions, the old supplier had, unaccountably, not only kept the data in a database, but had left it accessible online, and it had been snaffled by some naughty people.)

    All well and good and soberly reported.

    However, the Register's forum on the scandal was full of stories about access being granted to people who had left employment weeks, sometimes months previously, able to grant others perks, like vouchers for free drinks, money-off meal vouchers etc.

    So I recommended that everyone read the, shall we call them, 'less reverential' media, for a more detailed and, perhaps realistic account of what actually happens. I don't believe that every IT professional actually reads the Register, but maybe they should.

    (OK, Register overlords, please send my gold medal to the usual address. ;o) )

    1. Anonymous Coward
      Anonymous Coward

      Re: The Register - Organ of Record

      The forums are where you'll find the good stuff. Reminds me of an old Electrical Engineering professor of mine. He had a consulting contract many years ago to do some control systems tuning at a very large industrial facility. I remember his comnent about how you learn a lot about how things operate in your initial meetings. You learn even more by observing the process. You learn a ton when you're taking an elevator ride with one or more of the grunts who actually uses the stuff.

      1. Eclectic Man Silver badge
        Unhappy

        Re: The Register - Organ of Record

        You couldn't persuade your Electrical Engineering Prof to give courses to the MBA and PPE students, could you? I mean actually talking to and especially LISTENING to THE PEOPLE WHO DO THE ACTUAL WORK doesn't seem to be high on their list of priorities (at leat not in my personal experience, and yes, I've read quite a few MBA course descriptions, and the Oxford PPE course which is a route into parliament via McKinsey & Co for all too many people).

        Oh what's the point? Don't mind me, I'll just talk amongst myself for a while ....

        1. Doctor Syntax Silver badge

          Re: The Register - Organ of Record

          The art of consultancy is to listen to the people and then use that as the basis of the report with a high price tag. It's the price tag that's significant. The bean counters know the price of everything and the value of nothing. If the information is coming with the price tag of a low level worker it's worth what they paid for it. If it comes from someone costing a lot of money it must be worth a lot. But it's the same information.

  12. John H Woods Silver badge

    Is this not the perfect use case ...

    ... for a ZFS* snapshot?

    * other copy-on-write Filesystems with snapshot functionality are available.

  13. ZekeStone

    SOME???

    "Court documents indicate that the credit union had "some" of the data backed up and that it has spent more than $10,000 undoing the damage."

    Only SOME of the data was backed up? ALL of it should have been backed up. There's simply no excuse for this.

    1. A.P. Veening Silver badge

      Re: SOME???

      Only SOME of the data was backed up? ALL of it should have been backed up. There's simply no excuse for this.

      I'd say there is an excuse for data less than two seconds in the system not to be backed up as even back up takes some time.

      1. Bartholomew
        Pint

        Re: SOME???

        > I'd say there is an excuse for data less than two seconds in the system not to be backed up as even back up takes some time.

        Maybe not every 2 seconds, but you can have instant online read-only access to historic files with the right storage solution (lookup "Netapp snapshot" or "ZFS snapshot" or even "BTRFS snapshot" if you are a bit of a masochistic).

        One place I worked setup a system like the below (because the data on that volume had enough value):

        One snapshot every 15 mins (3 snapshots)

        One snapshot every hour (23 snapshots)

        One snapshot every day (30 snapshots)

        One snapshot weekly (7 snapshots)

        One snapshot monthly (6 snapshots)

        And the data was backed up to tape every night and sent to off site every morning, with tapes having a rotation that meant that there was at least 12 full sets of monthly tapes from seven years ago. 24 full sets of monthly tapes for six years ago ....

        For the above to work well it required knowledge all around. Someone continuously creating and deleting large files could cause storage problems for the following six months. Because once a file is deleted, it is not actually deleted until the last snapshot containing it's data is finally gone.

        The system described in the article sounds like was designed by some who once heard the word IT while passing someone in a corridor.

        1. Eclectic Man Silver badge

          Re: SOME???

          "I'd say there is an excuse for data less than two seconds in the system not to be backed up as even back up takes some time."

          Really? Any large, multi-user, database worth running is worth protecting with a journal which is written to with a complete list of changes as they are made so that they can be rolled back or forwards as necessary. This is not a new invention, the ICL Data Dictionary System I worked on in the 1980's had it.

          1. A.P. Veening Silver badge

            Re: SOME???

            Really? Any large, multi-user, database worth running is worth protecting with a journal which is written to with a complete list of changes as they are made so that they can be rolled back or forwards as necessary. This is not a new invention, the ICL Data Dictionary System I worked on in the 1980's had it.

            True, but journaling doesn't protect against dropping tables or even complete databases.

    2. captain veg Silver badge

      Re: SOME???

      Well, that depends on its value and the cost of retrieving it from elsewhere.

      I'm much more surprised that a part-time office grunt was even able to delete any file that she didn't own and weren't private to her.

      But P:\, eh? Do I detect Microsoft Windows?

      -A.

  14. Ashto5

    Blame the Women

    You should be blaming the Managers.

    It is always bad management that is the issue.

    Silly person did a silly thing, she should never have been able to do it and that should mitigate the damage, 10 years in prison that is outrageous

    Some killers don’t get that

  15. TheDrift
    IT Angle

    The Drift

    Surprising that someone who can't spell the Drive is able to achieve this task!

  16. Doctor Syntax Silver badge

    "Her petty revenge not only created a huge security risk for the bank"

    I think the security risk was created by whoever failed to revoke her access or maybe whoever failed to set up a procedure for doing so (according to TFA IT were only asked to revoke it, not told to do so). She only realised that risk. A risk is at least as likely to be the result of a failure to do something.

  17. Anonymous Coward
    Anonymous Coward

    Piss poor security

    At the NHS back in 2007 it was stated you could only have your NHS mail on a blackberry I believe it was, I'm trying to remember, I believe the ruling was aslong as your device can be encrypted or something, pass the exchange test to say your device was secured (annoying can't remember the exact term). Anyway, I'd set it up on my iTouch so I could get emails when walking around, was never given a blackberry as "You're just the temp engineer" (that ended up staying for 4 years). Turned out back then Apple did something that was flagged up that was naughty. It would tell exchange that the device it was on was secure when it wasn't, so you could bypass the new checks and still have access to NHS mail. Eventually Apple were forced to update this but if you never updated your iTouch you could still carry on as normal.

    Anyway. Eventually my leaving day arrived, I still had my NHS mail on my iTouch. Several months later, having just left the iTouch as is as too lazy to wipe the mail account, I noticed I was still getting the NHS mail. So despite having left several months earlier, they'd left my account active. Idiots.

    A local council I worked at, their service desk manager, who was useless had left an AD account wide open, that a 3rd party company used to remote in to do database management. They all came in one morning to find one of the big databases deleted and I believe the backup. Turned out a member of the 3rd party company had gone rogue, decided to remote in with the account left wide open and have his fun.

    1. Ken Moorhouse Silver badge

      Re: So despite having left several months earlier, they'd left my account active

      A bit belated, but I'd be inclined to send a recorded delivery letter informing them of this, so there can be no comeback on you in the future.

  18. Danny 2

    Open plan office so chat travels when you are trying to concentrate. A young developer was telling someone else about the time he'd rm-rf'ed a root. I thought stupid sod. Five minutes later he exclaimed, oh no, I've done it again just because I was talking about it. I thought stupid sod. Five minutes later I did it too because although I was concentrating on my work, the dark side of my brain was chewing over the ear worm. Stupid sod.

    I then got a sysadmin job abroad, not what I aspired to but I figured well within my skill-set. Some of the local staff were obviously hostile to my appointment and weird things were happening to the systems, especially the comms. I decided to spend a night locking it all down only to find none of the passwords worked on any of the kit. The next day I asked my boss about the sysadmin before me. He'd been fired, left under a cloud.

    "Ah. He is still your sysadmin then. I'm going to have to close everything down and reset to factory defaults to take control, and ignoring the sabotage that is a lot of work and I'm bound to make a few mistakes that will cause some disruption the next morning." Tomorrow morning? "God no. I'll start Friday evening, finish Sunday, be in early Monday after a few hours sleep to firefight. Should be sorted by end of Monday and I'll take Tuesday off."

    That won't look good on your first week! "I agree, you'll get a lot of criticism for giving this 'employee' unmonitored and unlimited control, and not checking on his his changes. I think it would reflect better on you if you monitor me on Saturday and Sunday and help out so this can't happen again."

  19. Jupiter234

    1. A part time employee has full read/write access to all critical files from business point of view.

    2. The company uses a shared drive containing all of their customer PII information / business information with full access to everyone without backups (I believe partial backup means they had some files copied to workstations ...) I know it is US but GDPR guys!

    4. No audit logging who does what

    5. The company operates based on files instead of a proper applications with processes, audit logging and permission system

    If this where a SOHO or a startup with 5 employees I could accept that but a financial institution?!!!! What?

    How this credit union even has a license?

    Conclusions:

    Every customer data is accessible to everyone. Yes the sensitive ones too.

    Every data can be modified without a trace.

    Every data can be leaked without a trace.

    Every process is manual without automation except Excel macros :)

    No to talk about hackers or APTs. I can image after this how this whole infra is soo exposed to anybody who wants to get in.

    And no, I'm not talking about revoking permissions because an insider could have done this any time... or is be able to...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like