back to article Engineer admits he wiped 456 Cisco WebEx VMs from AWS after leaving the biz, derailed 16,000 Teams accounts

A former Cisco employee pleaded guilty in a San Jose federal court on Wednesday to unlawfully accessing Switchzilla's Amazon Web Services infrastructure and damaging the networking giant's cloud computing resources. Sudhish Kasaba Ramesh, who worked at Cisco as a software engineer from July 2016 to April 2018, admitted in a …

  1. Imhotep

    This Is A Job For An American

    Is he in the country on an H1b visa to address a shortage of miscreants? I think we can supply our own, and even worse, thank you very much.

    1. Yet Another Anonymous coward Silver badge

      Re: This Is A Job For An American

      But this was a cyber-crime. It is increasingly difficult to hire white collar criminals in the USA, the market is so tight and so many of the best have secure government jobs.

      If it was possible to bring down the CISCO WebEx by shooting at it, then an American would be perfectly qualified

      1. bombastic bob Silver badge
        Trollface

        Re: This Is A Job For An American

        If it was possible to bring down the CISCO WebEx by shooting at it, then an American would be perfectly qualified

        That was almost funny! Yee Haaw! (heh)

  2. Sampler

    What an idiot

    As someone on a working visa (in a different part of the world) you don't do anything that may jeopardise either that visa or potential of the next (including residency).

    I've been made redundant by a company I know wouldn't have rescinded my access to most platforms (and that's just the accounts they were aware of) but, as tempting as it is to give a last fuck you, you move on, because the only one you're fucking is yourself.

    1. big_D Silver badge

      Re: What an idiot

      And we are professionals...

      I left one company and after a year, I noticed I still had access to the CEO's OneNote share! I wrote him an email to warn him. We had been to court over wrongful dismissal and I received a settlement, but I didn't want to endanger my situation with my new employer by doing something stupid.

      1. ericsmith881

        Re: What an idiot

        This is the proper grown-up response. I had an acrimonious parting with two prior companies. In both cases, they failed to change all the passwords or remove access to all platforms. I found out later because I continued to receive alerts (my personal email was on the list for emergencies). When I figured out I could still get into things, I notified them. I even went so far as to work with the outsourced IT firm that had "replaced" me and my team because they were too incompetent to do it themselves.

        Yeah, i could have given them the middle finger or pulled what this guy did. I knew enough about how to circumvent detection that I'd have gotten away with it. But ...I'm a professional. There were hundreds or thousands of people at the company I could've caused damage to by acting unprofessionally, and I had no beef with them. Throwing a tantrum is what children do, not adults in a respected profession.

      2. Anonymous Coward
        Anonymous Coward

        Re: What an idiot

        Hmm - maybe I would have accidentially gotten my very cheap odd-job laptop infested with some credential-stealing malware .... or left it on the front seat of a parked car, as per tradition!

    2. Bruce Ordway

      Re: What an idiot

      >>as tempting as it is to give a last fuck you,

      I remember an occasion where files went missing from a site, this was about the same time that an employee was let go. We had been co-workers for years, friendly be not what I'd call a close friend. When we ran into each other a few months later and I asked him if he had any ideas about the missing files. To my surprise, he confessed to having deleted the files, expressing his regret that he could have acted this way in a moment of anger and weakness.

      This was a revelation to me.

      Since that time I try not to underestimate what a person may do when under some emotional distress.

    3. Dave K

      Re: What an idiot

      >> the only one you're fucking is yourself.

      Absolutely. $2m is a drop in the ocean for Cisco. However 5 years in the slammer, $250,000 fine and potentially being booted out of the country is a serious personal cost for such stupidity. As is having a criminal record for when you next apply for a job.

      He can't even use the "did something stupid in the heat of the moment" excuse, seeing as it was 5 months after he left the company.

  3. sanmigueelbeer

    He accessed Cisco's AWS five months after quitting Cisco.

    I'm just sayin'.

    1. Anonymous Coward
      Anonymous Coward

      So what is it that you are just sayin'?

      And it is nice to see that you can cause bold face to appear and also italics. Isn't the world a grate place?

      1. ByeLaw101

        Well..

        "He accessed Cisco's AWS five months after quitting Cisco."

        So, how is it he was ABLE to access CISCO's VM's after leaving? Surely his access should have been removed when he left?

        1. sanmigueelbeer
          Happy

          how is it he was ABLE to access CISCO's VM's after leaving

          That is Cisco for you: Do what we say and not do what we do.

          To Cisco: How many of your corporate sites are running SD-WAN?

      2. Kane

        "And it is nice to see that you can cause bold face to appear and also italics."

        You may want to check this link here (it's an El Reg page, don't worry), scroll down to the section "Formatting".

        Ah, my mistake, you were attempting to be sarcarstic!

        (See, we can do hyperlinks as well!)

      3. a_yank_lurker

        Switchzilla has some incompetent admins because they have no idea what access rights he has and how to remove them. Their incompetence does not make him less guilty.

        1. Doctor Syntax Silver badge

          "Their incompetence does not make him less guilty."

          I totally agree but nevertheless it also makes them guilty of negligence. They have a duty of care to their customers and that duty includes removing the access credentials of leavers ASAP.

        2. ericsmith881

          I suspect the issue was more along the lines of him using credentials other than his own, something like a shared admin account or a service account. Such stuff is depressingly common even in big organizations that know better. It's difficult to determine who might know those credentials and a pain to change them if they affect lots of users/services. That's one reason (out of many) why they're a bad idea.

    2. bombastic bob Silver badge
      Alert

      He accessed Cisco's AWS five months after quitting Cisco.

      So why is a security-oriented networking company NOT sufficiently cleaning up their login info (like changing passwords, deleting stale logins) after an employee leaves? Unless it was accessed with a deliberately installed back door...

      Maybe I missed it but did they say WHY he left Cisco? Did he have a reason (in his own mind at any rate) to retaliate against them?

      If I were his current employer I'd smile and carefully assign him a limited number of tasks while changing all of the access codes, without his knowledge. And not fire him. Reason for termination: incarceration. Works for me.

      1. Anonymous Coward
        Anonymous Coward

        Incompetence multiples top-down, bottom-up and middle-out!

        Maybe some firms are not as security-oriented as we hope they are.

        Once an admin would shut down a VM or two accidentally, now they fire off an old script in a far-flung repo and it takes out several cloud-farms and many thousands of users.

        it is all part of the relentless rush to drive shareholder value. Many years ago, Switchzilla & Co developed new tech, and built sensible, innovative product. Now monster-corporations, they spend zillions tightening governance to effect better security. But they must also increase profit and market-cap, so reduce costs by outsourcing labour and automating everything. And, they leverage their IP by scaling services at such a rate that the reach and impact of simple failures multiply year on year at logarithmic levels.

        This is the cost of being such a BigIT: The incompetence multiples top-down, bottom-up and middle-out.

        1. EnviableOne

          Re: Incompetence multiples top-down, bottom-up and middle-out!

          Like a certain SysAdmin at AWS who killed the Virginia US-East-1 region for five hours

          https://www.theregister.com/2017/03/02/aws_s3_crash_result_of_fatfingered_command/

    3. Anonymous Coward
      Anonymous Coward

      @Sanmigueelbeer Good point.

      This is why going to the cloud is fraught with danger.

      The biggest security threat is going to be from an employee, former employee or contractor. Whether its malicious or not, this is where the security falls apart on the cloud.

      Can it be mitigated? Sure. But there's a cost.

      Now if you don't have your own On-Prem hardware... going native to the cloud may be a good way to control costs without having to make a fixed up front investment.

      This is where a 'trust no one' and going k8 w containers could have helped mitigate some of this. Also tightening down security.

  4. cantankerous swineherd

    unamerican

    surely he should have tried to make money out of this instead of just trashing everything?

    1. Version 1.0 Silver badge
      Meh

      Re: unamerican

      Maybe his new employer sees him as honest because he didn't sell the access to the Russians and now have him reworking their own system to make sure that this never happens to them.

      1. Doctor Syntax Silver badge

        Re: unamerican

        "Maybe his new employer sees him as honest because he didn't sell the access to the Russians"

        Not that we know of.

  5. mark l 2 Silver badge

    "his employer … is willing to work with him regarding the possibility of his remaining in the country and continuing to work for the company," the document [PDF] says."

    His current employer must be really struggling to find staff if they are willing to take someone on who caused $1.4m of damages to their previous employer?

    Unless of course he is now working sweeping the floor and making coffee since no one trusts him near any of their computers anymore

    1. big_D Silver badge
      Coat

      Or he is holding a dead-man's switch over their infrastructure...

      Mine's the one with the switch in the pocket.

    2. Marc 13
      Paris Hilton

      Er, isn't it 2.4 million?

      "cost Cisco roughly $1.4m in employee time for remediation and over $1m in customer refunds."

      Paris, 'cos Math is hard :-)

      1. matjaggard

        Maybe his new employer encouraged it

  6. Maelstorm Bronze badge
    FAIL

    And the point is...

    Sudhish Kasaba Ramesh, who worked at Cisco from July 2016 to April 2018, admitted in a plea agreement with prosecutors that he had deliberately connected to Cisco's AWS-hosted systems without authorization in September 2018 – five months after leaving the manufacturer.

    And the point is this: Cisco got bit because they didn't remove his login credentials after he left the company. Granted, what the guy did was illegal, and he should go to jail for it, but Cisco is also at fault for not adhering to industry standard IT practices. When I worked for the local telephone company, when someone left, their logins to the myriad of systems was purged within 24 hours. They keys and badges were taken too. And this was in the 1990's.

    1. iron Silver badge

      Re: And the point is...

      In my days as a BOFH we removed employee's access to systems while they were in the meeting telling them they were getting the boot. By the time they came out of the meeting their access was gone.

      1. Anonymous Coward
        Anonymous Coward

        Re: And the point is...

        "In my days as a BOFH we removed employee's access to systems while they were in the meeting telling them they were getting the boot."

        But did that remove ALL of their access?

        What about the test account they had for that new remote access system or the monitoring account that they were troubleshooting last week? Or maybe even the github or similar access that allowed them to modify a deployment script?

        While implementing an isolated system with high levels of security and compliance is doable, as that system matures there are often weaknesses.

        1. John Robson Silver badge

          Re: And the point is...

          Single sign on helps alot there...

          There should be a very limited set of systems that you have to revoke access from.

          1. Robert Carnegie Silver badge

            Re: And the point is...

            Yeah, each password that I use at work signs me onto a single system.

            1. It's just me
              Happy

              Re: And the point is...

              That's an unusual implementation of single sign-on.

          2. bombastic bob Silver badge
            Headmaster

            Re: And the point is...

            Single sign on helps a lot there...

            Unless it's "root"... oh, and I fixed the grammar too. See icon,

            1. John Robson Silver badge

              Re: And the point is...

              You have root passwords available for users?

              We have sudo permissions, no direct root logins.

              In general I have root ssh logins disabled anyway

          3. Anonymous Coward
            Anonymous Coward

            Re: And the point is...

            "Single sign on helps alot there..."

            While SSO is useful, I'm not sure it addresses the issues found with special access accounts used for applications/monitoring/testing by admins that can likely bypass the SSO process.

            I would suggest Cisco likely has a very robust user management workflow and that the accounts managed by SSO were correctly disabled and access removed based on comments by Cisco employees in other areas.

            Where it has likely fallen down is potentially a publicly hosted build/validation script being altered using a service account with too many permissions (i.e. write when it should only have read) or something deliberately prepared in advance (check for URL X - if its inaccessible, do Y).

      2. The Basis of everything is...
        Big Brother

        Re: And the point is...

        On what turned out to be my last day as a PFY at a certain formerly large engineering firm I processed my own redundancy through the HR system as a live test for the 15,000 who were to follow me out the door over the following months. The chief did check I didn't add an extra couple of 0's to the severance pay though.

    2. This is not a drill
      FAIL

      Re: And the point is...

      The problem is that many companies don't seem to think that Joiners, Leavers, and Movers processes extend past sett a flag in the HR systems and disabling their "network" (usually AD account).

      Even companies most important systems seem to get missed.

      I recently did an exercise at one company, there were over 170 different applications that different people needed to be logged into, 70+ the IT department had know idea about. 90% were not part of the JML process.

      With the advent of SaaS and Cloud Services it's so much easier to people in business so sign up for new services, and to the majority of people "authorisation" means getting their managers/directors sign-off.

      How many people (usually the marketing department) know their companies Facebook, Instagram, Twitter account (note singular) and how many people have left since it was last changed?

    3. bombastic bob Silver badge
      Devil

      Re: And the point is...

      not adhering to industry standard IT practices

      There is still a distinct possibility that the perpetrator inserted a back door into the AWS code, and used THAT to do the damage. At least, that's my take on it.

    4. Nunyabiznes

      Re: And the point is...

      It is quite possible that the responsible IT team didn't know he had moved on.

      HR here has a distressing propensity to neglect to mention to us that "Hey, we had to fire that dude who threatened to burn down the building and shoot his supervisor."

    5. keith_w

      Re: And the point is...

      When I was a system programmer, I got a call from my supervisor that was along the lines of "in 5 minutes, revoke <our boss's name>'s access to the mainframe and the PDP-11 and change the maintenance account password while you are at it." 10 minutes later our now ex-boss was walked out the door.

  7. jukejoint

    RE Tech Tools

    I well remember the Terry Childs fiasco in San Francisco in 2008. Principal network engineer for the City's IT department with Cisco CCIE certification was allowed to run amok and set up the City's system so that "he alone could fix it." Then when his behavior became insufferably egomaniacal & he was to be given a lateral position, went rogue and refused to allow anyone access to the City's WAN which supported 65 city agencies.

    Apparently a brilliant bundle of resentments, that one.

    Off to prison, and now released after serving his sentence.

    Oh yes and he concealed one or more prior convictions for which he also did time.

  8. CPU

    with great power...

    A pity we don't get to find out what axe he had to grind, that would be more interesting. I worked at a big City firm for 5 years, when I left (time to move on) I had to remove myself from the systems because they were not smart enough to know where all my accounts were. Luckily for them I didn't have any axes to grind ;-)

  9. Mike Friedman

    Wow. StichFix must be really hard up for IT talent if they're willing to keep on a person who plead guilty to sabotage. Remind me never to use their services for anything.

    I spent years as a freelance IT consultant and several times after I parted ways (occasionally unpleasantly) with a company I noticed that I was still getting important emails for them or had access to one of their systems. I informed them right away and deleted the saved password.

    Why? Because I'm not an unethical douchebag.

    1. AdamWill

      of course they are

      I mean, of course they're keen to keep him on. Now they know what happens if they don't. :P

  10. My other car WAS an IAV Stryker

    My leaving story from Stryker (vehicle) USA HQ

    The only emails I still get are updates to my still-active retirement/pension account.

    But I was in charge of a bunch of off-network test laptops, being a good little mini-BOFH (and patching regularly over a cellular Wi-Fi hotspot since there was little public WiFi in the place). And if I had really wanted to screw folks, I could have done everything from change the passwords to full nuke & pave, or possibly even just throw them in the garbage.

    Two reasons why not:

    1. These were my respected colleagues and peers -- the closest to "friends" I had aside from my missus -- and honestly I was leaving them in a total talent bind, so I didn't want to screw them also.

    2. Technically, the "customer" (US DoD) owned (paid for) them. While just binning them would look horrible on a future audit, eventually it would come back to me and I'd be in DEEEEEP shaving cream.

    So I dutifully handed them over, along with all the necessary test kit, asset list / tracking spreadsheet, and the like, along with everything else useful. All I took out of there was a load of office supplies included a Sharp printing calculator (I've told y'all about that before).

    * Note: I have to remind you it's Stryker the armored vehicle because my state (Michigan) is also home to Stryker medical products. A hospital bed may have wheels but isn't as fun to ride on.

  11. spireite Silver badge

    Time in clink > police

    Officers shoot black guy at close range in back.... sack em, and they'll prob get 5 years...

    This guy - probably 3 times that........

    1. Aquatyger

      Re: Time in clink > police

      The guy was reaching into a vehicle after being tasered. I'd shoot him too.

  12. earl grey
    Flame

    i'm having a hard time with this

    worked for my last company for over 40 years and when they let me go i had access to my computer and the relevant systems up until i left at the end of the day.

    no sabotage, no bad behaviour; just one more schmuck shoved out the door at the end of the day.

    Doing something like what this wanker did should have been an instant H1B revoke, green card option permanently revoked, and time in chokey, and then shipped out of country.

  13. razorfishsl

    I'm dealing with this situation at the moment.

    An out of control admin , causing havoc in the network...

    I want him GONE & passwords changed.

    managements comment .... just work it out with him....

  14. matjaggard

    Hard to have sympathy

    As I was reading this article, WebEx was failing to load for me. Maybe they have rogue employees every week?

  15. Aussie Doc
    Paris Hilton

    Hmm.

    If only Cisco somehow could have prevented this from happening.

    Even she could help --------------------------------------------------------------------------------- ^^^

  16. TheBorg

    Its not just IT companies that are rubbish at deletion of accounts and access. I rejoined Deutsche Bank in 2013 having left them in 2006. Upon my return HR queried whether I had worked for them before as they just re enabled my HR account .... it included my next of kin / emergency contact which was my deceased mother (she passed in 2006 just before I left for the second time) . Surprised that they had not deleted my details ... Not really. :)

    1. martynhare

      You don’t delete accounts until you’re certain you don’t need to keep the SID<->Name translations intact anymore, the list of security groups granted at the point of leaving or any other associated metadata. In general, you should not delete an account but instead anonymise it many years after systems have been updated to the point where there’s no valid audit trail any more anyway.

      Sure, add a pseudonym with a confidentially stored historical record which can be called upon if necessary, but to delete accounts risks losing the ability to review the system for past mistakes or deliberate misconducts years down the line.

  17. Hazmoid

    I have left a number of companies over the years.

    Not one of them did I consider going back into their systems and looking to see if they had, in fact, changed or deleted my account. I always kept my personal and work email separate, so that I did not have to worry about getting a copy of my mailbox.

    When I was laid off at a Stockbrokers during the GFC, I knew I was on the chopping block even before the HR lady called me into a meeting in the board room. I made sure that anything personal was on my external drive and not on the work computer. I was still good friends with my boss and warned him that the company had not cancelled my home internet service that they were paying for. He was happy to let me hang onto it until I could get it signed over as he was wary that he might need me to help out remotely.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like