back to article Hapless AWS engineer spilled passwords, keys, confidential internal training info, customer messages on public GitHub

An Amazon Web Services engineer published exchanges with customers and "system credentials including passwords, AWS key pairs, and private keys" to a public GitHub repository by accident. On 13 January, infosec biz UpGuard discovered a 954MB repository containing AWS resource templates – used to create cloud services – plus …

  1. Anonymous Coward
    Anonymous Coward

    Hmm, nanny state bad, nanny site good?

  2. alain williams Silver badge

    Should not passwords be one way encrypted ?

    I get the impression that passwords, etc, were published in plain text. I always thought that best practice was to store passwords encrypted/hashed and compare this when doing authentication.

    Have I got this wrong ?

    1. cbars Bronze badge

      Re: Should not passwords be one way encrypted ?

      Ho ho, what fun! Prepare for misinformation!!

      Server side hash: essential (and a salt, and just dont bother it's really hard to do right)

      Client side hash: changing one password into another (as you still have to send something to the application to verify, a MITM doesn't care what you started with, just what the server needs to see)

      Client side encryption: Recursion. ("What shall I do with this secret key that encrypts the secret key that encrypts the secret key that encrypts our AWS password??"

      "Stick it in the same project, but in the 'secret' folder, and make sure you don't load *that* to github!")

      If you're a dev with a repository that's logging into other services like AWS, you're client side...

    2. Bronek Kozicki

      Re: Should not passwords be one way encrypted ?

      You thought wrong, but have an upvote for asking the right question.

  3. Anonymous Coward
    Anonymous Coward

    Another take home...

    NEVER MIX WORK AND PLEASURE!

    Sorry to shout but it could very likely be a case of being lazy and not correctly separating work repos (and thus secured private repos unless your workplace is a tight arse, in which case just roll your own git/CVS locally) and personal pet projects on public repos.

    Personally I take it a step further and just keep work on a work dev machine and everything else on my own machines. No chance of forgetting which git I'm signed into.

    1. Anonymous Coward
      Anonymous Coward

      Re: Another take home...

      Other than scale (and perhaps liability) whats the difference between publishing a work password/key file on github vs publishing your personal password/key file on github?

      1. Ben Tasker
        Joke

        Re: Another take home...

        One of them, your boss shouts at you, the other he laughs at you

  4. chivo243 Silver badge
    Facepalm

    That's a talking to - hate to be him

    Sir, please step into our office. Yes, about that wide open repo now?

    One can only hope this is a lesson learned, and will be applied! Forever in perpetuity.

  5. Doctor Syntax Silver badge

    "A common reason is that developers trying out some new ideas hardcode credentials into applications and then publish the code without thinking through the implications."

    Even if these are new ideas being tried out if that's the mindset of the developers it doesn't inspire much confidence in the security of the finished product.

  6. hoofie

    Engineer ?

    Engineer eh ? I'll bet 100 quid the closest the person involved ever got to Engineering was taking a bus ride over a bridge.

    1. aregross
      IT Angle

      Re: Engineer ?

      Was just told today that my job description is changing from 'Customer Service Engineer' to 'Service Technician'

      Wonder why....!

      1. Anonymous Coward
        Anonymous Coward

        Re: Engineer ?

        Service "Technicians" usually have much more experience than Service "Engineers" because they spend most of their time helping "Engineers" get their problems fixed and the kit up and running.

        But they get paid less.

        1. Anonymous Coward
          Big Brother

          Re: Engineer ?

          Maybe its changing because of things like Boeing where people who considered themselves to be software engineers weren't and people lost their lives. So unless you are an actual engineer, you shouldn't be calling yourself one?

          Granted, I am a software engineer, but not a PE. Way back when... Software engineering or even computer engineering wasn't really appropriate for a PE. (Mechanical, Electrical, Civil tended to require it.) But I'm aging myself. I don't know when it changed.

      2. Mike Pellatt

        Re: Engineer ?

        Because unless you have a professional engineering qualification and undertake CPD, validated by a recognised professional association, you shouldn't have the label "engineer".

        Germany is (or at least, was) very, very, very hot on this.

        Our lives are as much in engineers' hands as doctors', and woe betide anyone who calls themselves a doctor of medicine when they're not. Should be the same for engineers.

        Presumably amongst the divergence that our wonderful UK Government wants from EU regulations, the current Eur Ing recognition will be one we have inflicted on us.

        1. Claptrap314 Silver badge

          Re: Engineer ?

          About such things, I tend to respond, "My ancestors left there 150 years ago for a reason."

          In the US, there have been lawsuits over these. The first amendment just keeps winning.

          Now, I would never condone someone representing themselves as a Professional Engineer who lacked the credentials. I would never fail to mock someone would hired a "Professional Engineer" without checking credentials.

          The professions of PE and MD differ dramatically. The customer of an MD is typically an individual of average intelligence with no knowledge of the profession, and who is somewhat stessed. If challenged, even with the help of the InterTubes, they would be unable to check the credentials of the MD.

          The customer of a PE is a business or a government, generally working through some sort of bidding process. The person doing so has as their job description to validate the quality of the bids.

          Of course, I charged my engineering calculus students 10 points out of 10 for sign errors--I don't want bridges built with gravity going the wrong way.

          1. Michael Wojcik Silver badge

            Re: Engineer ?

            Frankly, I'm not so complacent about the medical industry's co-option of the term "doctor", either. Both etymologically and in other fields it means "scholar", and many medical doctors, while upstanding members of their profession, do no research and don't even have much time to follow current research in clinical practice. (That's why Cochrane metastudies exist: so that a team of experts can review research in an area and digest it down into clinical recommendations.)

            And in the US, medical interns - who are not yet licensed medical doctors - are generally told to use the title "doctor" with the patients they see. They are scholars - they're still in school - but the medical profession wants to have it both ways: "doctor" meaning "student" and meaning "professional who has acquired some special credential".

            1. Martin-73 Silver badge

              Re: Engineer ?

              Technically doctor means 'has a Ph.D', surely? Isn't that the reason the USA has the term 'MD', to differentiate say, Brian May, from a cancer specialist?

          2. Martin-73 Silver badge

            Re: Engineer ?

            If you call yourself an engineer, be prepared to back it up with documents, in many US states... but not in others.

            Your own personal experience will dictate what 'feels right'. However, 100% on mocking people who hire an engineer and don't check the actual qualifications. It's the reason most engineers put their qualifications after their name.

        2. Martin-73 Silver badge

          Re: Engineer ?

          In some places engineer is a protected term. In others it is not.

          Example from my own field: Telephone Engineer.

          Back in the day, it was actually quite apt, you had to have multiple C&G electrical and mechanical engineering courses under your belt, along with BPO (british post office) training courses, because you were dealing with fault finding and repairing highly complex electromechanical systems to individual component level.

          The actual guys putting phones in were called technicians or installers.

          Time happens.

          2020: everyone working in the industry (fewer than ever before) is an engineer.

  7. eldakka

    Why do so many secrets end up in GitHub repositories? A common reason is that developers trying out some new ideas hardcode credentials into applications and then publish the code without thinking through the implications.
    I prefer: Some people are idiots.

    1. Loyal Commenter Silver badge

      I'm going to go with:

      Step 1: Ooh, lets start a new quick-and-dirty test project as a proof-of-concept for something or other

      Step 2: I'll create a folder for it and stick that in a new repository

      Step 3: (omitted) I'll create a .gitignore so I don't waste space with object files, and all my test config files

      Step 4: This bit needs some cloud services. I'll stick the key in the config file

      Step 5: It's Friday afternoon, time to go to the pub; before I go for the week, lets make sure I've committed everything I'm working on...

      1. malfeasance

        so-called engineers that don't understand the tools they use

        You're bang on, seen that so many times; which is confusing since you can easily have an excludes file in your ~/.gitignore a-la

        [core]

        excludesfile = /home/user/globals/gitignore

        And then in that gitignore file have a "__localonly" line; this means, doesn't matter what project I'm working on, I create a __localonly directory and stuff all the hard-coded nonsense in there.

        Never gets checked in, never gets pushed, doesn't ruin anyone elses pristine filesystem...

    2. zuckzuckgo Silver badge

      Maybe they were told to "Move fast and break things." and could not find anything else to break?

  8. Captain Scarlet
    Paris Hilton

    claimed no customer data or company systems were exposed

    So what they found was examples??

  9. Anonymous Coward
    Anonymous Coward

    Wah wah waaaaaah

  10. iron Silver badge

    So even if you do manage to secure your S3 bucket correctly an AWS engineer will just push the password to GitHub anyway. Great!

    1. Version 1.0 Silver badge

      Push the data to the cloud and you will see rain sometimes.

      1. Loyal Commenter Silver badge

        "The cloud" is just someone else's computer. I wouldn't let "Someone Else" use my computer...

  11. Unicornpiss
    Meh

    "no customer data or company systems were exposed."

    Translation: "We're too lazy to change all our passwords and notify everyone relevant, globally"

    1. Anonymous Coward
      Anonymous Coward

      Re: "no customer data or company systems were exposed."

      AWS Security will wake up engineers to rotate passwords if they are not able to rotate passwords themselves without impact. They will also contact any impacted customers (accounts with premium support would get a phone call) to rotate passwords or keys, and run automated audits of the use of the exposed keys and accounts.

      Basically action taken by this individual was a violation of security policies (and mandatory annual training that goes with it), each of which is a fire-able offense.

      AWS employees get personal AWS accounts that use SSO federation, so you never need the root credentials. Service accounts are secured similarly. The only passwords AWS engineers need to remember are their own network logins and 2FA PIN/password. There are systems available that take secure care of credential and key materials.

      Of course, there are services (including self-service git repos etc. etc.) available to engineers to backup any documents or code securely, there was no business need to use a github repo (or even an S3 bucket).

      1. Loyal Commenter Silver badge

        Re: "no customer data or company systems were exposed."

        The fact that this was an unstructured repo (not source code) and contained things such as training materials, makes me think this wasn't an engineer, but some breed of marketeer or trainer, who was using git as if it was a file managemet system such as sharepoint, for its versioning abilities, but nothing else. The fact that they seem to have had access to some client's keys implies that they might be relatively senior, in which case I expect that they will get off scott-free for their transgressions.

  12. Claptrap314 Silver badge

    You need a better story

    The simplest story is that someone put their home directory into git. Either to facilitate a migration between computers, or to facilitate data access.

    I've only heard about such things from techies. I really, really doubt marketers would have the wherewith to do such a thing.

    So--someone puts their homedir into git, then realizes that they need to push the data somewhere so that they can grab it from outside the company.

    Whoops. Github repos are public by default.

    So yes, no customer data impact. Limited Amazon corporate data leak. Significant Amazon data leak relative to the individual. Huge personal data leak for the individual.

    1. MachDiamond Silver badge

      Re: You need a better story

      "I really, really doubt marketers would have the wherewith to do such a thing."

      I've worked with lots of people that know just enough to be a huge menace. They watched a bad YouTube video that never covered the downside of the operation or the person stopped watching before they got to the warnings.

  13. MachDiamond Silver badge

    If not when

    Every time I see one of these stories it reinforces my aversion to posting data on the internet. GitHub, AWS, etc. Something that's open source from the start, no problem. PII/passwords/credentials, problem.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like