back to article Sole Equifax security worker at fault for failed patch, says former CEO

Recently-and-forcibly-retired Equifax CEO Rick Smith has laid the blame for his credit-check biz's IT security breach on a single member of the company's security team. In testimony on Tuesday before a US House subcommittee on consumer protection, Smith explained that Equifax has a protocol whereby news of important software …

  1. Anonymous Blowhard

    And what about the management team who set up a "system" that would break if one person forgot to do something?

    When you have fallible components, like humans, then the systems that include them have to have redundancy (not firing them) and resilience; usually at least two people involved to ensure that the check-lists are followed correctly, maybe a third person to test and sign off the updates. If these things aren't in place it's because the "system" is deemed to be of low importance and not worth spending money on to get it right.

    Putting the blame on a single person is just scape-goating of the worst kind.

    1. Bill Stewart

      "Human Error" is bogus in airplane safety too.

      Sorry, one guy who misses something is not much different from one guy being on vacation, or out sick. One manager saying "we can't do the update this week, because X" might have the ability to delay it, but if your system doesn't keep track of that PENDING SECURITY-CRITICAL UPDATE WHEN YOU'RE A FINANCIAL COMPANY, your system is broken by design.

      1. Mark 85

        Re: "Human Error" is bogus in airplane safety too.

        Spot on. Single point of failure here. Was the guy not there? Was he being pushed by manglement to do something they considered more important? Or was he someone sitting in some outsource shop perhaps in India? This is a fail at so many levels from top level manglement all the way to the poor schmuck who'll take the blame.

  2. Anonymous Coward
    Anonymous Coward

    One individual my arse. Where's the governance?

    1. Anonymous Coward
      Anonymous Coward

      Came here to say exactly this.

      In my company, we have oversight. We have a security function. Governance. We have dev-ops. Governance. We have a CR process that involves sign off from anyone affected by the possible changes. Governance. We have testing of patches before CRs. Governance.

      What we don't have is leaders that lack a spine and should be involved in freak yachting accidents.

    2. FlamingDeath Silver badge

      Out playing golf, doing secret handshakes

      where else?

  3. Banksy

    What a load of rubbish...

    So their IT team doesn't know a patch is available without someone sending an e-mail about it? They don't have patch management software? Individual software packages don't alert IT about patch availability?

    More likely scenario is someone was told not to apply the patch for one reason or another.

    1. Adam 52 Silver badge

      Re: What a load of rubbish...

      I think you are over estimating the effectiveness of patch management software.

      I can't, for the moment, think of any server side frameworks that notify about patches. Vulnerability scanners (and Equifax had one) can only scan what they know about, which is inevitably less than everything.

      1. Alan Brown Silver badge

        Re: What a load of rubbish...

        "I think you are over estimating the effectiveness of patch management software."

        And underestimating the effectiveness of decent trouble ticketing/inventory systems.

        Once systems have been flagged as requiring updates, a decent system will flag a warning if it's not done inside X time limit, which means that the team can look into why it didn't happen - and if someone's ordered it not be updated, there would be an audit trail on that too.

        trouble tickets aren't just for the endlusers and helldesk.

    2. phuzz Silver badge

      Re: What a load of rubbish...

      It's worse than that! Their IT team clearly don't even read The Register!

      Or any other IT publication that reported the Struts vulnerability come to that.

  4. JerseyDaveC

    Anyone got their auditor's phone number?

    I'd love to be a fly on the wall at their next ISO 27001 audit. Auditor: "You rely on one person for some of your critical patching, and there is globally known evidence that you demonstrably don't have a robust process. That'll be a Major Nonconformity, then ... I'll be back this time next month for you to show me evidence of improvement that convinces me not to take away your accreditation."

    1. smudge

      Re: Anyone got their auditor's phone number?

      Except, of course, that they will have that evidence ready long before the auditor rocks up.

      Now, if a previous audit had identied the problem and told them to deal with the risk, that would be more useful. Especially if they decided to accept the risk :)

      1. Naselus

        Re: Anyone got their auditor's phone number?

        That would be Deloitte...

  5. Dabooka

    What about intrusion detection

    Okay, even assuming his utterly bollocks excuse about 'Gerry in IT forgetting to insert floppy #2' holds some merit, I can't help but think that they clearly lacked in other areas.

    140m people. Think about that for a minute, that's lot of people. Someone needs to go the slammer for this but it's mire likely to be 'Gerry' than anyone with Chief in their title

  6. Andy E

    Double failure

    If you take Smith's words at face value then there was a double failure here. One guy failed to notify others to apply the patch and their vulnerability scanning software failed to pick it up. While the human element is fairly easy to fix I'm at a loss to see why their vulnerability scans didn't pick up on the known issue in the months following the release of the patch. Perhaps they aren't updating this software either? Would that make it a triple failure.

    If someone did mess up as Smith says then kudos to him for telling the truth about what happened without naming the individual and taking personal responsibility for it.

    1. netminder

      Re: Double failure

      There is a third failure. They failed to have prevention systems in place. WAFs, properly configured would have stopped the attack since it used known strings.

      But lets all ruin Gerry from securities life and career because we dropped a billion and still couldn't stop the simple stuff.

      1. Anonymous Coward
        Anonymous Coward

        Quadruple failure!

        They failed to detect the data being ex-filtrated. Upper management blaming this on one person? No failures there.

    2. Morten Bjoernsvik

      Re: Double failure

      As CEO your main job is to be responible. You are paid to be there when the shit hits the fan. Kicking downwards definitely not. No respect.

  7. Redstone

    Hackers Take Note:

    Every year, when IT Bob is on his two week vacation, Equifax will have no updates and is wide open!

    1. Gustavo Fring

      Re: Hackers Take Note:

      No dumbass ... they get Jillian from reception to step in and fill his shoes. Like when hes off over xmas?

  8. LaFin

    Mgmt investment in multple layers of security

    In which case this entire chain of questioning has missed the fundamental point of having multiple layers of security so that patching of one competent being late/missed does not open all mass data breaches......if this was a small business one might understand, but this is meant to be enterprise architecture and enterprise calls IT. Stinks of years of chronic under investment, and lack of understanding (or care) of basic security principles. CEO and CIO asses should be hung out to try, and stop blaming it on the little guy.

  9. fnusnu

    Where have I heard this before?

    Management exonorated, tech(ies) thrown under a bus.

    Oh yes, after every breach.

  10. chivo243 Silver badge

    If I'm ever a CEO

    I will remember this quote:

    Smith's reply was: “That is my understanding, sir.” (My handlers spoon fed me this canned response)

    So, Smith really had no knowledge of the issue, but preceded to dodge the question's intent, and then shift blame.

    That's a proper corporate screwing...

  11. Potemkine! Silver badge

    Bloody underling!

    I guess that blaming an intern would lack of credibility, so the CEO found another scapegoat to try to hide the real scandal about this story: Equifax owns zillions of personal data on people who have neither way to know about what these data are nor any mean to stop Equifax owning them.

    Talking about scapegoat, I would like to thank the EU for the GPRD!

  12. Will Godfrey Silver badge

    A sort of inverted honesty

    He has just proved beyond all possible doubt that he is a lying weasel.

    I doubt even a 5 year old would try to pull something like that and expect to be believed.

    1. Version 1.0 Silver badge

      Re: A sort of inverted honesty

      Typo alert: "He has just proved beyond all possible doubt that he is a lying weasel." should have read "He has just proved beyond all possible doubt that he is a candidate for promotion to upper management."

      1. hplasm

        Re: A sort of inverted honesty

        Typo alert: "He has just proved beyond all possible doubt that he is a lying weasel." should have read "He has just proved beyond all possible doubt that he is a lying weasel AND THEREFORE that he is a candidate for promotion to upper management."


  13. Amorous Cowherder
    Thumb Up

    Yep it was all Fred's fault, no denying!

    Fine, blame that one person, make an example of them, drag them through the courts for as long as they live just to punish them for what they did but know this, it will happen again. I will stake my house on that bet because the management teams allowed themselves to be in a position that made them reliant on just one person, one single point of failure. In IT we live and die trying to avoid the "single point of failure", this is exactly what happens when that is allowed to happen and it goes catastrophically bad.

  14. Anonymous Coward
    Anonymous Coward

    I wonder if this is even true

    This sounds like a system they developed after the fact to insure it wouldn't happen again. Have someone responsible for notifying the right people about the patch, with the sanity check of automated scanning following up in case something goes wrong in the manual process.

    Let's say Bob the sysadmin did notify people that 'struts needs to be patched', along with dozens of other patches that were required that same week, not to mention all the ones that came before, and came after. Knowing what you need to patch isn't the hard part, it is having a process down that lets you actually do it in a timely manner.

    This is a patch that's critical in hindsight, but at the time it wouldn't have looked very important compared to some. There are SO MANY patches coming out from so many sources, I wouldn't be surprised if this was considered lower priority and was in some stage of application (maybe had been applied on a QA system and was sitting in a CR for eventual application in production) but they don't want to admit that because it makes them look bad.

  15. steviebuk Silver badge

    I call bullshit...

    ...and did they question (I haven't watched the video) on why there was that single point of failure? That engineer, if the story is actually true. What would of happened if that person was off sick?

    1. chivo243 Silver badge

      Re: I call bullshit...


      Or hit by a bus? Or thrown under one!

    2. Cynic_999

      Re: I call bullshit...

      Most companies have a plan in place to deal with the times when an employee is absent from work for whatever reason. What they usually don't have is a way of detecting when an employee is at their desk but suddenly failing to do their job. Perhaps his dog died. Perhaps he was just informed that his wife has terminal cancer. There are many reasons why a good & reliable employee can suddenly drop some major balls. Which is why you need at least one person shadowing every vital position.

      One pilot can easily fly a modern airliner but airlines pay for twice as many pilots than are needed to do the job. The other pilot's job is to ensure that the pilot flying the aircraft is doing it correctly.

      1. Anonymous C0ward

        Re: I call bullshit...

        And do you think the likes of Ryanair would continue to do so if regulators didn't insist on it?

  16. streaky


    1. Run Nessus

    2. ????

    3. Profit!!!

    This can't possibly be how a fortune 1000 company and one of the world's largest holders of critically private personal information secures data. Where's your fucking red team?

    Shit is cultural from the CEO down.

    1. wheelybird

      Re: So..

      Run OpenVAS. It's free and so EVEN MORE PROFIT!!!

  17. Christoph

    Patch checking - initial basic scheme

    All incoming patch notifications shall be logged in a database.

    All applications of that patch shall be logged against that notification.

    All un-applied patches shall be listed, and notifications with increasing levels of urgency shall be sent out the longer the patch is un-applied.

    Hardly rocket science.

    1. Brewster's Angle Grinder Silver badge

      Re: Patch checking - initial basic scheme

      If the first step fails to happen, then the remainder aren't worth the time they took to type.

      1. Christoph

        Re: Patch checking - initial basic scheme

        Yes, it obviously needs more detail - that's why I said initial basic scheme. There's all sorts of bits that need adding to get it resilient, but they don't seem to have managed even that very basic initial setup.

    2. Anonymous C0ward

      Re: Patch checking - initial basic scheme

      It also assumes you have an up-to-date inventory.

  18. Nimby

    It's 2017 and NO ONE practices basic security yet.

    It's sad, but like every breach before it (and undoubtedly every breach after) by every major company and/or government agency, basic concepts of security that are industry-known were just plain ignored. Every single one has been and will be a "WTF?!" moment, and this one is no exception to that by any means.

    Of course with limited-to-no accountability, is this really a surprise? Expect much more of the same in the future. As long as the government does so little, so will the children it herds. We have ridiculously complex building codes for planning/building a house to keep people safe, but we have next-to-nothing for critical life-impacting data storage.

    What makes this one worse than all of the others is that it did not even involve "customers", as that would imply people signed up to something. No, this is a company that you can't even opt out of. They nom nom nom all your data to provide a questionable "service" and too bad to you.

    1. Anonymous Coward
      Anonymous Coward

      Re: It's 2017 and NO ONE practices basic security yet.

      At least their details will be out there too.

    2. Rasslin ' in the mud

      Re: It's 2017 and NO ONE practices basic security yet.

      Speaking from many years' experience with building codes and building inspectors, I assure you those aren't useful as models for how to manage anything unless you propose to cite them as bad examples.

  19. wolfetone Silver badge

    I wonder if the sole Equifax employee thats "to blame" for this has also been forced in to retirement or is still on the payroll waiting for his P45?

    Capitalism is a wonderous thing when the CEO gets to retire, wipe his hands of the whole sorry mess on a lovely pension while he throws the staff under him under the Pirelli's of a bus.

    1. Anonymous Coward
      Anonymous Coward

      I hope he has kept all the passwords secret, maybe he will do us a favour and shut all the servers down before he leaves.

  20. DainB Bronze badge

    Not plausible

    If they have in-house admins looking after such a critical piece of infrastructure as externally facing website I do not believe for a second that those people would miss a patch for a systems they know and care about. Just not possible.

    Now if we consider that there might be someone in India, Romania or other proverbial Mongolia who is supposed to be looking after web farm but would not give two shits (or simply not allowed to do anything) without SR and CR which was never logged it makes much more sense.

  21. Chairo

    He laid the blame

    how lame!

  22. Anonymous Coward
    Anonymous Coward


    I always thought the logic for directors being paid so much was that they were responsible.

    Therefore he was acting fraudulently and should repay everything he earned.

  23. Anonymous Coward
    Anonymous Coward

    Only one

    Of course there was only one person responsible, the rest were outsourced a long time ago.

  24. 0laf

    Yes it was one "Lone Wolf"! His CV was so good as well, he'd been working previously on security for Talk Talk, wireless infrastructure for TK Max and before that designed diesel emission systems for VW.

    It's almost like this guy is a professional blame hound.

  25. Richard Pennington 1

    Next question

    OK, how do I go about cutting all links with any company who sends my data to Equifax?

    1. Bronek Kozicki

      Re: Next question

      Start with closing you bank account, any credit cards you have and closing all debts you might have. Then replace contract phone with pay-as-you-go one and sell your car. If you are very careful not to enter into new financial liabilities for the next few years then who knows, maybe your record won't be updated - but then it might start showing AWOL, so it's not very good either.

  26. jms222

    Why does system rely on one component

    I still fail to understand how a vast quantity of data leaked because of the failure of one component, Struts in this case. These components will always have security issues that pop up and are not fixed in time. Or indeed not discovered for months.

    The backend database or its monitoring system simply should not allow the usual trickle of data requested by the web backend to turn into a torrent (pun intended). We really are talking about a torrent and nothing subtle here.


    That's spectacular.

    That CVE was in the weekly US-CERT alert email. It was discussed in the tech media. It was on The Register front page. The initial fix was in April's Quarterly Patch Set, and somebody in the company should have got an email about that. Here's the advisory that I got in my email. It's pretty clear about the risk.

    Equifax has (or had) 9500 employees, and only ONE person was responsible for keeping an eye on the alerts?

    Nah, sorry, it's not that employee's fault. What they have there is a failure to take this stuff seriously.

  28. Drew Scriver

    Management is not responsible for their decisions...

    If indeed it is true that the failure of "a single person" lead to the issue, the obvious conclusion is that it was *management* failed to implement proper controls. Leaving the monitoring of the "news about critical vulnerabilities" to a single person would seem to point to management, not this person.

    I wonder what would happen if this "single person" were to be forced to appear before the House Committee. Chances are that we'll hear a) he's overworked, b) he lacks authority, and c) there were ample warnings but the company tends to downplay or ignore those.

    Remember when [a very large bank in the US] got hacked? As a customer, I was concerned. Yet, when I contacted them (multiple times) that Qualys rated them as "F" on their main web site this fell on deaf ears. I even went to a branch and talked to the branch manager. Mentioned PCI-DSS. Turns out she'd never even heard of that, not did she share my concerns. I closed my account on the spot.

    Similar story with [one of the largest ISPs in the US]. Eventually I did speak to an IT-engineer who was in fact in the know. He let slip "we know, but there's nothing we can do about it because management doesn't listen to us".

    Couple of years ago I had to cover for a co-worker who was on vacation and configure a web interface for an application from a large European vendor. Since it processed credit cards I figured it ought to be PCI-DSS compliant. One quick look told me it wasn't - first giveaway was that the vendor name appeared in all (public-facing) URLs. Second was that it was installed (per their instructions) in the default locations.

    Did a quick test and discovered that the error log was a text file in the root of the web site... Every error was written there - even credit card failures (e.g. address verification errors). And, you guessed it, *all* the transaction details were there: name, address, card number, expiration, CVV2, etc.

    Now came the fun part of alerting the vendor. And, you guessed it, I got the expected, "Oh no, we are in fact PCI-DA certified". And, you guessed it again, they alerted my executive management to complain that I was a roadblock...

    Fortunately our security officer stood his ground and blocked the project from going forward, but I'm afraid that these situations are commonplace.

    This will not improve unless:

    1) Legislation is enacted to hold executives personally responsible for willful failure (or ignorance).

    2) A clearinghouse is set up where consumers and security experts can report vulnerabilities. Reports would have to become public automatically a set time after a patch is available.

    3) Companies are mandated to create processes so employees and customers can report security issues. Again, with full disclosure after x days.

    4) Fines for failing to properly protect PII.

    All without exceptions for small companies, non-profits, government agencies, and the like.

  29. Pat 9

    “It did not help that hurricane Irma took down two of our larger call centres in the early days after the breach,” he said.

    Hurricane Irma was in September, the breach was discovered in July. Sounds like he is trying to use any excuse he can to make it sound like it was beyond their control.

  30. dmacleo

    PFY needs to build him a robot....

  31. Patched Out

    One deterrent ...

    Would be to change the laws so that if a company (or the government) has such a data breach, no matter what the reason, they should be required to provide the effected individuals with fraud identification, protection and insurance FOR LIFE.

    This only providing one year protection crap is completely useless.

    1. Anonymous C0ward

      Re: One deterrent ...

      From a different supplier. Because otherwise the emperor is just generously donating me his non-existent clothes.

  32. tomban

    What we've got here...

    Is FAILURE, to communicate.

    Some men you just can't reach.

    1. Nunyabiznes

      Re: What we've got here...

      +1 for the Cool Hand Luke reference.

  33. Aodhhan

    No malicious insider prevention/detection?

    What I'm hearing is the CEO and CIO of a firm selling itself as a 'security guru' for large corporations didn't itself didn't have the foresight to implement controls to prevent a system breach in the event someone didn't follow policy and procedures whether by accident or with malice intent?

    In other words... the security brains didn't implement a malicious insider attack prevention/detection strategy? These are the policies, procedures and checks put into place to ensure an insider attack (whether on purpose or by accident) doesn't occur or is quickly detected.

    Also... if automated scans aren't taking place, then most organizations will work overnight and demand the vendor get on site immediately to rectify the problem. This itself is also a management screw up.

    So stop blaming one blue collar individual. Processes, procedures and QA/QC/Audit, not to mention communication is all on management and executive leadership.

    What I see is poor risk management caused by lack of education and experience by everyone from the CEO down to the first line supervisors at Equifax.

  34. Anonymous Coward
    Anonymous Coward

    When good people get punished

    ..we all suffer. Why work in IT security if you are personally punished for any failing? We all lose if the talent moves to something less risky. And the talent will move if it's clear that lack of mitigation strategies continue to be ignored. As everyone is saying here, it's not a working system without it. Those who will continue the work will spend otherwise productive time doing cya. So we lose talent, lose productivity, never address the root issue. We will all certainly suffer if the worst of this particular breach comes to pass.

    Taking one more step back, there needs to be mitigation for the overall data and personal liability problem. Otherwise companies that get punished for mishandling personal data for their product will stop performing their functions, stop storing personal data altogether... oh wait

  35. tmecimore

    Roasting Scape goat

    So Equifax must have been using some super secret vuln reporting system. That only one person had access to, and no reporting or access for review by enterprise partners. I know what a roasting goat smells like, and this scapegoat mimics that.

  36. GnuTzu

    Face Palm: Patch Policy Requirements Should be Obvious

    I can't believe how much trouble I've had getting organizations to establish accountability with mandatory patch tracking, review, and installation, with management oversight--and that's in PCI DSS environments. People don't seem to like something that looks like regulation. Well, identity's are now pwned. Don't we get an "I told you so".

  37. The Wild Tomcat

    Not so difficult, folks

    When I was working for IBM's Strategic Outsourcing unit, I worked two (unpaid) weekends a month installing patches. Our team covered a base of 400-something servers. Every patch. Every server. Every time.

    This is something that's stupefyingly easy to get right.

  38. WhatTheHellDoIknow?

    A little confused

    When was there ever a patch notification? The remediation was to upgrade to a newer version of Apache Struts.

    Pretty sure 1 person is not responsible for ensuring an entire system upgrade takes place

  39. Boris the Cockroach Silver badge

    Well this

    just proves that the CEO/CIO of equifuckup are worth every penny of their salaries and did their best to run the company for the benefit of its shareholders and to give great and timely customer service, we therefore dismiss all charges against them and they are free to go.

    Bob the IT intern , you stand charged and have been proven guilty of allowing the hack by not updating stuff, therefore crashing the company and costing the shareholders millions of dollars .. oh and ruining the lives of millions of people, we therefore sentence you to 140 million life imprisonment terms. or the death penalty.. whichever you prefer

    I think I'm too cynical sometimes.....

  40. Atilla_the_bun

    What about SOX?

    You know in the US we have this little thing called The Sarbanes Oxley Act of 2002, or SOX by those in the know. I would dearly love to hear from someone who is a lawyer (certainly not me) if SOX is relevant here. Why? Because it puts C Suite people behind bars for carp like this. IMHO this needs to happen here.

  41. Mike Shepherd

    "Sole...worker at fault..."

    That's convenient. No need, then, for the company to shrivel and die like Arthur Andersen.

  42. Sudosu Bronze badge

    Security resource identified.

    According to sources at the firm the former security guy's name is Tibor Jankovsky,

    1. diodesign (Written by Reg staff) Silver badge

      Re: Security resource identified.

      ^ Simpsons joke...


  43. shawnfromnh

    Maybe it's because the head of security was a music major and why isn't she being blamed is my question. Also Equifax just got a 7.5 million dollar contract with the IRS, how the hell can they be trusted with anything after this last fiasco, this is crap and the IRS person that signed off on this needs to be fired or whatever dept.

  44. A Nonny Moose

    He's right y'know

    He's right, this whole mess is entirely the fault of one individual, and that individual is:

    Rick Smith

  45. Anonymous Coward
    Anonymous Coward

    Corporate-Congress Parent-Teacher-Meetings

    No politician ever asks, how could you let failure hinge on a single person:

    1. in a company of 10,000 workers,

    2. in a world where security is a critical issue for every single corporation,

    3. where your corporation is highly visible due to its precious data-vault,

    4. when you have questionable security due to being hacked before...

    ....And the Answer is: Its all lies... It never happened... Internally and externally its easier to just find a willing victim or person to single-out to fall on their sword. That way everyone else can feel warm and cozy about themselves and the job they're doing.. Better for morale this way!


    So why are they letting Smith get away with answers like that??? Its like the MSM re-printing corporate-PR-statements about 'your security / privacy' is important to us', when clearly its nothing but the opposite.

    Why? These meetings are all for show just like big banking. Put out fires before the lynch-mob gets here. Smith is such a professional corporate suit type too, unlike parent-teacher meetings, he probably had a dozen PR experts groom him for this type of 'question time'... He then concluded the lone-wolf blame-game would placate congress the best.... Sick!

  46. razorfishsl

    LOL,,,, what a crock.

  47. Anonymous Coward
    Anonymous Coward

    The Blame Game

    Psychological Theories: Impression management, Self-Preservation, and Self-denial. While I acknowledge that I do not have all of the information on this event, to blame a single individual is shameful. This is a testament to the powerful blaming the ditch digger. Anyone in IT understands the complexities surrounding security patch updates. For example, software vendors are ostensible and specific to warn customers that security patches are NOT regression tested. Further, management in some organizations drive what is updated and when (e.g. Change management). The application of any patch on any system ALWAYS comes with potential consequences.

  48. This post has been deleted by its author

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon