back to article Software bug costs Citigroup $7m after legit transactions mistaken for test data for 15 years

A programming blunder in its reporting software has led to Citigroup being fined $7m (£5m). According to the US Securities and Exchange Commission (SEC), that error [PDF] resulted in the financial regulator being sent incomplete "blue sheet" information for a remarkable 15 years – from May 1999 to April 2014. The mistake was …

  1. MrMur

    Mistake, or...

    A programming mistake or was it a case "Ah Mr. Corleone, I gather sir is interested in one of our 'discrete' accounts.... yes sir, we call it the 10-alpha account. It's much more convenient than Panama"

  2. RaidOne

    And SEC did not catch it?

    So SEC asked for the data, received it (I suppose, cause 8 months have passed until City confessed) and then they did not discover any error? I mean, the opposite banks involved in those transactions missed them as well? Or SEC did nothing with the data?

    1. Bronek Kozicki

      Re: And SEC did not catch it?

      I'd be surprised if SEC bothered to correlate transaction data from all the banks sending the data, and it wouldn't make sense to correlate data from some banks only.

    2. allthecoolshortnamesweretaken

      Re: And SEC did not catch it?

      If you know that you don't know something, you'll start looking for it.

      If you don't know that you don't know something, well ...

  3. Christoph

    One of the simplest checks of all

    Here are the details of all our transactions. Here is the grand total of all our transactions. WTF they don't match up?

    1. Old Handle

      Re: One of the simplest checks of all

      Presumably the "testing data" would be excluded from the grand total at all.

      1. Christoph

        Re: One of the simplest checks of all

        No, you come at the grand total by a different route, checking at a different point.

        For example, every time a transaction actually happens the stored procedure which implements it (and does all the security checks, and cannot be amended except with a *lot* of safeguards) also adds the amount to the grand total. There's various other possible ways.

        1. ckdizz

          Re: One of the simplest checks of all

          Yeah, that's what should happen in the FMIS. But first, you're talking about reporting, not accounting, which although it uses some of the same data, doesn't do the checks that accounting does. You might have the reporting function do double entry checking, but why would you when the FMIS does that.

          Secondly, you're talking about a bank that implemented a new coding system and didn't bother checking if it worked properly. For 15 years. That kind of precludes them from even thinking about doing a redundant check on the reporting data, let alone being able to implement it without sinking 30% of the world's cargo ships at the same time.

    2. Anonymous Coward
      Anonymous Coward

      Re: One of the simplest checks of all

      > WTF they don't match up?

      They would likely match up, but they should have always had extra money in their accounts. You'd think someone would have noticed.

  4. Dave Pickles

    Interesting footnote

    "The reporting logic treated letters, such as A, B, C, as being less than the number 0. Therefore, 10B was treated as being a three-digit number with a value less than 100."

    EBCDIC presumably?

    1. JMiles

      Re: Interesting footnote

      Who you calling an EBCDIC!

      (Just kidding of course). Having seen something very similar happen at a large 3 letter NGO based in the land of gold and chocolate... I can see how this sort of stuff happens when code gets outsourced, then offshored and little bits of knowledge get lost along the way and then one day. BAM.

      Twice in my career I've found such errors when inspecting code trying to debug it and finding serious errors in financial calculations. Both times the project management were in shock and took remedial action but hushed it up from going higher up the chain out of embarrassment.

      1. AustinTX

        Re: Interesting footnote

        Ohh, ohhh, you and your fancy *extended* binary-coded decimal interchange code! I suppose you work entirely from the command-line? None of that silly realtime operating-system conversation operative environment?

      2. Anonymous Coward
        Anonymous Coward

        Re: Interesting footnote

        Once when contracting to Ericsson AU - many years ago - I found a similar error... then mentioned it to my supervisor (very old lady near retirement, reporting to the CFO).

        She told me to ignore it, and not mention it to anyone. Which I thought was weird.

        I mentioned it to the CFO later on (as a young, dumb, clueless person does) who was shaken at the time.

        Needless to say, I was immediately fired by my supervisor. Who naturally, probably had a good excuse to the CFO and never needed to follow up it.

        At least, not in the few months until she retired. :/

        Thankfully, I've learned to be more discrete over the years when finding corruption and things of similar ilk.

        a) Document things properly

        b) Have (distributed) backups, that others will access

        c) Have a chat to the corruptor. I can/perhaps/maybe overlook the problem... but what do I get out of it if I do? (document this)

        d) If possible, find someone external to report to. Else, Wikileaks/similar.

    2. Anonymous Coward
      Anonymous Coward

      Re: Interesting footnote

      "EBCDIC presumably?"

      I'm thinking perhaps the original programmers didn't know or had forgotten about that and assumed ASCII expecting strcmp() or a similar string comparison function to work ok. Either that or they never expected letters to appear in those 3 positions. Either way, clearly their testing procedures sucked.

      1. Antron Argaiv Silver badge
        WTF?

        Re: Interesting footnote

        Hmmm...the field used to be a numeric one and then became alpha, but someone didn't go back and check that the code was compatible?

        Sounds like something right out of The Daily WTF...

    3. Anonymous Coward
      Anonymous Coward

      Re: Interesting footnote

      > The reporting logic treated letters, such as A, B, C, as being less than the number 0.

      That seems unlikely to me. I'd believe it if they said they excluded anything over 89.

      That or if they said it was a Y2K/Windows 9 type thing where it would look if the first two digits were 9 or 10 and skip it.

    4. Mpeler
      Coat

      Re: Interesting footnote

      Another half-ASCII solution foisted upon us by the banking industry...

      (Grabs me gold bars - hmmm, I didn't think they were THIS light)...

    5. Mpeler
      Coat

      Re: Interesting footnote

      And they discussed the results over punch...

  5. AndrueC Silver badge
    Facepalm

    I once created an array of months and left the last one out. Luckily we spotted it eventually. I'm pretty sure it came to light on the 7th of Monday :)

    1. Mpeler
      Pint

      Better than the 12th of never...

  6. Pirate Dave Silver badge
    Pirate

    2,300 requests in 15 years? So about 3 per week then? That seems like a lot. Is that a lot, or fairly normal?

    1. Pascal Monett Silver badge

      It's likely a drop in the water. Banks manage thousands of transactions every minute. 3 per week verges on basically invisible.

      Except that, small numbers of regular transactions could mean large sums. We won't get any info on that part though.

      1. Bronek Kozicki

        Almost there. The number of non-reported transactions was close to 27,000 ; the number of requests (that is, how many reports were created) is 2,300 . Meaning on average each report missed nearly 12 transaction. That's tiny but as you correctly note, there might (or might not) be large sums involved, or other signs of "noncompliant behaviour" which were effectively hidden from SEC.

        1. Anonymous Coward
          Anonymous Coward

          What it does further highlight is how ineffective and useless their regulator(s) are.

          For 15 years they have been receiving (albeit unintentionally) falsified information and haven't spotted it. Not even remotely acceptable. If they don't notice that without being prompted then how are they going to spot deliberate wrongdoing being covered up? What are their regulators actually regulating?!

      2. Pirate Dave Silver badge

        "Banks manage thousands of transactions every minute."

        That's transactions. I'm talking about 3 official requests from the government for transaction REPORTS every week for 15 years. Which to me seems a lot - ie - did they think Citi was up to no good and was keeping a close watch on them? Or is that amount just par for the course for banks?

    2. razorfishsl

      It is completely irrelevant if it is a "lot" or not.

      How would you feel if a bank 'lost' any of your transactions for reporting.

      Let us not forget that it is the monitory value not just the transaction count that is the problem.

      Sorry... we forgot to report this 1 billion $ transaction....

  7. AustinTX

    If You Profit From Breaking The Law

    If you profit from breaking the law, the government will demand a small cut of said profits.

    It's too bad those "three strikes" laws don't apply to Corporate Persons.

    1. AustinTX

      Re: If You Profit From Breaking The Law

      This small cut of the profits is called "a fine" because it makes your illegal profit just fine.

  8. Mephistro
    Facepalm

    "...and it must pay the largest penalty to date for blue sheet violations."

    Seven million dollars? You fucking kidding me?

    1. DNTP

      Re: "largest penalty to date"

      How much do you guys want to bet that someone or some group within the company made more than 7 million bucks out of this?

      1. Anonymous Coward
        Anonymous Coward

        Re: "largest penalty to date"

        "How much do you guys want to bet that someone or some group within the company made more than 7 million bucks out of this?"

        I wouldn't bet much at all. These were just normal transactions that due to a bug didn't get included. There is no reason to think any of them were suspicious or the bank made any extra money out of not forwarding them to the SEC. I know the banks are hardly shiny white , but in this case it clearly looks like cock up , not corruption.

  9. Oengus
    Facepalm

    A better question

    I would question what "testing" data is doing in a production system? Reporting from a production system should never filter out "testing" data because it shouldn't be in the system in the first place.

    1. Duncan Macdonald

      Re: A better question

      Sometimes the only way to be certain that the production system actually performs in the same way as the development system is to exercise it with the same data. Without that check it is possible for the production system to have an unnoticed coding difference from the development system (Yes in theory this should never happen but in the real world it does happen that a patch fails to make it from development to production.) It is also possible that size differences between the development system and the production system can lead to errors (eg if there is a fixed size array that overflows) as in many companies the development/test systems are not as large as the production systems.

    2. Anonymous Coward
      Anonymous Coward

      Re: A better question

      "I would question what "testing" data is doing in a production system?"

      It wasn't. The software was written to filter out a certain range of codes that was used for testing. This is perfectly normal, eg certain credit & debit card number ranges are for test purposes only.

  10. Anonymous Coward
    Anonymous Coward

    $7m for 15 years?

    Slap on the wrist

  11. raving angry loony

    Another way to look at it

    They got away with the crime for 15 years, probably made many millions out of it, and now only have to pay a piddling fine that amounts to only a few hours of income, with no other negative repercussions.

    Could someone explain to me again why the people who run corporations would voluntarily follow laws that reduce profit if this is the only kind of "punishment" they're likely to get? Or am I right in claiming that the laws of this planet are only applicable to those who don't have the money to buy those who make them?

    1. Anonymous Coward
      Anonymous Coward

      Re: Another way to look at it

      'They got away with the crime for 15 years, probably made many millions out of it, and now only have to pay a piddling fine that amounts to only a few hours of income, with no other negative repercussions.'

      A mistake, not a crime, perhaps a mistake that should have been detected earlier, but unexpected consequence of changes to systems are some of the hardest things to test effectively.

      There is no evidence that they made any money from this at all.

      Embarassing -yes, evidence of incompetence - maybe, criminal - no.

      1. raving angry loony

        Re: Another way to look at it

        Dear A.C. and the downvoters (there's a band name in there...): Before you suggest that repeated errors aren't criminal, I suggest that some day you make "mistakes" reporting your taxes. For 15 years. Then see how law enforcement deals with you.

        I'm starting to wonder if this really was the "mistake" they claim it was.

    2. Fatman
      FAIL

      Re: got away...

      <quote>They got away with the crime for 15 years,</quote>

      I feel the real crime was in not promptly reporting the issue to the SEC, and for that reason alone, they should have faced a higher fine. It would be nice to extract the fine from whoever sat on that decision's paycheck/pension/golden parachute.

      Please allow these assumptions (and I do know the ALTERNATE spelling of "assume"):

      1) The original coder never expected branch codes to have an alphabetic character in them. (In hindsight, one should not have been allowed to create an alphanumeric branch code.)

      2) WHOEVER decided to implement alphanumeric (10A, 10B, etc) branch codes did not run that decision past the IT staff responsible for maintaining the code. They had no clue to what someone in Ops was doing.

      3) No one ever bothered to test the code once the alphanumeric codes were put into use because of 1) and 2) above. This is the reason why the time span was measured in years.

      4) Most likely (and I am assuming here) when the discrepancy was discovered, the shit hit the fan. The logic mistake was fixed pronto.

      What should have happened was CGMI immediately informing the SEC, and CGMI immediately provide corrected reports.

      BUT some corporate asshole sat on the decision, and hoped it would go away. Well, it didn't. CGMI may have come out better if the "optics" of the incident were better. The "optics" being:

      1) they fucked up royally,

      2) they quickly fixed the fuckup,

      3) they provided only a partial correction of the misleading reports,

      4) they sat on reporting to the SEC for months which imparts the stench of "coverup"

      IM(NS)HO, who ever at CGMI 'sat' on the decision NOT to promptly inform the SEC should be sent on a new career path without any "parachute" (severance pay).

  12. LawLessLessLaw

    impacted

    Has it impacted you?

    No, my volume has been constant.

  13. Anonymous Coward
    Anonymous Coward

    10B or not 10B that is the question

    Whether tis nobler in the mind to suffer the unexpected output of data, or to use your arms to cover it up against the SEC....

    1. kmac499

      Re: 10B or not 10B that is the question

      NewsFlash :- New Secret Tax Haven found

      "CitiGroup revelations show the holiday resort of 10B in South Wales to be a secret hideaway for gazzillions of dollars. This also came as a surprise to the Bank which had told it's clients the money was safe near the warm sunny sub tropical beaches of New South Wales in Oz."

  14. Anonymous Coward
    Anonymous Coward

    Plus ca change...

    Nearly 40 years ago, I used to do regular preventive maintenance on some of Citibank's VAXes in London. For years they followed a standard backup routine, copying all their (presumably valuable) data onto reels of magtape which were then filed away in a special archive room. There were hundreds of reels in there.

    One day it was discovered that the backup procedure had been wrong from Day One, and not a single one of those tapes contained any useful backup data. We didn't hear which heads rolled, but presumably some did. It sounds as if the culture has not changed a lot.

    1. Anonymous Coward
      Anonymous Coward

      Re: Plus ca change...

      Yup...also been there, this time during an AS/400 upgrade. We needed some stuff off a recent backup. It turned out that the backup was corrupt. It also turned out that no one had ever tested the restore process, and that all the carefully taken backups were unusable!!

      Lesson: Do the backups....but test the restore process too.

      1. Dan McIntyre

        Re: Plus ca change...

        AS/400. That takes me back. Working at a large UK based insurance company that's only accessible directly and not through any price comparison sites, we had 3 AS/400s that the European breakdown system ran on (this insurance company also has as part of it's group a breakdown recovery company).

        They were named after members of the 1980's royal family and doing the backups was a royal PITA as it meant everyone using the system had to log off or be forced to log off. The consoles were perched on top of the cabinets so we had to climb one of those kick stools to get to them. Wouldn't be able to do it now but it wouldn't surprise me if they were still in use.

  15. DaveyDaveDave

    Calm Down

    To those of you frothing at the mouth about an evil corporation making a huge profit out of breaking the rules, I suggest you re-read the article and the report. It doesn't say anywhere that anyone made any profit, they just failed to correctly report on a relatively small proportion of trades completed by their customers. That reporting was just used to help the SEC in monitoring and investigating problems. Not reporting it didn't help that monitoring, but it also didn't directly hinder it.

    They've also subsequently identified and reported those trades, and the fact that there's no mention of them being problematic presumably means they were run of the mill trades, and it wasn't hiding anything untoward. If there were more to it, there'd be a fraud investigation, not a paltry fine for incorrect reporting.

    1. Bronek Kozicki

      Re: Calm Down

      Still, the fine $7m seems appropriate and proportionate.

    2. Version 1.0 Silver badge

      Re: Calm Down

      A fine not big enough to hurt since it appears to be just a result of a bug - but interesting all the same, I'd bet a few people are giving the old code a thorough testing in other banks now and the SEC will be asking additional questions at the next sit-down with each bank.

    3. Anonymous Coward
      Anonymous Coward

      Re: Calm Down

      > To those of you frothing at the mouth about an evil corporation making a huge profit out of breaking the rules, I suggest you re-read the article and the report. It doesn't say anywhere that anyone made any profit, they just failed to correctly report on a relatively small proportion of trades completed by their customers. That reporting was just used to help the SEC in monitoring and investigating problems. Not reporting it didn't help that monitoring, but it also didn't directly hinder it.

      This is all true. But, if the bug affects all reporting from these branches, then any money laundering going on through those branches also would not be reported up for investigation. So the question is: did someone spot the bug but leave it in and sell the knowledge onto money launderers? Probably not, but until the SEC demand the historic reports be re-run we'll never know.

      1. DaveyDaveDave

        Re: Calm Down

        Although it doesn't say it explicitly, the report does say "On April 23, 2015, CGMI finally uploaded to SIAC the data omitted from the 2,382 EBS submissions to the Commission", which to me sounds like the missed data has been submitted. Maybe it's yet to be analysed, and then maybe a fraud case will come out of it, but the implication is that there's currently no suspected naughtiness.

  16. Anonymous Coward
    Anonymous Coward

    ID 10 T or not ID 10 T is more the question

    right?

  17. disgruntled yank

    10Afly, New Jersey?

    Is Tenafly, New Jersey, the new on-shore spot for hiding money? Imagine the movie PR: "Tenafly: the clothes and hair aren't as cool as in Superfly, but there's lots more money."

    1. Tom 7

      Re: 10Afly, New Jersey?

      Bladder control problems in insects?

  18. Uplink

    Pocket change

    It's like that time when I got fined 7 pence. Yeah... That never happened and never will.

  19. Stevie

    Bah!

    SEC fines customers of Citigroup. Ftfy.

    Ah the SEC, so culpable in the sub-prime TITSUP, so evasive when it comes to anyone doing time for collusion.

  20. Antron Argaiv Silver badge
    Childcatcher

    Putting things in perspective

    WSJ says Citigroup's 2016Q1 profit was $3.5 Billion. (down 27%).

    The $7 million fine is 0.2% of their most recent quarterly profit.

    THAT will surely teach them to be more careful.

  21. imanidiot Silver badge

    Fess up or keep Schtumm

    What would have happened if Citibank just never bothered to report this error? Given the omission of the data was not noticed from either side for 15 years, would SEC have ever noticed? I presume Citibank could save themselves 15 million.

    1. Tom 7

      Re: Fess up or keep Schtumm

      I like to think some kids with Raspberry Pi's and a couple of degrees of moral compass will get jobs in places like this and actually know how to do JOINs on the odd table or two and start chucking up anomalies. If someone at the SEC notice that Citibank had noticed and fixed the problem without letting them know the $7M would look like very very very very small change.

  22. jeffdyer

    How are 10B or 10C within the range 089 to 100?

    I am mystified.

  23. Anonymous Coward
    Anonymous Coward

    What a surprise

    That's interesting ! This is why there is QA which is seen as a necessary evil and ignored, certainly at my company. But hopefully this will carry on, because it is very expensive and the developers make very few mistakes normally. This is why my company (Wirecard) is planning on going into business with Citigroup selling prepaid credit cards in the USA, some of my best people are on it. With the money laundry that we are already being accused of, the BAFIN and SEC will not know where to start !

  24. hapticz

    Hillary Clintons Boo Boooooo...

    well, this is like Madame Clintons classified email server. "I didnt intend to do that all, im sorry please forgive me!" NO, NO nO! being a simple programming oversight (lost in the tomes of ever present code revisions, tweaks, crashes and change of personell?) a nice hefty FINE of a few million bucks could placate most any USA agencys (seeing as their life blood is essentially the flow of MONEY). as for the security (national defense, financial balance, regulations and other trivial issues), the taxpayers get the general drift that making mistakes will end them in prison, while the fat cats get their fat wads of cash nibbled and shared among the others.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like