back to article How one bad algorithm cost traders $440m

Knight Capital, a firm that specialises in executing trades for retail brokers, took $440m in cash losses Wednesday due to a faulty test of new trading software. This morning reports were calling it a trading “glitch", which isn’t nearly as accurate as the term I’d use: “f**king disaster". The broad outline of the story is …

COMMENTS

This topic is closed for new posts.
  1. Sigmund Fraud

    The "bid" price is what you’d pay the holder of the stock if you want to buy their shares. The "ask" price is what they’ll pay to buy those same shares from you.

    Isn't the "ask" price the price which they are asking for the shares?

    1. Steve Renouf
      Alert

      Nah... It's the other way round.

      The bid price is what they'll bid you to buy them off you - the ask price is what they'll ask you to pay for selling them to you.

      1. Anonymous Custard Silver badge
        FAIL

        Re: Nah... It's the other way round.

        So the confusion's not just in the software :)

        Wouldn't it have made more sense to have the program test mode just do one trade, or perhaps a limited span of trade time (say 1s or 10s)? Wall things off, just so if things go wrong they aren't going wrong many times repeatedly? Or is that too obvious a first step in code-testing.

        Or for that matter have a dummy "market" that the thing could talk to in isolation, to test the bare-bones fundamentals? This is like testing a brand new car design by putting your family in it and dumping it on the motorway at rush-hour...

        Still nice to know in this day and age we can have rapid and highly efficient automation of even our very best cock-ups (to err is human, to really foul up you need a computer...)

        1. Anonymous Coward
          Anonymous Coward

          Re: Nah... It's the other way round.

          "Dummy market", as in "latex dummies" ;) Goo-goo :P

        2. Nigel 11
          Flame

          Re: Nah... It's the other way round.

          The obvious safety is to have a trading limit, if only for debugging! After it's done $(LIMIT) of trades, it stops, until a human has checked the goings-on. Once they're happy it's working they might set the limit to effective infinity.

          The other obvious limit is $(MAX_LOSS), but the danger there is that it might think it was making a profit when it wasn't (which appears to be pretty much what happened).

          Personally I think program trading is one of the casino-finance things which ought to be banned, or at least heavily restricted. Otherwise, sooner or later there's going to be another bug like this one with added naughts. In other words it won't just bankrupt a bunch of speculators, but it'll also do major damage to the real economy.

          1. Destroy All Monsters Silver badge
            Trollface

            Re: Nah... It's the other way round.

            > The "bid" price is what you’d pay the holder of the stock if you want to buy their shares.

            YES

            > The "ask" price is what they’ll pay to buy those same shares from you.

            NO

            > The bid price is what they'll bid you to buy them off you

            YES

            > the ask price is what they'll ask you to pay for selling them to you.

            YES

            > The ask is what the seller is asking, and the bid is what the buyer is bidding.

            YES

            > The ask is above the bid (article gets this backwards).

            RATIONALLY, YES

            > Buying "at market" means paying the asking price.

            YES

            > Selling "at market" means selling at the bid price (article gets this backwards too).

            YES

    2. JetSetJim Silver badge
      Happy

      Either way...

      First rule of trading: "buy high, sell low".

      Oh, wait...

    3. Eddie Edwards
      Boffin

      Yes, those two paragraphs get it a bit mixed up. The ask is what the seller is asking, and the bid is what the buyer is bidding. The ask is above the bid (article gets this backwards). Buying "at market" means paying the asking price. Selling "at market" means selling at the bid price (article gets this backwards too).

      E.g. I am ASKING £120,000 for my house. Someone is BIDDING £100,000. Either I need to drop my asking price or he needs to raise his bid, in order to make a trade. The spread is £20,000. To buy "at market" he'd offer £120,000. To sell "at market" I'd drop to £100,000.

      1. stanimir

        The ask is *always* greater or equal to the bid unless there are few sources and an option for arbitrage - here comes the microseconds decisions/orders. Market price is getting either the bid or the ask, depending which side you are at. And of course, it might be possible to place a bid or an ask on an arbitrary price in the "book".

        I am just shocked how such basic info can be so eff'd up - makes the entire article look totally amateurish.

        Source: my job

        (+1 from me too)

    4. stanimir

      Isn't the "ask" price the price which they are asking for the shares?

      it is.

  2. Anonymous John

    I've seen one report that they've asked JPMorgan Chase & Co for financing. Why anyone would want to lend them money right now, isn't clear to me.

    1. Anonymous Coward
      Anonymous Coward

      Because the CEOs probably went to school together and the loan will be covered by tax payer guarantees anyway

  3. Steven 1
    FAIL

    As no mess'in

  4. Stanislaw
    Terminator

    Our robot servants are getting more subtle

    Clearly they have decided violence is going to be too wasteful. Economics is faster.

    Shouldn't this one be filed under Rise Of The Machines?

    1. James 51

      Re: Our robot servants are getting more subtle

      Wasn't this part of the plot in the first and third batman movies?

      1. Jabber 44
        Thumb Up

        Re: Our robot servants are getting more subtle

        Saw the third one last night, I thought I was dreaming when the report of this on the radio woke me from my slumber !

    2. Dodgy Geezer Silver badge
      Mushroom

      Re: Our robot servants are getting more subtle

      XKCD currently has a 'what-if' covering the ROTM - they call it 'robot apocalypse'. See here http://what-if.xkcd.com/5/

      Randall says that he used to work in robotics, so he knows what he's talking about, and that current robots, if they turned against us, could not kill many of us - he thinks transport systems would be the best at killing humans, and not many of those would die.

      There is no comment system on his web site for me to inform him that he's completely wrong. The best thing at killing humans is 'other humans'. And machines could easily arrange for that to happen. All our food production depends on machines, as well as our living infrastructure like gas, water and electricity. They could stop that. More importantly, as indicated in the OP, all our money is held as data on machines. All they have to do is set that to zero, and we can't run our current society.

      I reckoned that without electric machines, we couldn't even go back to the steam age, because we do not have any steam engines. We'd have to go back to about 1700. The 'carrying capacity' of the world in 1700 was about 1/10 of its current population, so we'd have to kill 9/10 of us in the fight for food. That's worse than any war, and we would probably overshoot - maybe leaving us with 1/100 of our current population.

      I reckon that would be quite an effective 'Robot Armageddon'...

      1. Iain
        Go

        Re: Our robot servants are getting more subtle

        There is an xkcd forum, it used to be linked off the main page.

        Join in the fun here: http://forums.xkcd.com/viewtopic.php?f=7&t=88073

  5. Alister Silver badge

    Reminds me of something I read in a Tom Clancy novel - Debt of Honour I think it was.

    1. Anonymous Coward
      Anonymous Coward

      Oh well, at least you weren't reading a John Grisham, that's a week of part-time reading I'm never getting back.

      No-one ever tell him you were supposed to put an ending in books? I mean never mind all the technical business/law pratting around and faux-culture ;)

      1. Anonymous Coward
        Anonymous Coward

        I stopped reading Clancy when he couldn't get his European geography right.

        Getting the right name for a country's capital city is not exactly advanced research.

        1. Anonymous Coward
          Anonymous Coward

          Like Jeremy Clarkson

          "Oh I near Lake Lausanne "

          I think he meant Geneva

        2. Dave Bell

          Tom Clancy?

          Yes, he used to just believe the weapon manufacturers' specifications. Now he believes the promised delivery dates as well.

    2. 404

      Debt of Honor is was...

      .... easter egg that wiped all stock transactions in the NYSE control software- which led to war with Japan- which led to a 747 being flown into the Capital building while El Presidente was addressing both branches of Congress.

      Actually, that was the first thing I thought of @9/11 - somebody read the book.

  6. Anonymous Coward
    Anonymous Coward

    The whole point is surely that HFT is a scam intended to ensure that only companies with very deep pockets that can locate very close to the exchange servers can make money. It's equivalent to being allowed to play roulette but to place your bets even as the ball is falling into a number, and it is guaranteed to remove any element of, you know, actual investment of the "I think this iPhone thing is a really good idea, let's buy some Apple shares and see how it pans out" variety.

    The result of casino trading is that the total theoretical capitalisation of share prices exceeds the entire GNP by a large factor, which is pure bubble (just as at one time the theoretical value of Tokyo real estate was more than the entire theoretical value of the rest of the world).

    Knight Capital should be made to pay the lot, and HFT should be banned and replaced with stock trading on time bounds of, say, ten minutes. But that will prevent the traders from getting their bonuses back at our expense.

    1. Destroy All Monsters Silver badge
      Holmes

      Yeah but why should anyone care whether HFT?

      Actual problems only occur when fractional reserve banking with bailouts from the central banks enter the show. Then the donkey goes wild, virgins are raped and barbarians burn down the libraries. Which is of course the situation we are in.

      As politicians like votes and easy application of bandages to dead and napalmized bodies, and the left is firmly wedded to the fantasy of giving everyone free money in the name of egalitarianization, this ain't gonna be fixed. Instead, stupid ideas are emerging from stupid minds like taxing trades or forcing the number of trades per second below some arbitrary value. Sucks.

  7. This post has been deleted by its author

  8. Tim Worstal

    Intriguing.....

    It looks like it was also interacting with a new NYSE program as well. And possibly that it was that interaction which caused the problems, not the testing of the new algo alone.....

    1. Blane Bramble
      FAIL

      Re: Intriguing.....

      Surely the problem is it was buying shares and then immediately selling them at a lower price. Doesn't matter who or what you are trading with if you do that.

  9. Richard Wharram

    Did they just...

    ....put the buy and sell prices the wrong way round? Badly named columns perhaps :)

    1. Titus Technophobe

      Re: Did they just...

      That seems the sensible thing to me, surely the 'Ask' price should be higher in any trading entity than the 'Bid' price...... otherwise they would be the losing money hand over fist all the time?

  10. Anonymous Coward
    Anonymous Coward

    Why did it take 40 mins..

    ...for some to pull the power to the damn server!

    1. Wang N Staines
      Happy

      Re: Why did it take 40 mins..

      They went for a coffee break to pat themselves on the back for a job well done.

  11. Thomas 18
    Thumb Down

    Run for a day?

    Why would they run a test for an entire day. If it was designed to buy large quantities of stock why not just run it for 1 hour or, hell, 5 minutes first.

    1. Robert E A Harvey
      FAIL

      Re: Run for a day?

      Why run it live at all: surely it could be tested with live input data and dummy sales?

  12. Mark Allread

    What a knightmare

    1. Jemma
      FAIL

      Spellcasting...

      "d...u...m...b...f...*...*...k...s..."

      Ineptitude on a truely mindnumbing scale. Worst of all we can expect a Andrew "Apocalypse Dull" Sorkin missive on the subject soon.

  13. Crisp

    How the hell did this "feature" escape testing?

    I cannot believe that no one thought to run a simulation or a set of unit tests before unleashing this beast on an unsuspecting market.

    1. AdamWill

      Re: How the hell did this "feature" escape testing?

      They probably did. Any QA person will tell you our Rule #0, which is that everything will always go wrong. If you test a program in a test environment exhaustively for five years, testing every possible action or interaction it might ever have to undergo, and finally bash all the bugs out of it, then what you'll find out on day #1 of production is the way in which your test environment didn't _quite_ match the deployment environment.

      1. Hey Nonny Nonny Mouse

        Re: How the hell did this "feature" escape testing?

        Or more likely, that the deployment environment was changed a week before you completed testing.

  14. CheesyTheClown
    WTF?

    Gamblers deserve to lose

    When you're a gambler, part of gambling is losing and taking it like a proper loser. Get shit faced and land face flat in the street.

    I really believe that giving them a break on the trades 30% over market rate is wrong. They lost, fair and square. They tried to beat the system by getting a computer program to trade automatically and as a result, they lost their asses. And before anyone talks about the innocent victims who had invested in them... They were gamblers too and they lost.

    I would love to see someone go into a Vegas casino and lose everything they own and then ask for a break under the excuse "I didn't know how to play the game". Why should the NYSE casino be any different?

    1. Graham Bartlett
      Facepalm

      Re: Gamblers deserve to lose

      Oh, it's better than that. It's going into a casino, losing your shirt, and then asking for a break with the justification "I had a system but it didn't work".

      1. Anonymous Coward
        Anonymous Coward

        Re: Gamblers deserve to lose

        Surprisingly enough, Vegas casinos do not want you to lose your shirt, it is bad publicity. They would love you to walk in with a dollar and out with a million. The publicity is worth far more than the loss and they also know several things:

        1. You are unlikely to permanently walk away so they will get a chunk of that million back.

        2. Every news station in the country (and several worldwide) will probably publicise it.

        3. Booking for their hotel/casino will significantly increase.

        4. They will make more than that million from the increased business.

        5. The publicity will last for years.

        6. There is little to no chance of anybody else doing the same.

    2. Anonymous Coward
      Anonymous Coward

      Re: Gamblers deserve to lose

      When you are a gambler you play within the rules of the game, otherwise it is called cheating.

      If the rules of the game say they get refunded for trades 30% over market rate then it is right that they get refunded.

      If the rules of the game say they can use a computer to trade automatically then they can use computers to trade automatically.

    3. Vladimir Plouzhnikov

      Re: Gamblers deserve to lose

      Be careful what you wish for.

      Any investment is a gamble, whether you invest for a microsecond or for 30 years. Your pension fund is based on investing your contributions into various assets (mostly stocks). You blanket statement "gamblers deserve to lose" then translates as "you deserve to end up without a pension". This applies to any other investments you may make in your life as well.

      1. Nigel 11
        FAIL

        Re: Gamblers deserve to lose

        Wrong.

        Gambling is investing when the expectation value for the profit is negative. Bet in a casino, and averaged over enough bets, you are losing 1/37 of your capital for each bet (in a UK casino with one zero on the wheel).

        Investing is when the expectation value for the profit is positive. You might imagine an eccentric millionaire who returned double your stake on all zeros, so you could predict that you'd be up by 1/37 for every spin of the wheel. Buy shares in a big blue-chip company, and that return is called a dividend.

        Of course, investing in companies is to some extent a matter of judgement rather than probabilistic certainty. You have to be right often enough if you're going to make a profit. A few people can even judge horses right, and for them a betting slip is an investment not a gamble. They're rare. Successful stock-market investmenting is easier, because an honestly run company is trying to make money for its shareholders, not lose them money.. (You might guess that I do not invest in banks. I've long believed banks run on the principle of "heads I win, tails you lose". )

        There's also a social benefit to investing even when it makes a loss. You're lending your money to a company that hopefully is in a business that you believe ought to be carried on. It's often said this applies only to those who buy new equity issues. Wrong. The initial investors probably wouldn't invest, if there was no way ever to sell up. A company is supposed to last longer than the few decades between having earned surplus capital, and wanting to spend it on one's retirement. The stockmarket is a mechanism that makes people more willing to invest their capital in the first place.

        It becomes less clear when instead of doing one's own investing, one delegates it to a pension fund manager or suchlike. That's because your delegate would like to put as much of your money into his pockets as he can get away with. That, however, has nothing to do with whether it's investing or gambling.

        1. Dave Bell

          Re: Gamblers deserve to lose

          There are all sorts of interesting things that come out in the math, such as the way that the Casino can win even in an absolutely fair game--they have more money and can afford a longer run of losses. And the still famous "man who broke the bank at Monte Carlo" only "broke" that particular table. They are allocated so much money and no more.

          If that program had been in casino, it would have had a hard limit on how much it could lose.

          So I've no great problem with the system having some limits built in to catch extreme conditions. Intended or not, that program ended up manipulating the market. But the speed with which everything happened: not for the first time, sets me to asking how economic theory can have any meaning for events that are too fast for a human, even the simplified rational human of theory, to comprehend. When the system is dealing faster than a human's physical reaction time, never mind the time needed to think.

          I can remember when my money took weeks to travel from my bank to a US magazine publisher, by some arcane special procedure. I can remember a man who regularly travelled to buy goods in the USA, at auctions, and carried cash because the banking system couldn't keep up with him. Now getting the money around the world can be almost instant.

          My acquaintance can still be standing at a different auction every day, bidding against people he can see. That market hasn't changed. They money just gets delivered more easily But the stock markets have become inhuman virtual spaces. Can you look at the other computer and see the giveaway twitch of an eye that tells you your next bid could win?

          (I got the chance to read some of the latest work from Charlie Stross, a book built around interstellar trade, where the distinction between fast and slow money is hugely significant. And the hero is an accountant. Should be in print next summer--book publishing is still about slow money.)

        2. Vladimir Plouzhnikov

          @AdamWill and Nigel11

          Oh, no, guys, please don't deceive yourself.

          Whether you invest on a whim or after a careful research and deliberation, whether in stocks, oil, gold or land - any of that is taking a risk and that is what a gamble (aka speculation) is.

          Google for a definition of gamble and the applicable result is "an enterprise undertaken or attempted with a risk of loss and a chance of profit or success."

          In general, any position in the market that increases your risk (exposes you to the market fluctuation of value) is a speculative position and any position that decreases that risk is a hedging position.

          There is nothing wrong with speculation/gamble if it's done properly and in a controlled way and it is impossible to make a return without taking risk, so gamble is an inevitable part of our lives. And if someone thinks that he never gambles/speculates he should think twice because even if he doesn't do it directly, someone else does it for him. Which is what I said in my first post.

      2. AdamWill

        Re: Gamblers deserve to lose

        "Your pension fund is based on investing your contributions into various assets (mostly stocks). "

        That's an assumption too far. Many people have control over the investment of their pension fund. I do; I could put the whole thing in gold or government bonds if I wanted to.

  15. Why Not?
    Facepalm

    Nick Leeson Software inc.

    You would have thought part of the Algorithm would be to set boundaries of maximum trades?

    I assumed that all stock exchanges had a test area where such things could be verified prior to going live?

    1. Blane Bramble
      WTF?

      Re: Nick Leeson Software inc.

      At the very least you'd think they would set a maximum loss setting for the test. I would assume the point of the algorithm/system is to make money, so sensibly that should be 0 - trading goes negative, stop and report it.

      1. Why Not?

        Re: Nick Leeson Software inc.

        More complicated as I understand it.

        I think its a bit like fixing the betting on a big fight, you systematically bet on the challenger until the odds lengthen on the Champion then you bet the farm on him.

        Losing is part of business, how much you lose is the key thing.

  16. Quinch

    So a flubbed program cost a company botting stock exchanges a few hundred million dollars?

    Please observe as I shed a single salty tear.

  17. Jim Wormold

    Doesn't it work both ways?

    If the company has lost $440M, then haven't others made that money?

    Isn't this how the markets work? Why does this company get a rebate because theit software was faulty and made legitimate but "silly" trades? Isn't that the risk - that your algorithms don't always perform perfectly?

    1. Blane Bramble

      Re: Doesn't it work both ways?

      You seem to be misunderstanding modern trading. The point is to pay yourself huge amounts of money for a high-pressure, high-risk job, and then ensure that you stack the system so that someone else (preferably a tax payer) takes that risk. Trebles and Bollinger all round!

    2. TeeCee Gold badge
      Happy

      Re: Doesn't it work both ways?

      If the company has lost $440M, then haven't others made that money?

      Yes, that's exactly how it works. This is not the equivalent of piling $440m on the floor and setting fire to it, it's the equivalent of opening the office windows and chucking $440m out into the street.

      1. Danny 14

        Re: Doesn't it work both ways?

        Im sure the new startup company with its 15p share that traded 2 million times in half an hour got a huge smack of cash.

    3. Cyclist

      Re: Doesn't it work both ways?

      $440m has gone to <many> recipients from one supplier.It's all still there as dollars but more thinly spread around.

      Not sure what help that is, it's still gone from where it was to where they didn't want it to be.

    4. Anonymous Coward
      Anonymous Coward

      Re: Doesn't it work both ways?

      "If the company has lost $440M, then haven't others made that money?"

      While the trading program was buying and selling the same share 40 times a second another one was selling and buying 40 times a second. Millions of profit from owning one share. Drinks are on me.

  18. Peter Clarke 1
    Coat

    Sounds Familiar ...

    Looks like the IT guy from RBS soon got a new job then.

    1. Alan(UK)
      Joke

      Re: Sounds Familiar ...

      I had put my coffee down but I still nearly chocked when I read your post.

  19. Anonymous Coward
    Anonymous Coward

    40 Times Per Second?

    Is it possible it was also signing up for outlook.com email accounts too?

  20. Matt Bryant Silver badge
    Happy

    Wow!

    My most expensive coding error was once writing a program that left an ISDN line to New York up for a week back in the days when that meant a phone bill running into four digits! I don't feel so bad about it now. Anyone know if they wrote the bot inhouse or used a third party, just so I know which "geniuses" to avoid hiring?

    1. John Smith 19 Gold badge
      Happy

      Re: Wow!

      "My most expensive coding error was once writing a program that left an ISDN line to New York up for a week back in the days when that meant a phone bill running into four digits!"

      Yes that is impressively stupid.

      1. Kubla Cant Silver badge
        Unhappy

        Re: Wow!

        It's easier than you think. There used to be ISDN modem/routers that "spoofed" the connection - the router made it look like the connection was up, but only dialed when there was traffic. The trouble is, network protocols like NetBEUI used to like to chat to every other node on the network every few milliseconds.

        Never mind New York, I've seen a four-figure bill between London and Bristol as a result.

        1. Nigel 11
          Unhappy

          Re: Wow!

          Reminds me of a plain ordinary modem, where the connection to the telephone socket wasn't sufficiently secure. One day we got a four-digit phone bill. It turmed out a cleaner had been unplugging the modem, plugging in a telephone, and having long conversations with somebody in the West Indies.

        2. JimC
          Boffin

          Re: Wow!

          Just leaving the line up for a week is only a half hearted effort.

          I don't know if its true, but I heard a story many years ago of a link over a satellite connection where there was a minimum call charge equivalent to (IIRC) three minutes connection. Our hero had an inactivity timeout of 30 seconds. He also had a snmp agent on the far side of the link that reported when the line dropped.

          So every 30 seconds the line dropped, the SNMP agent sent out a trap, and the line went up to send the trap. So that was a three minute cost call every 30 seconds - or six times the cost of just leaving the line up continuously.

          But as I say its probably a tenth hand story, and may have grown in the telling. But if you used to do this stuff its perilously credible...

  21. Jason Bloomberg Silver badge
    Thumb Up

    New Word Record

    at the end of the debacle 45 minutes later, Knight Capital had lost $440m

    So almost $10m per minute, $1m every six seconds, $160K per second, all flushed down the drain.

    That's quite impressive. You'd have thought there might be some sort of 'how we're doing' display which might have indicated the company were spiralling into disaster, some sort of loss monitor that might abort the program pretty damned sharpish.

    1. Anonymous Coward
      Anonymous Coward

      Re: New Word Record

      You'd have thought there might be some sort of 'how we're doing' display -

      there is, but it doesn't show small change

      1. Danny 14

        Re: New Word Record

        not flushed down the drain. Other companies share prices will have done well from their insane spending spree.

  22. The Axe

    3

    "ISV software bugs, bad documentation, and human error from Knight Capital"

    Trouble comes in threes.

  23. Still Water
    FAIL

    Hang on...

    ...they were testing software on a system that was capable of putting through live trades? This is what test environments are for. They only have themselves to blame.

    If this got through a test environment and passed, then really, they need to sack their entire QA team and start again...

    1. Anonymous Coward
      Anonymous Coward

      Re: Hang on...

      "If this got through a test environment and passed, then really, they need to sack their entire QA team and start again..."

      They would sack the QA team, but the QA team are all on holiday on their luxury yacht in Monaco at the moment...

    2. Lars Silver badge
      Happy

      Re: Hang on...

      Hang on Sill Water, if you believe that testing will always reveal every error then you just have no experience.

      I remember a program I wrote long ago, it worked for about three months but then started to tilt every now and then. In the beginning the users, four ladies, just booted the machine, and started again. Eventually it did it more and more often and I had to look into it.

      The error was easy to find and easy to mend and the reason was that one of the girls developed an incredible speed with her five finger numeric input.

      I could never have found that error, nor anybody working in our company by testing it.

      (A faster processor would not have revealed that error ever.)

      I have corrected programs that worked OK for many years until then suddenly there was a problem.

      The testing surrounding is never exactly the same as real life and the input, no matter how hard you try, is never as complete as in real life.

      Still I think there should have been some means to prevent it from going on like it did.

      Also I am sure any additional programming and testing that perhaps the programing department might have asked for, might have been rejected by those who decide, because of costs and time tables.

      Much bigger losses of money have occurred because of programing errors. And they will always remain no matter what we do.

  24. Jess--

    it was reversed buy and sell (as has been suggested) then the software would have thought it was 440 million in profit,

    every time it bought and then sold at a 15 cent loss per share it would see it as a 15 cent profit per share.

    if you had just built a system that you were sure was going to make you a lot of money would you turn it off when it said it was making you a lot of money?

    might explain why it took them 45 minutes to shut it down

    1. Jess--

      just realised I missed the word "if" from the beginning of that last post

      it should read "If it was reversed buy and sell"

  25. amanfromMars 1 Silver badge

    Look on the bright side ....

    "They may get at least a partial reprieve. The NYSE will reverse trades in six stocks during the time period when the prices were at least 30 per cent outside the normal trading range for the stocks. This will significantly defray much of Knight Capital’s losses for the day, but we don’t know if it’s enough to allow the firm to survive the blow."

    Thus proving once again that it is a rigged, [mistakes and error disallowed] and not free market trading platform.

    1. AdamWill

      Re: Look on the bright side ....

      Well, 'rigged' is a bit of a harsh term. Most major stock exchanges are, to some degree, regulated, based on years of experience that indicate that things which are broadly detrimental to the stock exchange and the economy as a whole can result without regulation. All sensible involved players agree on this - the stock exchanges, the traders, the traded companies, and the government regulators. It's not really controversial.

      It only becomes a tad embarrassing for the people who _will_ insist on waving Spherical Cow World free market economic theory around as if it applied without modification to the real world.

  26. DJDUBLO
    Trollface

    Robert Harris?

    Anyone else read "The Fear Index"? Sounds like someone tried to build their own "Vixal-4" but got it a bit wrong....

  27. Anonymous Coward
    Anonymous Coward

    Good, loose more fcukers.

    Maybe they should all do that until only one person has all the money, then we can shoot him and get the taxpayers money back as death duties.

  28. Wombling_Free

    Hang on, money can't just vanish...

    Yeah, some morons lost $440mil.

    But doesn't that mean that someone, somewhere, MADE $440mil? Does the NYSE reverse trades only when their buddies are in trouble, or will they bail out anyone for making a dud trade?

    For every tale of one smarmed suit making a mint on the stocks, there are a thousand stories suckers that lost it all.

  29. mark 63 Silver badge
    Terminator

    greed greed greed

    i cannot express what a waste of fucking time the stock market is.

    people say "oh it creates jobs , it stimulates the economy , it prevents inflation , it sets prices - you're not seeing the bigger picture"

    Bullshit they need to step back further to see the picture.

    The fact that this is all automated now makes it even more pointless.

    When bots are used in FPS games online its called cheating.

    When bots are used in gambling online its called cheating.

    on the stock market its the only way to keep up.

    if you've removed the human element then what does the whole thing signify?

    it really is the beginning of the "Rise of the machines"

    1. Vladimir Plouzhnikov

      Re: greed greed greed

      "on the stock market its the only way to keep up."

      How is that so?

    2. Charles 9 Silver badge

      Re: greed greed greed

      "When bots are used in FPS games online its called cheating."

      Not all bots in games are bad. If they're used as player substitutes, they can help add elements to the game, though their results can be rather mixed. Still, in some instances bot players (recognized as such) are welcomed.

      Now, in terms of aimbots, speedhacks, and so on, yes, THAT's cheating.

    3. AdamWill

      Re: greed greed greed

      "if you've removed the human element then what does the whole thing signify?"

      it's...how companies are funded.

      You start from the principle that people can join together to found companies in which each person owns a fractional stake - the 'joint stock company' of history, dating back several centuries - then refine it for a few hundred years, and you get a stock market. What is your alternative to a stock market? How are companies to be owned?

      1. mrfill
        Alert

        Re: greed greed greed

        Not quite... it is how companies are INITIALLY funded. The company initially owns the share and sells it to X and pockets the money. When X sells that share to Y, the company makes not a bean and every other transaction after the initial sale creates zero for the company.

        Once you sell your company to the market, you cease to own it so do not profit from sales. However, you are still liable to pay any dividend declared by the board who are appointed by the company owners i.e. the shareholders.

        1. Danny 14

          Re: greed greed greed

          if you do well though then the price goes up. Once past a point you stock split and get more cash.

  30. John Smith 19 Gold badge
    Stop

    It's quite possible to *not* set up a test environment.

    The original architect/developers of the systems *assume* that is this a bespoke one off system for internal use in 1 company there is no *need* to set up a the option to set up multiple "company" identifiers or to copy chunks off the live system (to allow quick population of the internal files for a test environment).

    And you know what happens when you assume.

    I speak from experience on this area of development and the pucker factor is *much* higher when the first test is on live data on a fully live system with all users having access than when the 1st test is in a test environment whose key elements you've coded yourself.

    That said none of the companies were in the FS industry, where I would have thought a detailed *instrumented* test environment (IE damm near everything logged so after a test run you can see *exactly* what the software was doing and to what data) was SOP, not a nice-to-have option.

  31. MrT

    Ah, Mr Leeson...

    ...thank you for your application as software developer. Have you worked in banking before...?

    1. NellyD

      Re: Ah, Mr Leeson...

      Why, yes! I have Mr Apotheker.

  32. Simon R. Bone
    FAIL

    was...

    ... it buying Facebook shares?

  33. Arthur 1
    WTF?

    <--- see icon

    Why wasn't it trading in a sandbox? Who does what sounds like very preliminary tests in a live environment?

    1. Anonymous Coward
      Anonymous Coward

      Re: <--- see icon

      Cause sandboxes are for kids, real men test live.

  34. Fuzz

    crop report

    sounds like someone tampering with the crop report to me

    1. nsld
      Coat

      Re: crop report

      Was it Beaks?

      1. MrT

        Re: crop report

        Beef jerky time...

  35. Tom 13

    Re: "...but we don’t know if it’s enough to allow the firm to survive the blow."

    'scuse me! Crazy right-wing 'Merkin here and I got what seems like a really simple question:

    WHY would we want this company to survive? What they did was bat-shit stupid, they should go bankrupt in payouts for their mistakes.

    1. Anonymous Coward
      Anonymous Coward

      Re: "...but we don’t know if it’s enough to allow the firm to survive the blow."

      Bankrupt? Please don't be naive, a simple phone call fixes everything "Hi mom, hi dad, more trustfund plz"

    2. John Smith 19 Gold badge

      Re: "...but we don’t know if it’s enough to allow the firm to survive the blow."

      As someone once observed "Capitalism without bankruptcy is like religion without Hell."

      However....

      It's the Financial Services industry.

      It cannot, cannot be *allowed* to fail because well civilization would fail, panic in the streets, dogs and cats living together outside of wedlock etc.

  36. F'tangF'tang

    Computer says Buy

    The electronic order book trading system (in UK its SETS SEAQ etc...) Just takes market orders for buys and sells from the market makers (or market manipulators as they are sometimes affectionately known), or any joe public with a trading account who wants to be put "on the book" to trade in a particular stock.

    All the software does, is match buyers and sellers. It makes a "touch" price (the best price offered to either buy or sell) of a stock based on the orders on the book, listing hierarchical best prices offered by the sellers one side, and the buyers on the other.(Bulls buy, bears sell... ish). Identical prices are ranked by the time the order is submitted to the market. Usually, you would never want to buy at market price, as 90% of the time, when you get a live quote to either buy or sell, you will get an improvement on the market price (by a large % with some illiquid AIM type stocks)... getting inside the spread as it is known...

    You will usually see market makers on both sides of the book, being both a buyer, and a seller...Obviously at predetermined levels, so as not to electronically trade with themselves...

    You can see it all happen real time with Level 2 data, with the columns of all the buyers and sellers on the book and the prices being asked, and the trades getting executed and removed from the order book as they are completed... as opposed to Level 1 which just lists the highest Bid and the lowest Offer price with no visibility of the orders below the touch price (market depth)...

    All good fun, but sadly not far different to going to a casino nowadays...

  37. This post has been deleted by its author

  38. chris lively
    Facepalm

    They probably shouldnt have off shored some of the development to English challenged countries

  39. Anonymous Coward
    Anonymous Coward

    How to test such software?

    Wouldn't it mean writing a dummy market?

    Might that not be an even bigger project than the trading software itself?

    What is the professional/usual way to handle this?

    1. Anonymous Coward
      Anonymous Coward

      Re: How to test such software?

      Usually yes, but not necessary. Since it was a test though, they should have set it to run 1000 transactions, or for one second, as other posters already noted.

      Anyway, let's think what would have happened if the software did not had that error. Would someone have complained if Knight Capital made a lot of money? Would those trades be reversed just because they were ran by a test system? Somehow I don't think so.

  40. This post has been deleted by its author

  41. Anonymous Coward
    Anonymous Coward

    What Cantor’s e-Speed taught me about the frailties of Algo Trading (HFT)!

    Speaking from experience here's three more reasons why Algo Trading / HFT's mess up. I was working at a large bank for US interest rate traders once. As an example, the traders wanted a mechanism to move large blocks of 30 year bonds without moving the market. So the goal was to design a black box (BB) to move small parcels throughout the day, trading on defined limits of the market at that time. The latter phrase proving crucial here!

    The problem is: whenever its a busy morning the BB is reliant on timely receipt of MESSAGES and the timely order of those messages from the exchange’s trading systems, which is itself a kind of BB. And all of this withstanding the usual glitches like network latency, bottlenecks, outages etc What I was discovered was :-

    #1. You can't recreate a live market in Beta. Why not? A. Because the beta system is not anything near as liquid, B. the beta is often not even running the same Exchange software version, and C. your black box is playing with other robots-- and not humans and robots and real-world actualities.

    #2. There are significant subtleties in the way a busy morning can affect the ORDER in which your black box receives messages into its queue! I found cantor's e-Speed system sent messages out of order making it tricky to pair up last trade prices with current market pricing. It meant trade confirms with the actual SIZE moved were delayed well beyond where the market was now. In short, my BB frequently found itself in ill-defined state.. What to do next.…? Where was the market now? How much size was actually executed? What should the next BID / ASK posting be?

    #3. The exchange provider only shares some technical subtleties with its clients, that is-- unless your are one of its darlings that generate the most fees: hedge funds and Goldman, Deutsche et al... What this means is that you are often operating in the dark. In addition the Exchanges systems can be the actual cause of the glitch. Without warning Cantor re-numerated trader operator ID’s overnight. Anyone who hadn’t re-logged on was unwittingly now using the wrong trader ID. Boy, that was messy! Traders getting another traders confirms, trades executing in the wrong books etc.

  42. h4rm0ny

    Somewhere, someone, is feeling awful right now.

    They probably think, if they'd just set some config variable or added some check, and that it's all their fault. I wonder how they will choose to respond? Will they tear themselves apart, will they blame others (rightly or wrongly), will they shrug and say that it's just money? I wonder.

  43. DoobyUK
    FAIL

    Regardless of whether it was unit tested or not, did no-one think it *might* be a good idea on its first run to not let it actually trade, but rather just output a feed of what it thought it should trade - you know, so someone could check it? or even have an 'authorisation' step while the thing was on its first run.

    Regarding the bid/ask spread thing, I do wonder if they had a 'stop loss' set too close and not taking account of the spread, so you buy at the ask with a 10% stop loss, but the spread is 20% so it immediately sells (big %, just examples)

  44. Anonymous Coward
    Anonymous Coward

    delayed pertrubed feeds

    We dont do fast trades but we do test using delayed data feeds.

    We are small enough that we dont need to perterb the data feeds to simulate our effects but even this is possible as this sort of modelling is relatively easy when you use a delayed data feed - the test feed generator knows "in advance" how the maket actually responded and you can apply a reasonable simulation fi what would happen if your (simulated large trades) affected the markets.

    IMHO Running the same datafeed multiple times with various "bounds" scenarios should also be part of the test plan.

    The problem is the larger the "player", the more expensive and less predictable testing becomes.

    Being small can be an advantage :-)

    Also the large players tend to be risking other peoples money - not their own - so the risk of live tests can be ignored.

  45. despairing citizen
    FAIL

    Amatuer testing times

    Where systems have to be tested with live feeds, you isolate the system in such a way that the usual output is blocked, (not exactly rocket science)

    Was doing this back in 1998 to test what happened to large financial systems (mutli-Bn) when they thought they where working in 2015, 2025, 2035, which meant we where adjusting the inputs as well.

    Would suggest that the various stock exchanges set down minimum standards for testing, and revoke the connections of any outfit that does not meet them, much the same way as PCI DSS does security standards.

  46. Destroy All Monsters Silver badge
    Devil

    So...

    That means the program was burning money 171 TIMES FASTER THAN THE AFGHAN WAR (according to 2009 numbers)

  47. Destroy All Monsters Silver badge
    Devil

    Reminds me of this gem, which you may be able to find on the Internets for free:

    "Social Impact of Information System Failures"

    IEEE Computer, June 2009 (Vol 42. Issue 6, pages 58-65); Tamai, T.

    "The case of Mizuho Securities versus the Tokyo Stock Exchange (TSE) is archived in the 12 December 2005 issue of the Risks Digest (http://catless.ncl.ac.uk/risks/24.12.html), and additional information can be obtained from sources such as the Times (www.timesonline.co.uk/tol/news/world/asia/article755598.ece) and the New York Times (www.nytimes.com/2005/12/13/business/worldbusiness/13glitch.html?_r=1), among others.

    The incident started with the mistyping of an order to sell a share of J-Com, a start-up recruiting company, on the day its shares were first offered to the public. An employee at Mizuho Securities, intending to sell one share at 610,000 yen, mistakenly typed an order to sell 610,000 shares at 1 yen.

    What happened after that was beyond imagination. The order went through and was accepted by the Tokyo Stock Exchange Order System. Mizuho noticed the blunder and tried to withdraw the order, but the cancel command failed repeatedly. Thus, it was obliged to start buying back the shares itself to cut the loss. In the end, Mizuho’s total loss amounted to 40 billion yen ($225 million). Four days later, TSE called a news conference and admitted that the cancel command issued by Mizuho failed because of a program error in the TSE system. Mizuho demanded compensation for the loss, but TSE refused. Then, Mizuho sued TSE for damages."

    ....We then segue into design horror....

    "The part of the system that handles order cancellation appears to have low modularity. The logic in part B of the flowchart made a wrong judgment because the information telling it that the target order had induced the reverse special quote had been temporarily written on the Stock Brand DB by the order matching module and had already been cleared. This implies an accidental module coupling between the order matching and order cancelling modules. The order cancellation module appears to have insufficient cohesion as different functions are overloaded. It is not clear how the tasks of searching the target order to be canceled, determining cancellability, executing cancellation, and updating the database are this module’s responsibility."

    etc. etc.

  48. Andus McCoatover
    Joke

    I'm buying shares in Knight Capital.

    As long as I can use their software to trade them.

    Steal underpants.

    ??????????

    Profit!!!

  49. Henry Wertz 1 Gold badge
    Thumb Up

    HFT traders are parasites

    "Yeah but why should anyone care whether HFT?"

    Because malfunctions can collapse stock prices. Because HFT lets certain "favored" traders be a parasite on everybody else. These HFT systems can actually see in-progress trades and force themselves ahead of those traders; you find someone willing to sell you a stock at $10, and a willing buyer at $10.05, you'll find once you put your buy in at $10 that the stock is now $10.04. Does the HFT trader deserve that 4 cents more than you? Hell no! But they get it.

    I'm just glad the stock exchange has finally gotten some balls and said they are not cancelling these trades and it is non-negotiable. These HFTs (not King's specifically...) have malfunctioned at least half a dozen times before, and before -- although the stock exchange was perfectly happy to let HFT traders make hundreds of millions parasitically, if the HFT software malfunctioned and LOST hundreds of millions (in other words, letting others get some of the money back the HFT took), the stock exchange would CANCEL THE TRADES!

  50. Big-nosed Pengie
    Facepalm

    $440M?

    So about .00001 second's worth of trades?

  51. perlcat
    Facepalm

    life imitates art:

    Michael Bolton: Okay! I must have put a decimal point in the wrong place or something. Shit! I always do that. I always mess up some mundane detail.

  52. Paul 129
    Mushroom

    What if they hadn't turned it off?

    First off I'm talking with no real knowledge of how the system works.... But isn't the problem a little more than one player going belly up?

    How can the exchange protect everyone if one of these things goes completely rougue?

    Only trades up till the company is bankrupt is valid? How could they unwind thousands of trades past that point? Could they even tell where the point is? The follow on effects could be colosal. More to the point you could get to the situation where the tangle is too big to fail, a huge injection of cash is required to unwind the system enough that its business as usual.

    Do you know of any big financial institutions who would play that, minimized loss scenario? It's not as if any past players have such a track record now, is it?

  53. phil harris

    Ha Ha Ha !

    Traders getting shafted by software :)

    I think it was a deliberate bug put in by a developer who's money is worth nothing in the bank anymore, and who never gets bonuses ;)

  54. daveeff
    FAIL

    ask / buy

    As there seems so much confusion on which is ask / buy maybe they got backwards and then tested it on a dummy system that also had it backwards.

    Testing is only as good as the spec (and usually a lot worse!).

  55. Sceptic Tank
    FAIL

    Should have...

    They should have used a quality random number generator to make their picks - the way most financial "advisers" do.

  56. gb030104
    Joke

    On a plus side

    The executives awarded them selves a huge bonus for coping in a difficult time

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2020