back to article Having bank problems? I feel bad for you son: I've got 25 million problems, but a bulk upload ain't one

Sunday is gone and Monday is here. To ring in the week, please join us in welcoming the latest addition to the shedload of shame that is The Register's Who, Me? column. Today's toe-curler comes from a reader we shall refer to as "Geoff" who, as a newly qualified COBOL and Assembly-language programmer, was working for what he …

  1. Korev Silver badge
    Joke

    Looking at the continuous drip of banking computer systems **** ups in Britain; I wonder if "Geoff" has now moved to the UK....

    1. Chloe Cresswell

      Naw, he admitted it. You wouldn't get that in the current climate.. ;)

      1. Velv
        Facepalm

        Nope, definitely wouldn't get that in the current climate.

        Wouldn't get any of the major UK banks using live data for testing, no Sir.

        And definitely wouldn't get any of the major UK banks hosting their test systems on their Production network without isolation from Production. No Sir, wouldn't happen.

        #sarcasm

        1. katrinab Silver badge
          Joke

          UK banks doing testing or having test systems. Not going to happen.

    2. katrinab Silver badge
      Flame

      Unlikely.

      They just go live without any sort of testing, see for example TSB.

  2. Blockchain commentard

    Bit like me in the mid-80's. First time on night shift, get told to run the End Of Day processing by taking 'Option 1' - the pre EOD backup. Unfortunately the terminal I used (conveniently already logged on), had a different Option 1. Their's was to purge that days transactions. Whoops. And management still made us run the EOD without any transactions (cos they're stupid). The following day, the input girls had to enter the transactions again, we ran the EOD again (6 hours), then that days transactions were entered (overtime here we come) and that days EOD was run. Funnily enough, training was provided after that and procedures implemented !!!

  3. LDS Silver badge
    Coat

    Did the CCM run a timer?

    Quite precise guys, there...

    1. caffeine addict

      10 minutes, not a second more...

      I was once doing some electrical work in my grandparents house when my grandmother came in and told me the hoover wasn't working. I explained that it was because I'd turned the power off, and it would be off for another ten minutes.

      Ten minutes later I (and the opinel knife I was using) were flung across the room. Moments later, my grandmother appeared to tell me the hoover had stopped working again.

      I kept that knife for years. The brown streak of copper wire that had plated itself next to the missing chunk of steel seemed an apt reminder not to give people short timescales...

      1. Doctor Syntax Silver badge

        Re: 10 minutes, not a second more...

        I liked the days when we had real fuses, not resettable circuit breakers. You could take a fuse out and keep it in your pocket to prevent that sort of thing.

        1. LDS Silver badge

          Re: 10 minutes, not a second more...

          My circuit breakers are in a closet I can remove the key from and keep in my pocket.... :-)

          1. Anonymous Custard Silver badge
            Headmaster

            Re: 10 minutes, not a second more...

            Indeed - LOTO (lock-out, tag-out) is your friend...

            1. caffeine addict

              Re: 10 minutes, not a second more...

              Much as I'd have loved to have locked my grandmother out, she knew which neighbours had the spare keys...

              1. caffeine addict

                Re: 10 minutes, not a second more...

                Oh... you meant from the fuse box...

            2. Krassi

              Re: 10 minutes, not a second more...

              LOTO indeed - for on-site engineering it is standard practice since whenever, and very rigorously enforced at good sites: you don't call up your mate in the control room to cut the power, you go in person, isolate, put your padlock on, keep the key. When you've finished, roll call, back to the control room and unlock. Some places everyone in the team has an individual lock & has to be there in person to lock and unlock.

              A bit of crypto wizardry & the IT equivalent should be trivial. But we'd rather a quick bodge job.

              1. Michael Wojcik Silver badge

                Re: 10 minutes, not a second more...

                I don't know about trivial - the devil's in the implementation - but achievable, certainly. You'd probably want a N-of-M quorum scheme or some kind of key escrow in case the locker forgot his passphrase or was struck by a meteorite on lunch break or something.

                1. Grooke
                  Joke

                  Re: 10 minutes, not a second more...

                  "This implementation has an exploitable timing attack vulnerability that I can use to break the encryption and electrocute my colleague."

            3. John Doe 12

              Re: 10 minutes, not a second more...

              "Indeed - LOTO (lock-out, tag-out) is your friend..."

              Don't be so sure!! When I was just starting out as an electrical apprentice in the 1990's I went to the stores with another newbie to get two padlocks (one for each of us) so we could work with the 1500v overhead line safely using the locking out system. I tried my key in my colleagues padlock and it successfully opened!! So much for THAT :-D

              1. Rich 11 Silver badge

                Re: 10 minutes, not a second more...

                I tried my key in my colleagues padlock and it successfully opened!!

                I hope that supplier didn't also bid for the locks in the nuclear missile silos.

                1. Korev Silver badge
              2. NorthIowan

                Re: I tried my key in my colleagues padlock and it successfully opened!!

                Not surprised. I started in production mainframe testing a very long time ago. So we all had our own toolboxes with a padlock. It was a new production site so all the toolboxes and padlock were bought at the same time. Turned out most people could unlock most other peoples padlocks. There were a lucky few that had padlocks that were safe from most of the other people.

                Oh and then there are US built RVs (campers). Turns out 60% of the outside storage compartment locks are keyed to "CH751" so all the keys and locks are the same. One of my locks just failed so I will probably switch them all. None are storage compartments but still, it's the principle.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: I tried my key in my colleagues padlock and it successfully opened!!

                  CH751 also happens to be the key used by a certain 3 letter ATM manufacturer to lock numerous ATM's rear access console (note, not the safe - just the rear console) - but still :-)

                  I remember the first time I realized that my mailbox key could be used to open the ATM console...

              3. Ken Moorhouse Silver badge

                Re: I tried my key in my colleagues padlock and it successfully opened!!

                There's production padlocks... and there's test padlocks... (That's not meant to be taken seriously BTW).

                ===

                Many years' ago I devised a system to automate the registration of codes that are used in a safety-lock system for a well-known client in that field. Certain codes could not be used as it would weaken the structure of the lock. Pythagoras came in handy for this purpose, and so I can say that the maths I did at school was not in vain (belated thank you to Messrs Hogarth and Kelly).

            4. Michael Wojcik Silver badge

              Re: 10 minutes, not a second more...

              Lock-out breakers are required for at least some appliance circuits in the US, if there isn't a cut-off switch that's within view of the appliance. At least that used to be the NLB requirement - it's probably changed in the years since I last looked at it.

              Basically, the idea was if the installer / maintenance person couldn't see that no one had turned the power back on, he (or she) needed assurance that it wouldn't be. And that's with little ol' 110V US circuits. With higher voltages I'd want lock-out regardless.

              1. Kiwi

                Re: 10 minutes, not a second more...

                Basically, the idea was if the installer / maintenance person couldn't see that no one had turned the power back on, he (or she) needed assurance that it wouldn't be. And that's with little ol' 110V US circuits. With higher voltages I'd want lock-out regardless.

                A cheap hasp and lock on your fuse box door? Easy to break in an emergency, but family should know not to damage it unless it's a life/death emergency.

        2. Anonymous Coward
          Anonymous Coward

          Re: 10 minutes, not a second more...

          These days, you can get locks to put on the breaker and/or isolator.

          Handcuffs also help...

          1. Yet Another Anonymous coward Silver badge

            Re: 10 minutes, not a second more...

            >These days, you can get locks to put on the breaker and/or isolator.

            Spent all night hunting a vacuum leak and left the pumps running to pump down.

            Carefully locked out the telescope with the official lock out hasp and padlock, took the key, wrote it up in the engineering log and went to bed.

            The day shift had to override the lockouts to move the telescope because "they couldn't find me"

            Arrived to a vacuum pump dangling 20m above the ground attached to an instrument that was 3 years of work for a whole team.

            Idiot proof is easy, bloody stupid proof is harder.

            1. Nunyabiznes

              Re: 10 minutes, not a second more...

              Another Murphy-ism:

              You can't make anything foolproof because fools are so ingenious.

        3. Annihilator Silver badge

          Re: 10 minutes, not a second more...

          A stern post-it note over the switch helps too. But depends on who finds the note, and whether they like you or not.

          Interestingly, there are locks you can get for them. Google "MCB toggle lock". I agree, it's nowhere near as handy as pocketing the fuse, but safer in that someone can't put a spare fuse in. Or a nail.

          1. 0laf Silver badge
            Alert

            Re: 10 minutes, not a second more...

            Yep that situation sounds quite familiar. I'm sure electricity can't be as dangerous as it's made out, I know a large number of people who have had big shocks and waked it off.

            I've had a few close encounters and a mate nearly threw himself down the stairs when he was working on a lighton the landing and trusted his (now wife) gf to turn the power off.

            1. Arthur the cat Silver badge
              Boffin

              Re: 10 minutes, not a second more...

              I'm sure electricity can't be as dangerous as it's made out, I know a large number of people who have had big shocks and waked it off.

              When I was a kid my uncle had a shop selling TVs and radios and offering repair services. One of the repair men was known for his trick of stripping the insulation off wires with his teeth, until the inevitable day came when he failed to spot the other end was plugged into the mains. He came to about 10 minutes later, on the other side of the room from where he'd been, and with a head sized and shaped dent in the plaster on the wall. Afterwards he used wire strippers like the other repair men.

              Icon as he probably looked like that for a brief moment.

              1. DJV Silver badge

                @Arthur the cat

                My second job was an apprentice TV engineer. When we were overworked they'd ask in a guy who'd retired several years earlier to come and help out. He had a habit of checking whether something was live or not by deliberately sticking his fingers on or in it (light bulb sockets were a common thing he'd test in that way). He seemed to be immune to electric shocks or maybe was so used to them that they no longer bothered him!

                1. Rich 11 Silver badge

                  Re: @Arthur the cat

                  That must have been my dad.

                  When I was eight he plugged me into the Xmas tree lights, just to demonstrate that electricity wasn't necessarily something to be scared of but it was something to be respected.

              2. Alien8n

                Re: 10 minutes, not a second more...

                The issue isn't with short shocks. Had a few in my years, got thrown across a room when the back of the plug separated while trying to pull it out, and a screwdriver in a light fitting not realising the pull switch was on (socket was broken and I was replacing it).

                Where it does become dangerous though is if you grip the wires. I remember an incident with a teacher at school once who managed to grab both ends of the wire in separate hands, the resultant electric surge through his body then caused his fingers to clamp shut on the wires making it impossible for him to drop the wires. I believe this was solved by someone turning off the socket that the device was ultimately plugged into, resultant damage was burning to his hands and a new healthier respect for electricity for everyone concerned.

            2. Prst. V.Jeltz Silver badge

              Re: 10 minutes, not a second more...

              " I'm sure electricity can't be as dangerous as it's made out"

              The electroboom! guy on YouTube has lots of knowledge on this - and practical experience

              1. Andy Taylor
                Alert

                Re: 10 minutes, not a second more...

                I'd also check out Big Clive's experiment. It's genuinely scary but you know he was OK because who else would have uploaded the video?

                Obligatory: Do Not Try This At Home.

                https://www.youtube.com/watch?v=-5R-KBa18ME

            3. rcoombe
              Mushroom

              Re: 10 minutes, not a second more...

              " I'm sure electricity can't be as dangerous as it's made out"

              In school, my friend and I used to enjoy sticking a nail in the earth hole to open a socket up and then putting another nail in the live one. Felt like getting your thumb hit by a hammer.

              Then we came up with the bright idea of doing this with two sockets:

              - Stick nail in earth hole to open both up

              - Hold plastic end of 2 opened compasses

              - Stick in live holes

              - Turn compasses so they touch

              - Big bang!

              - Electricity trips out in that block of the school

              - Teacher tells us we're bloody idiots and lucky to be alive

            4. Cynical Pie
              Joke

              Re: 10 minutes, not a second more...

              I smell a life insurance claim

          2. caffeine addict

            Re: 10 minutes, not a second more...

            How did I not know these existed...?

          3. Boris the Cockroach Silver badge
            Flame

            Re: 10 minutes, not a second more...

            We use colour coded padlocks..... even then I'm sometimes forced to remove the RCB at the main distribution board... and lock that.

            Can I turn the power on? comes the question, and I dont know why they bother because whatever version of 'no fucking way mate' I answer , it seems to translate to "Yes go ahead" inside their brains....

        4. diver_dave

          Re: 10 minutes, not a second more...

          Indeed.

          A friend of mine working on a large development in central London had a rather interesting experience.

          He actually padlocked the cabinet closed and placarded after isolating the 3 phase supply. Went up two floors to continue installing busbars and starting termination. About 20 Min later he dropped a bar bolt. Which apparently then proceeded to short the bars with much banging, flashing and smoke.

          Rather rattled (possible understatement warning) he flew back down to the plant room to find the Architect, site manager and owner had cropped the lock off so they could look at the lighting on a finished floor.

          There was a subsequent "free and Frank expression of views and opinions management".

          Followed by the culprits having a rather large whipround.

          According to him he made more in 5 minutes than he normally made in a month as a third year apprentice.

          He still talks about it 30 years later

          Dave

        5. swm Silver badge

          Re: 10 minutes, not a second more...

          When I was wiring up heavy 3-phase equipment I would padlock the fuse box when I was working.

        6. Scott 26
          Mushroom

          Re: 10 minutes, not a second more...

          >I liked the days when we had real fuses, not resettable circuit breakers. You could take a fuse out and keep it in your pocket to prevent that sort of thing.

          Some bugger around here keeps taking the fuses, so I carry a 4" nail in my pocket.

      2. oldgreyguy

        Re: 10 minutes, not a second more...

        ahh yes, I have a huge screwdriver with a bit of foreign metal fused forever on its tip.

        in the 60's, living in the south (north carolina), my wife and I scraped and saved up to buy a window air conditioning unit.

        The day arrived... we went to Sears, laid down our hard earned cash, rushed back to the apartment. After getting the unit secured in the window frame, looked down and saw that the wall outlet was different. I went down to the basement, unscrewed the fuse for that circuit, went to the hardware store and returned with the proper outlet. down on my hands and knees, removed the cover plate, popped the first wire off, went for the second one... Blinding flash, very loud pop... I am now sitting on my ass two feet away, with my rather large screwdriver welded to the socket.

        Wife comes in and asks "Did you turn the power off just now?"

        Moral: take little round fuse with you, til you are ready.

        yep, still use the screwdriver, could clean up the spot weld I suppose, but it is a reminder of "bad things happen"

        1. Anonymous Custard Silver badge
          Headmaster

          Re: 10 minutes, not a second more...

          Also reminds me of the induction/site orientation training for one of our customers (global multinational semiconductor manufacturer) who is very big on safety that I had a few years back. Up comes the various slides, unless the training lady gets to the one about LOTO (lock-out / tag-out).

          I looks at this, and looks around the room to see if anyone else has spotted the issue. Apparently not, so I ask the trainer "what's actually wrong with that image?". She looks and ponders for a moment, then says she can't see anything. At this point I point out that her slide seems to show someone has actually managed to lock-out a breaker in the ON position rather than the OFF one (which I didn't think possible, but they seemed to have managed it). Cue her going white, before commenting that she's been using these slides for years and no-one has spotted that until now.

          Had a colleague have to do the training recently and asked him to check that slide out for me. Seems that the image is still in it, but it's now a question posed of "what's wrong with this image" asked to the trainees...

          1. A.P. Veening Silver badge

            Re: 10 minutes, not a second more...

            Had a colleague have to do the training recently and asked him to check that slide out for me. Seems that the image is still in it, but it's now a question posed of "what's wrong with this image" asked to the trainees...

            That is the correct solution, must be a qualified and experienced teacher.

          2. Martin an gof Silver badge

            Re: 10 minutes, not a second more...

            "what's wrong with this image"

            Reminds me of the recent story about a Health and Safety course at the University of South Wales*:

            University of South Wales sorry over health and safety course

            To quote: a lecturer got "very basic scientific information" wrong - for example he claimed that bleach was an acid when it's an alkaline, says the Times.

            He also said that "voltage" was named after Voltaire, the French philosopher - when it's in fact named after the Italian physicist Alessandro Volta.

            The inquiry found the lecturer, who was teaching safety and business risk modules, suggested that oil could be heated to 360C - when it can actually catch fire at 250C.

            It's claimed he also told students that the "most important thing" they had to do in the workplace was to "keep your job and not be prosecuted".

            M.

            *Slight declaration of interest - the University of South Wales used to be called the University of Glamorgan (with a glorious sign on the A470 which pointed the way to the "University of Glam" - wish I'd been able to get a photograph) and before that it was the Polytechnic of Wales which I attended in the 1980s and 1990s. Before that it was the South Wales Mining School and there was still a mock-up "mine" under one of the buildings when I was there.

            Of course, it is a vastly different institution these days, but sadly it seems that the quality of teaching perhaps doesn't live up to the quality of all the nice new campuses.

        2. Yet Another Anonymous coward Silver badge

          Re: 10 minutes, not a second more...

          One of the kitchen sockets blew in my new apartment

          Turned of the breaker and, because Mrs Coward didn't have any stupid children, plugged a lamp in to the socket to check it was off.

          Took the cover off and got zapped.

          Turns out in Merkin houses the top and bottom sockets on a double outlet are on different circuits, turning off the top one leaves the bottom one live....

          Fortunately their electricity is like their beer so only a little tingle instead of the, "fsck me" 240V arm tingler.

          1. jtaylor Bronze badge

            Re: 10 minutes, not a second more...

            "Turns out in Merkin houses the top and bottom sockets on a double outlet are on different circuits, turning off the top one leaves the bottom one live...."

            Well, each outlet can be independently wired. It lets you control one outlet with a switch while leaving the other hot, useful for lighting. Different circuits, though? That's diabolical.

            1. Yet Another Anonymous coward Silver badge

              Re: 10 minutes, not a second more...

              > Different circuits, though? That's diabolical.

              It's apparently to increase the current.

              You can plug your slow 110v kettle into the low one and George Formby grill into the upper one

              1. Andrew Norton

                Re: 10 minutes, not a second more...

                Slow is right.

                Takes 5 minutes to boil a full kettle's worth (1.7l) - because that's how big my tea mug is. (every time I've made an FCC submission over the last 2.5 years, I include a photo of my drinking from it, from my bigger, better insulated and non-corporate freebie mug, just to maybe annoy Pai)

                And yes, the 110 does feel like a tingle, like you've just almost hit your funny bone. now the 230 from a dual phase socket (dryer, AC unit, cooker etc) will make me jump a little. (I'm used to it, from being drunk every single afternoon lab session through uni, at the electrical engineer and electronics department at liverpool uni, which is literally across the road from the union.

          2. H in The Hague Silver badge

            Re: 10 minutes, not a second more...

            "the top and bottom sockets on a double outlet are on different circuits"

            The instructions for leccy work of one of my customers state very clearly:

            ... every electric circuit shall be proved dead at the point of work, just before starting work

            Saved me more than once, together with my paranoia.

            Years ago I was going to replace some socket outlets for a friend who'd moved into an oldish house. Turned all the circuits off at the consumer unit/switchboard. Checked outlet: no voltage between the contacts. Being paranoid I also checked each contact to earth - tester lights up. Turns out some *##@! had wired the consumer unit up with the single pole MCBs in the neutral rather than the live conductor! So 230 wiggly volts still on all circuits :(

            Another time I was going to replace a switch, late Friday afternoon. Turned the power off. Took the cover of the switch and tested it - still live. Then realised I was tired and had only thought about turning the power off but not actually done it.

            1. Yet Another Anonymous coward Silver badge

              Re: 10 minutes, not a second more...

              >.. every electric circuit shall be proved dead at the point of work, just before starting work

              That was the annoying bit - I had checked the bad socket was dead

              I didn't check the other socket in the same outlet was also dead ....

          3. This post has been deleted by its author

        3. Anonymous Coward
          Anonymous Coward

          Re: 10 minutes, not a second more...

          My grandfather, back in the bad old days of the 1950s, used to check all his workers' tool boxes, and if necessary insulate the shafts of their screwdrivers and the handles of pliers himself to make sure it was done properly.

          He taught me to work on circuits as if they were live "because one day they will be".

          This was in the context of signalling systems where it sometimes became necessary to fix a loose or broken contact, and you can't stop the East Coast main line while you do it.

          1. Kiwi
            Pint

            Re: 10 minutes, not a second more...

            He taught me to work on circuits as if they were live "because one day they will be".

            Lovely fellow that.

            I was taught the same way.

            Has probably saved me a little bit of hassle. Especially in a place where the wiring was, well, lets just say "incapable of passing any level of safety inspection". Treating everything as live - even when the breakers and building mains were off - meant I only got to cuss about how messed up the site was and suggest we talk with the powerco about a larger level of disconnect than getting taken home in a nice wooden box.

            Me mum helped with some of my training as well.. We had a dead TV and I'd been working on it (still only a kid at the time, but had been building and fixing circuits enough to be confident of solving this). I got up and left the room. Mum came in, saw me gone, wondered if I'd been successful, plugged in the TV, left the room. I came back.. "Oh yes, must check that". Touched something live. Picked myself up off the opposite wall, then went and had a little discussion with my mother about not plugging stuff back in without warning people. (given that 240v can kill easily enough, she was a tad upset she'd nearly killed her kid).

            1. Anonymous Coward
              Anonymous Coward

              Re: 10 minutes, not a second more...

              From work days, I remember the tale of the company that occupied two adjacent buildings. It was sold off as two going concerns. Each building had its own switchboard, meters and so on.

              Came the day when, as part of the sale, an electrician turned up to check the wiring. He turned off the main breaker to his building, and half the lights went out.

              The wiring was so confused that it more or less had to be redone from scratch in places. As stuff had been added, holes had been made in the wall and things wired across. It was possible that some machinery was getting two phases from one switchboard and the third phase from another.

              In another company sale, power was turned off while the building was checked, and at length someone arrived from the golf club half a mile away to ask if they too had a power cut. The former owner, a keen golfer, had diverted the firm's supply to the clubhouse.

          2. Alien8n

            Re: 10 minutes, not a second more...

            Strangely enough I was taught electrical work by my blind step father. He'd actually worked at one of the power stations before going blind, but was still able to talk a teenager through how to replace plugs and rewire sockets.

        4. Sparkypatrick
          Alert

          Re: 10 minutes, not a second more...

          I used to carry a pair of pliers with a notch melted into the cutting section as a reminder to always double (or triple) check I had isolated the correct circuit. I eventually replaced it with one that was more effective at cutting - and with better insulation - but only after the lesson was well hammered home.

      3. Aladdin Sane Silver badge

        Re: 10 minutes, not a second more...

        My parents still have the bread knife with a 5mm semi circular hole in the edge from where my brother tried cutting through the vacuum cleaner cable whilst wearing his wellies. Apparently his exact words were "Mummy, the hoover went bang and bit me!"

      4. Anonymous Coward
        Anonymous Coward

        Re: 10 minutes, not a second more...

        Surely everyone knows that you always advise of at least twice as much downtime as you need?

        That way, you can either look supremely l33t when you get the job done in less time, or still have some time to back out or fix in the vanishingly unlikely event that something may have gone wrong with your carefully laid plans «coughs».

        But, yes, make sure that the person in charge of the "switch back to normal" switch (or asking the keeper of the switch to turn it) is you, and nobody else!

    2. HarryBl

      Re: Did the CCM run a timer?

      When I was working on a ship's radar scanner I always made sure I had the fuses in my pocket before I climbed the mast.

  4. steviebuk Silver badge

    I've also learnt...

    ....to do what Scotty from Star Trek said to do. Say it will take longer than it actually will, makes you look better when you do it in less time. That would of also helped out with the "edit".

    1. Martin Summers Silver badge

      Re: I've also learnt...

      Or better still in this case no time frame should have been given. He should have said I will call you when I am finished with testing.

      As for the guy who waited 10 minutes before switching it back to live, I'd say he has some of the blame here as taking things literally like that is either not caring, dumbness or malevolence. If you know a test is being carried out, assuming that it's definitely going to be all over within the given time frame on such a critical system is pretty stupid. Nothing at all would have been hurt by the guy picking up the phone and checking if all was clear.

      1. Anonymous Coward Silver badge
        Big Brother

        Re: I've also learnt...

        I would go so far as to say "most of the blame".

        Even if he had monitored the transaction and seen that it had finished, he wouldn't have known whether Geoff had actually finished his testing. So an approach of "it's been 12 minutes and I haven't heard, let's phone and check" would've avoided a whole heap of hassle.

    2. Kiwi

      Re: I've also learnt...

      Say it will take longer than it actually will, makes you look better when you do it in less time.

      I often did that but still went to time. Just wanted to be sure I had plenty of time up my sleeve for unexpecteds.

      I did find that a customer will be exceedingly pissed off if you go over a year long job by merely a few seconds, but will be barely happy if you complete a multi-decade job in a few days.

      But it is much nicer to say "we're done ahead of schedule" instead of "I'm sorry we're late, we're going to need more time".

      If only I could've managed to get co-workers etc to promise long rather than try to please the customer and promise short - so many promises at 10am of "Oh yes, we can scan your 4TB drive and have it all done and back to you by lunch time" (NOT exaggerating! It happened at least once - 2 hours to recover a large HDD!). I always doubled then tripled expected timeframes with HDD issues.

  5. Nick Kew
    FAIL

    A fine wunch

    This is great. Insights into cockups in a consumer-facing biz, where we can all see ourselves as potential victims!

    Obviously this comes from a different era, when there was inadequate separation of testing and production systems. But ... hang on ... a 56k modem dates it to not so very long ago: if I'd read this story when those were current I'd've thought the same - surely such a story must be from the already-distant past ...

    1. TrumpSlurp the Troll
      Windows

      Re: A fine wunch - 56k modem?

      It may be not that long since you owned a 56k modem.

      Then again it is almost 20 years since I had 128k to my home office using two bonded 64k ISDN links.

      Major computer centres with very expensive 56k T1 leased lines would be perhaps a couple of decades earlier, minimum? Banks spend money. They have a lot of it (mainly ours).

    2. Anonymous Coward
      Anonymous Coward

      Re: A fine wunch

      56K leased lines have been available since the 1970's, although I'm not sure if they are still available today given the availability of alternatives and things like ISDN/E1/T1 being decommissioned and replaced by DSL/Ethernet.

      56K likely means it was a US-based telecommunications system based around 56K timeslots/T.1 rather than a European system based around 64K timeslots/E.1.

      You will be thinking of 56K modems which were available for communication via PSTN lines (V.90/K56/V.92) from the late 1990's.

    3. Anonymous Coward
      Anonymous Coward

      Re: A fine wunch

      56k modems of the non dial-up have been around for decades. I remember visiting a bank's data centre in the early 80s that had a GPO (BT) 48kbps V.35 analogue wideband modem that was fed by special cable (co-ax?) to the nearest trunk unit where it replaced 12 analogue voice channels on a trunk circuit. It must have cost an absolute fortune to run, but obviously worth the money. (I think the rest of their branch network was running 1200bps, but then everything was character-based)

      1. Anonymous Coward
        Anonymous Coward

        Re: A fine wunch

        fed by special cable (co-ax?)

        Unlikely to have been co-ax, it was probably a 4-wire circuit, two twisted pairs (one for each direction).

        1. Anonymous Coward
          Anonymous Coward

          Re: A fine wunch

          Very likely to have been co-ax.

          Standard T1/E1 circuit presentation was either unbalanced (75-ohm coax with BNC connectors) or balanced (120 ohm UTP with RJ-48 connectors which is basically RJ45 with different pinouts...)

          1. diver_dave

            Re: A fine wunch

            Thanks...

            Mention BNC...

            Ruin my Monday..

            Back to therapy!

            1. Kiwi
              Pint

              Re: A fine wunch

              Mention BNC...

              Ruin my Monday..

              Back to therapy!

              I'll join ya.

              Just the other day I came across an 8-bit ISA LAN card with a BNC connector as it's only connector (save for the ISA).

              Think I'll need some IPA to forget the ISA, and a whole load of therapy to forget the associated 'network' that went with it :(

          2. Anonymous Coward
            Anonymous Coward

            Re: A fine wunch

            Standard T1/E1 circuit presentation

            That's a 1.5 or 2Mbit/s digital link, not a 48kbit/s analogue modem.

            1. Anonymous Coward
              Anonymous Coward

              Re: A fine wunch

              "That's a 1.5 or 2Mbit/s digital link, not a 48kbit/s analogue modem."

              The assumption was that the 56k modem was likely to be a digital line rather than an analogue modem given the time frame of the incident.

              1. Anonymous Coward
                Anonymous Coward

                Re: A fine wunch

                I remember visiting a bank's data centre in the early 80s that had a GPO (BT) 48kbps V.35 analogue wideband modem that was fed by special cable (co-ax?)

    4. katrinab Silver badge

      Re: A fine wunch

      56k modems came out in February 1997. Most people started buying them in 1998 when the V90 standard was published, and the first modems that complied with it came to the market. That's about 21-22 years ago. I moved to broadband in around 2001.

      1. Anonymous Coward
        Anonymous Coward

        Re: A fine wunch

        Years used to be so long and progress so quick in the late 90s. I distinctly remember having a 56k modem and a large phone bill for ages and ages, but in hindsight, it can't have been more than 3 years., since in 2000 the student housing was renovated and we got 100 MBit(!) connectivity.

        That same summer, I was able to build a Duron 600 with a giant 15 GB IBM Deathstar HDD to replace the old Pentium... it's been all downhill from there, I'm sure.

    5. dfsmith

      Bandwidth

      I want to know how he thought he could use the "upload service" from a 56kbps modem for 25 megatransactions in 600 seconds.

      1. Olivier2553

        Re: Bandwidth

        one character per transaction maybe (1.37 to be exact).

  6. Pascal Monett Silver badge

    "the bank's reputation would be left in 'tatters' "

    Yeah, as if that matters. A certain TSB is still open and still has most of its customers, if I'm not mistaken.

    1. Jimmy2Cows Silver badge

      Re: "the bank's reputation would be left in 'tatters' "

      Ah but they're only the customers Lloyds didn't want, so TSB doesn't really give a shit.

      1. Anonymous Coward
        Anonymous Coward

        Re: "the bank's reputation would be left in 'tatters' "

        We didn't get a choice. The town had a Lloyds and a Halifax so the Lloyds had to go.

        At least I now know (fingers crossed) Nationwide has vastly superior IT systems compared to either Lloyds or TSB.

        1. Colonel Mad

          Re: "the bank's reputation would be left in 'tatters' "

          Mm, not sure about that, I had an issue with my Nationwide mortgage!

        2. Anonymous Coward
          Anonymous Coward

          Nationwide has vastly superior IT systems

          Is that after the recent (past few years) multi-day FUBAR they suffered?

          1. Anonymous Coward
            Anonymous Coward

            Re: Nationwide has vastly superior IT systems

            I am just writing about my experience since the TSB foulup and our getting a Nationwide account. We've kept the TSB one open for the people who pay us money periodically and seem to have trouble reading emails about change of bank details.

      2. Doctor Syntax Silver badge

        Re: "the bank's reputation would be left in 'tatters' "

        When I was a Lloyds customer neither the former TSB nor the Lloyds branches gave a shit anyway. The staff who'd been in the local branch Lloyds had closed were OK but the big town branch? No way.

    2. Anonymous South African Coward Silver badge

      Re: "the bank's reputation would be left in 'tatters' "

      Was wondering when TSB's about to be mentioned... :)

      How's things going at TSB by the way?

    3. katrinab Silver badge

      Re: "the bank's reputation would be left in 'tatters' "

      I guess I'm still technically a customer. There is £1 left in the bank account, and no movements since I cleared the rest of the balance out after the crisis started.

    4. Anonymous Coward
      Anonymous Coward

      Re: "the bank's reputation would be left in 'tatters' "

      Now that TSB is reducing the interest rate on its current account, I no longer really have any reason to keep my "backup current account" with them. I am wondering how many customers were still remaining with them for that sole same reason, and whether they are now starting to see a final exodus?

      Their newish and infamous internet banking site still really is shonky: alarming pauses when moving from one page to the next (I have a nasty feeling that rather far too much of it is 'powered' by dodgy JavaScript, rather than server-side), and it seems to lose track of where it was far too often, and then requires you to login again.

      It turns out that I can now beat the interest income on my account just by taking the switching bonus from another bank, so that's what I'm doing.

      Early signs are also not encouraging: the application form crashed out with an error message on final submission (but later turned out to have been successful anyway), and the website has some rather flakey and indeed almost phishy-looking grammar, but once I've taken their money, I'll just move my account again to a more reliable bank, if there actually is another one, that is (my main current account provider is very reliable, but almost all of the other banks seem to have flaws of some kind!).

  7. Will Godfrey Silver badge
    Coat

    Jealous now.

    I've made some interesting screw-ups over the years, but never had the opportunity for a truly great one like this.

    1. Anonymous Custard Silver badge
      Trollface

      Re: Jealous now.

      As the old saying goes - "Be careful what you wish for..."

  8. This post has been deleted by its author

  9. Waseem Alkurdi
    Thumb Up

    Reversing the debits into credits?

    That's one fucking brilliant move.

    1. Doctor Syntax Silver badge

      Re: Reversing the debits into credits?

      It's the obvious move. The tricky thing is the legend or, if it's a company's accounting system, the transaction type you put against it to explain it on the statements.

    2. Uncle Slacky Silver badge
      Windows

      Re: Reversing the debits into credits?

      Reminds me of the time I tried to help a young programmer (ex-YTS trainee, but with a lot of natural talent) simplify an accounting system he was writing. To save needing two separate routines for credits and debits, I recommended simply multiplying by a value which would be either -1 for debits or +1 for credits, the rest of the calculation being identical. Unfortunately his maths knowledge wasn't great and he didn't trust my solution.

      25 years later, he's now CEO of the same firm (its direct descendant, to be exact).

      Icon: closest thing to an old fart I could find.

      1. Scott 53

        Re: Reversing the debits into credits?

        I wouldn't trust it either without confirming that you didn't have to use ±1.0 to avoid an implicit conversion to integer.

        1. Uncle Slacky Silver badge

          Re: Reversing the debits into credits?

          No, IIRC the language concerned (Dataflex, v2.3b for anyone who can remember back that far) didn't do implicit conversions.

      2. katrinab Silver badge

        Re: Reversing the debits into credits?

        The usual convention is to have positive numbers for debits and negative numbers for credits.

      3. Alien8n

        Re: Reversing the debits into credits?

        An old IT manager at one place I used to work for (no longer there, strangely enough) wrote a debit/credit system to handle partial payments to customer accounts. Needless to say he f$&ked it up royally to the point that any payment not an exact amount of the bill due resulted in random debits and credits being applied, often for pennies here and there, but in some cases for thousands of pounds. The transactions were all for less than £15 (the monthly subscription the customer was paying).

        Eventually got to the bottom of it and once fixed the thousands of pounds that customers actually owed suddenly became known to the client. I had great pleasure ripping into his "expertise" in the disciplinary meeting as he blamed me for his cock up. Said IT manager was the same one who claimed he did 6 degree courses in 18 months (all concurrently), was an Olympic archer, classically trained pianist, friend of Richard Branson, owned a Cray supercomputer that he kept in his barn, ran his own multi-million dollar business selling data to the oil industry, and yet couldn't understand the maths behind calculating VAT.

        1. Uncle Slacky Silver badge
          Trollface

          Re: Reversing the debits into credits?

          > Said IT manager was the same one who claimed he did 6 degree courses in 18 months (all concurrently), was an Olympic archer, classically trained pianist, friend of Richard Branson, owned a Cray supercomputer that he kept in his barn, ran his own multi-million dollar business selling data to the oil industry, and yet couldn't understand the maths behind calculating VAT.

          He wasn't called Jake, by any chance?

          1. Alien8n

            Re: Reversing the debits into credits?

            Dick actually :p (Richard, but everyone called him Dick, usually emphasised in such a way to imply the suffix "head")

          2. Kiwi
            Pint

            Re: Reversing the debits into credits?

            He wasn't called Jake, by any chance?

            Bit late to the party but that was exactly my thought when I read that! :)

    3. Nick Kew

      Re: Reversing the debits into credits?

      patch --reverse

      A regular in the developer toolkit. Along with change-control and CMS variants catering for everything from failed/abandoned experiments to actual cockups.

      1. Anonymous Coward
        Anonymous Coward

        Re: Reversing the debits into credits?

        Not that appreciated in banking transactions, though, although I would personally appreciate a fully worked through undo function (could bugger up quite a lot f crime as well :) ). The amount of hassle people have to go through to undo misallocated funds is painful. No doubt that is to teach them to never do that again, but to err is human. To bill for it is banking.

    4. A.P. Veening Silver badge

      Re: Reversing the debits into credits?

      Even worse, I'd say there must be multiple readers and commenters here who have done that. Most probably won't admit it, but I can as I used that to repair some manager's (not mine) screw up. I wrote a program from scratch to create a file with special formatting using the original input file in a couple of hours, made me late for lunch. To make things even more interesting was that the problem was in Prague, while I was working in Amsterdam.

      1. LeahroyNake

        Re: Reversing the debits into credits?

        I put something similar together to help a customer migrate to a new printing system. They were used to using the old print submission application where they could set rotations, scale, copies etc... let's just say it was quicker to spend half a day writing a small programme that took the output from the old system and repackaging it so that it would output on the new machine than training several users.

        The manufacturer was very interested in my code and kept asking me to implement additional options....

  10. Huw D

    Make mine a double...

    I am reminded of my first temp job as a tape monkey for a UK bank, where we got asked to process a tape and despite us verifying that the tape had been processed, got asked to run it again.

    Doubled transactions, obviously.

    Root cause was devs testing in production.

  11. Kubla Cant Silver badge

    Reversals

    I'm sure there have been occasions when I've seen a trio of entries on my bank statement where a transaction is duplicated, then the duplicate is reversed out. I've never paid much attention because the final balance is correct and the interest implication, if any, is trivial. I'd guess it's probably caused by a duplicated Direct Debit request.

  12. Anonymous Coward
    Anonymous Coward

    I call bullshit

    Sorry but every time I read these I think hmm did this really happen? You always get some smart arse replying saying you can't have lived in the real world unless you've experienced this. Well, I do, and haven't. Sure mistakes happen but when you're talking about these environments and such trivial mistakes - time after time - it seems very unlikely. Especially when they all just happen write in to The Reg because they want to discuss it?

    Firstly - if it's real - someone name the bank or link to a story about it. If it made headline news then it should be pretty easy to find even if it was decades ago.

    Secondly - how the fuck would anyone be employed with that level of responsibility who would let a cock-up like that occur? That's like saying an air traffic controller gave the wrong instructions to an aircraft because their watch broke and they didn't know the exact time. The stories get more bizarre when you're talking about organisations with high levels of responsibility (i.e. banks, airports, etc) which coincide with absolutely trivial errors. Again and again. Really?

    Yes people screw up but these just seem like made up stories, as do a lot of the comments where people come out with pub-level bullshit like "that's nothing, back in the 80s I put the wrong floppy disk in and the plant melted down". Did you? Did you really?

    1. Anonymous Coward
      Anonymous Coward

      Re: I call bullshit

      My what a narrow world you live in my friend.

      Yes really. I've worked in Banks and seen equivalent stupidity (done some too :) )

      1. andy 103

        Re: I call bullshit

        My what a narrow world you live in my friend.

        Before suggesting I'm narrow minded have you ever considered why:

        1. Any time these stories occur people write into The Reg - such that every single week - they can produce a story?

        2. People are so willing to discuss them with the World + Dog, even though the stories are always anonymised and about something that the organisation in question would absolutely not want anyone involved talking about?

        3. How an organisation has become the "biggest" in their sector, whilst also seemingly having bugger all in place to stop a trivial error? See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.

        4. Why you can never find independent and publicly verifiable stories to back any of it up, even when The Reg claims it's a story reported by mainstream media?

        These are just a small section of the issues. I don't think it's worth going further if you're too narrow minded to accept some of these stories might just be made up.

        I've worked in Banks

        Hopefully not in any position of responsibility...

        1. Anonymous Coward
          Anonymous Coward

          Re: I call bullshit

          Can I suggest politely that if you wish to post as AC, you remember to do so when you reply?

          The world is indeed a big and wonderful place in which things happen that most of us wot not of.

          1. andy 103

            Re: I call bullshit

            Can I suggest politely that if you wish to post as AC, you remember to do so when you reply?

            Well I thought it was hypocritical I was moaning about the stories being anonymised whilst posting anonymously so wanted to pre-empt that. Quite happy to make my point.

            I'll gladly be proven wrong if someone can show an independent source that backs the story up.

            FTFA: it was "the biggest bank in the country" and the value was in "billions" and "The story even made the front page of the newspapers". So, should be pretty easy to find something that reports a story this big, no?

            1. MonkeyCee

              Re: I call bullshit

              While being quite a low level bank employee* in NZ I'd get a notification email about once a month that something had gone wrong with the clearing house, typically with multiple deposits. It got sorted in the next processing, so I presumed some sort of batching system.

              Generally it was felt that it was better to overpay deposits, since those are often how people are paid, and delaying payment buggers up standing orders, like rent.

              For me it meant that I'd get a few calls asking if we were going to sot out their three paycheques, or should they talk to their work, you could tell them that while unfortunately two of them would be gone by tomorrow, the bank had it in hand.

              No idea what it looked like on the end of the bank that over paid.

              As for the size of it, any bank handling government accounts can have scales of daily balances that are mind boggling. The hot money accounts also move billions in and out of NZ for about eight hours at a time.

              That's for a dinky country that gets missed off maps with a few million population. I would expect there to be thousands of banks doing transactions of that size in the world.

              * most of the job was tech support for their banking app, with a team of actual bankers we could pass over actual banking stuff. Level 2 phone monkey rather than actual banker. As reflected by my wages :)

        2. Anonymous Coward Silver badge
          Facepalm

          Re: I call bullshit

          1: He never said "narrow minded". Living in a narrow world is very different.

          2: These stories happened a long time ago and the reg readership is very large. If you've got 40 years of experience to draw on from a million users, it's entirely possible to get 52 stories a year. And people continue to make cockups today which may well become publishable stories in a few years time.

          3: Just because you haven't read something in the media, doesn't mean it hasn't been published. Again, bear in mind that it may have been published 40 years ago in a foreign country.

          4: Banks were some of the earliest adopters of computerisation. They had to devise best-practice themselves. A lot of what we consider obvious now wasn't at the time and they will have made many mistakes as part of the learning process. Thankfully the state of the industry has improved and lessons learned spread (largely through stories like this being told to colleagues down the pub) such that major cockups are far less major or frequent now.

          And yes, I saw some absolutely shocking practice in the bank I did some work in a long time ago!

          1. andy 103
            Facepalm

            Re: I call bullshit

            a million users ?????

            Even Wikipedia says the following of The Reg:

            In 2011 it was read daily by over 350,000 users

            In October 2013, Alexa reported that the site ranked #3,140

            In April 2015 ... the site ranking dropped to #3,430

            continues to decline, at 11 April 2016 ... ranking for the site of #4,750

            ...

            By 12 September 2018 ... ranking had continued to fall still further to #7,194.

            Oh dear.

            1. John Brown (no body) Silver badge

              Re: I call bullshit

              A drop in ranking doesn't necessarily reflect readership falls. It's just as likely other sites are increasing readership more rapidly.

              Anyway, it's unlikely anyone, even El Reg can quantify the overall man years of experience in the readership but there are most definitely commentards posting here with many decade of life in the IT field. It's approaching 40 years for me so even assuming an average of 20 years across 300,000 users, that gives 6 million years of experience to draw from.

            2. Anonymous Coward
              Anonymous Coward

              Re: I call bullshit

              I'd be very surprised if (m)any Reg readers are the sort to have the Alexa toolbar installed in their browsers, and if they also use JavaScript or web bug tracking, a fair proportion of readers will probably block those as well, so I wouldn't really take Alexa rankings particularly seriously…

        3. Anonymous Coward
          Anonymous Coward

          Re: I call bullshit

          This really shouldn't need explaining, but when you have an incident such as in this article you want to vent to someone, help anyone else avoid the same mistake, and maybe just tell a good story.

          People are willing to discuss because they're anonymised; they wouldn't be doing so if the names were included, and this offers plausible deniability.

          I've only recently outlined how to make a change to a testing environment to ensure that its data cannot potentially feed into production if the correct steps aren't followed, because the environments team had decided that non production systems for the Super Special Customer should run on the production system.

          Then there's undocumented processes created which go wrong because the team involved didn't fully understand what they're doing, and most importantly didn't run it past anyone else for verification. In one particular case things broke after a year of running this process, because a (legitimate) change was made on the system which violated an (incorrect) assumption when creating the process.

          This happens All The Time, although processes are improving. Then the customer gives you grief for development and testing estimates.

        4. Mark 85 Silver badge

          Re: I call bullshit

          See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.

          As late as the 2000 or so, I worked with several insurance companies that the idea of having a dev and a production environment was lost on them. We worked on live data. Scary to say the least.

        5. Kiwi

          Re: I call bullshit

          My what a narrow world you live in my friend.

          Before suggesting I'm narrow minded have you ever considered why:

          1. Any time these stories occur people write into The Reg - such that every single week - they can produce a story?

          I believe El Reg has at least a couple of dozen readers. Some of us have been in or around IT or engineering or other stuff for decades. Some of us have made or seen mistakes made more than once.

          It could be said El Reg is the best IT news site in the business (well, even if just by me :) ) - there is just a slim chance that a few people will congregate here who know of such mistakes being made. And we do have a tendency to blow our own trumpets. Those "atmospheric noise" videos on YT? Just the physical manifestation of an El Reg comments thread.

          2. People are so willing to discuss them with the World + Dog, even though the stories are always anonymised and about something that the organisation in question would absolutely not want anyone involved talking about?

          I took you up on your challenge and tried to hunt for the case referenced in the article. I quickly gave up because, well, this is NOT a rare thing. This is just one of thousands of results, and I was several pages in before I gave up here.

          These sort of errors happen a lot (not just this one but those in other stories). People are right to say you've had a narrow experience of life. A few changes of details to keep things anonymous, but even without that how is a company going to know they're the ones being talked about? These things happen a lot

          Just look at the article on BGP taking out Farcebork, clodfool and scamathon. Think that wasn't a bit of 'human error' - or maybe it just didn't happen after all?

          3. How an organisation has become the "biggest" in their sector, whilst also seemingly having bugger all in place to stop a trivial error? See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.

          Again, these people have been doing this for decades. Some of the people who read here played a big role in developing the internet we know today, and the OS's we use. The idea of separating dev/production - do you have even the slighest inkling of how it came about as a standard?

          It's obvious now - and of course now we all have oodles of cash to throw at doubling up on IT and massive numbers of VM's and huge raid arrays.. But way back when, testing had to be done on the same hardware during downtime, and hope like hell backups were there.. Or just be sure you did a good job. I've taken overseas trips on the overtime pay from individual series of tests because they had to be done out of hours, and we only had one plant to test on so that mean working through the night to get things ready, tested, and the results cleaned up and the line back to being ready for the next day's production run. Duplicate kit can be prohibitively expensive (as can mistakes on the only production line in the factory :( )

          How do you think locking off and tagging off switch gear came about? IIRC the last straw was when an engineer had her had crushed (fatally) by some robotics, because she was working out of site of anyone else and it was thought she'd left the area (I am happy to be corrected on any of these details), possibly even at a Ford plant or some other US-based car manufacturing plant. A solution that wasn't obvious until enough people died. Other solutions were tried after other incidents, but individual locks for each engineer? This poor lady was the catalyst for something that keeps the rest of us safer.

          Some of us have been working far longer than the current standards have been around.

          4. Why you can never find independent and publicly verifiable stories to back any of it up, even when The Reg claims it's a story reported by mainstream media?

          Take a look at MSM over the last few decades. For this story alone you have thousands of candidates (and I even wrote one very sweary post as, well, if this happened in NZ then I was a victim of the screw up and it actually made quite a mess of my finances that took a long time to clear - yes I was a customer of our largest bank, yes it made front page (and TV) news, and yes it was some filthy scoundrel at a bank double-dipping "accidentally" - but the post was too much even by El Reg standards so I deleted it. And yes, nearly 30 years on I am still very pissed at that person!)

          These are just a small section of the issues. I don't think it's worth going further if you're too narrow minded to accept some of these stories might just be made up.

          Maybe. Probably. Probably if not 'made up' then at least some alteration of some key facts to keep it anonymous.

          I've worked in Banks

          Hopefully not in any position of responsibility...

          Actually, I kinda hope with your limited RW experience that you are not in places of any real responsibility. "You're not a real engineer until you've made at least one $50k mistake". With time, you'll make the sort of screwups that teach you how to manage serious problems - maybe a new 'never again' standard will be developed because of you. God willing, you'll survive the experience and in 20 years some 20something will be telling you it was made up, 'how can someone have made such a mistake when we have the Andy103 Law to prevent people doing such things?'

          1. Alien8n

            Re: I call bullshit

            "You're not a real engineer until you've made at least one $50k mistake"

            Only $50k? I've known people make £500k mistakes and still somehow keep their jobs. I've also been on the receiving end of some engineer's mistakes and had the pleasure of phoning the CEO on a Sunday to tell him I was shutting down a £50M a day production line until the next day when every single device hit test and started failing because of said engineering cock ups. Not once, but twice.

            For those interested: IGBTs that were supposed to be 5V rated. On silicon these should be tested to about 5.2V, the waferfab set their pass limit to 7V because they did such a bad job on them which resulted in a 90% fail rate at final test. The second time the IGBTs were built with a design flaw that meant current leaked across the gates. Ironically the design flaw meant that any that didn't fail the leakage test were nigh on indestructible and would pass avalanche testing at a spec way above anything they were designed for. From memory these ones had about 70% fail rates.

    2. Arthur the cat Silver badge

      Re: I call bullshit

      Sorry but every time I read these I think hmm did this really happen?

      If you think this story unlikely you should hear some of the ones an ex-RAF friend of mine told about working in the missile division of a major UK deathtech company in the 70s and 80s. The one about the constipated 4 star US general and the anti-helicopter missile always got a laugh.

      1. Anonymous Coward
        Anonymous Coward

        Re: I call bullshit

        I recall the extremely dirty dual-fuel engine on a certain British tank, and how the crews would attempt to start it up if a suitable MoD official was standing in the right place to be covered in soot. I've seen it with my own eyes and watched a major trying to control his amusement. Yet other people have told me it couldn't possibly have happened.

        1. Arthur the cat Silver badge

          Re: I call bullshit

          Looks like the Phantom Downvoter is out and about today.

          1. Anonymous Coward
            Anonymous Coward

            Re: I call bullshit

            I don't think andy 103 is a phantom, but his knickers seem to have been well and truly twisted.

            1. Arthur the cat Silver badge

              Re: I call bullshit

              I wasn't thinking of andy103. A few years back I noticed that every once in a while an entire thread of comments, none of which were particularly bad or controversial, would receive a single downvote each. As a suitably amusing conspiracy theory I concluded that the Phantom Downvoter lurks amongst us, unknown and unseen, waiting until he(*) spots a potential victim thread, at which point every remark in the thread gets mercilessly downvoted over the next 24 hours, although only by one vote as the Phantom Downloader is but a single man(**).

              (*) she, it, they, whatever …

              (**) woman, robot, sentient hive mind, whatever …

              1. Alien8n

                Re: I call bullshit

                I've noticed that as well. I'm wondering if every downvote I've had is actually the same individual every time, silently expressing his displeasure that I corrected his grammar or some such back in 2001.

              2. Roopee

                Re: I call bullshit

                I was sorely tempted to downvote you, but decided it would be unethical (impersonation)!

      2. The Oncoming Scorn Silver badge
        Black Helicopters

        Re: I call bullshit

        I for one would like to hear that story about the constipated 4 star US general and the anti-helicopter missile - Even if only "Second-hand".

        Icon needs no explanation obviously.

        1. Arthur the cat Silver badge

          Re: I call bullshit

          I for one would like to hear that story about the constipated 4 star US general and the anti-helicopter missile

          OK, here goes. It dates back to the early 80s and is second hand so some details are a bit hazy, but the basics are true, even if andy 103 doesn't believe them.

          As the UK is a relatively small and well occupied country and thus not the sort of place to fire off experimental missiles willy nilly, my friend and his colleagues tended to do their testing on US Army facilities in the south western desert, occasionally alongside other firms. IIRC on this particular occasion they were having a first firing of their anti-helicopter missile while another company was having a major demonstration of an anti-tank system, hence the presence of the 4 star general.

          A key point is that unlike the usual heat seeking technique and explosive warhead used by most anti-aircraft missiles, AH missiles work by detecting the radar signature of the rotating blades and simply flying into the rotor to smash the blades off. They're tested by putting a mocked up helicopter on a tall stand a distance away. (You can probably find videos of such tests online.)

          A couple of other key points are that the south western desert can get a tad warm (and for some bureaucratic reason these tests were usually scheduled for the summer), and the test facility latrines were not terribly salubrious at the best of times. Therefore the general had pulled rank and had a spacious and well air conditioned portable latrine block shipped in for his own use, and ordered it sited well away from the normal facilities.

          The test the general was there for was taking a while to set up, so he took the opportunity for what was intended to be a leisurely enjoyment of the air conditioned facilities just before my friend's group had their first test. They got the mock helicopter running, checked everything and fired the missile. Unfortunately the radar aperture was fairly wide and it picked up two sets of rotating blades and headed for the nearest, which was of course the AC fans on the general's current place of abode. The missile went straight through the aircon unit and smashed its way out through opposite wall, half demolishing the place. Luckily it was large enough that the general was out of the direct line of flight so avoided a face full of missile exhaust, but when he appeared from the wreckage in an understandably irate state he was a little smokey. Much shouting and blame storming ensued which took a while to sort out. However, one of the general's staff had a quiet amused word with the British team to tell them that since the incident the general had at least stopped complaining about his constipation.

          1. ma1010

            Re: I call bullshit

            I have a story from (most likely) the same testing facility. An engineer fired some sort of shoulder-launched heat-seeker at a target. Something went wrong with the missile as it flew: it fell, hit the ground, then bounced back up into the air, flipped over and came straight for him.

            After it flipped over, the hottest thing the missile "saw" was the launcher tube. Fortunately, the tester was an engineer and realized what was happening, so he threw the tube as far as he could and hit the ground - and lived to tell the story. (Fortunately, the test missile didn't have an explosive warhead.)

    3. Anonymous Coward
      Anonymous Coward

      Re: I call bullshit

      If you've worked in any reasonably busy environment for some years, you will have seen numerous examples of this sort of thing, and may even have caused a few yourself. I'd imagine that El Reg's postbag is bursting at the seams with examples people send in, and the ones published are those that are most impressive/alarming.

      Just off the top of my head, I can think of a couple that have happened in my workplace (hence the AC post here...) that I have seen first hand:

      • A colleague doing support on the client's network and when they were done, rather than disconnecting their RDP session, accidentally shutting down the client's server (due to the version of windows at the time having the shut down and log off options next to each other, and the client not having the forethought to remove the shut down option)
      • Another colleague wanting to delete a row from a client's database, and forgetting the WHERE clause (goodbye whole table). An abject lesson in taking backups, and of running your DML statement as a SELECT first to sanity check the results.

      Thankfully, I've never done anything to accidentally destroy a client's data or interfere with their operations, but this is mostly because as a dev, I'm protected by a support desk who do most of the donkey work. Never assume that you can't make a mistake yourself though, a cock-up is just a slip of the keyboard away.

      1. Robert Carnegie Silver badge

        That's not the word.

        I was briefly intrigued but... "object lesson", please. A particular example that demonstrates a general principle, i.e. make sure that you have backups, undos, and other protections of the production estate from prestidigimistakemaking. Now THERE is a word (nearly).

      2. Anonymous Coward
        Anonymous Coward

        Re: I call bullshit

        Hmm, if the table held card payments then we may have worked together...

      3. Anonymous Coward
        Anonymous Coward

        Re: I call bullshit

        I know someone who did a very similar trick, first ever Citrix server logged in as full admin and did a shutdown rather than log off, I don’t think it was in production at the time.

        In the first week at a company I caught the hair trigger (less than 1mm of movement) bypass switch on the ups and powered down the comms room.

        And a colleague who demonstrated an as400 command line with pwrdwnsys *immed and pressed enter......

        Oh and I turned on vtp on a nexus 7k and then spend the next 12 hours resetting an entire site overnight....

        1. A.P. Veening Silver badge

          Re: I call bullshit

          And a colleague who demonstrated an as400 command line with pwrdwnsys *immed and pressed enter.

          (Un)fortunately, that command now needs some special authority in the latest versions of OS/400, it happened too often.

      4. mr_souter_Working

        Re: I call bullshit

        "A colleague doing support on the client's network and when they were done, rather than disconnecting their RDP session, accidentally shutting down the client's server (due to the version of windows at the time having the shut down and log off options next to each other, and the client not having the forethought to remove the shut down option)"

        Been there - done that - cue an immediate call to the customer to let them know that their server was about to shut down, and if he wouldn't mind very kindly toddling over to it and pressing the power button.

        very small company - only a couple of staff, and the servers were in the MD's office - he was fine with it, after all, we all make mistakes from time to time - and was very happy with the service that we provided.

    4. swm Silver badge

      Re: I call bullshit

      I heard of an 8-bit microprocessor controlling conveyor belts at a strip mine. The machine dropped a bit and one of the conveyor belts stopped. Another belt didn't and overflowed the stopped belt. When large thumps were heard coming from the parking lot they discovered the problem and several scrap cars.

      The first landing on the moon was almost aborted during the final descent when the guidance computer ran out of real time. The error was eventually traced to a mis-thrown switch whose position was specified incorrectly on the check list.

      The initial ARPANET had an error/misconception about routing and caused the entire network to crash hard on a single bit error. The network could not be restarted until new routing code was loaded into all of the IMPs.

      At the end of WW2 a soldier fired a bazooka at a pile of ammonium nitrate to start it burning. It didn't, but rather exploded, making a huge crater.

      Some mistakes are less visible than others.

    5. Will Godfrey Silver badge
      Facepalm

      Re: I call bullshit

      Well, if you think that's bullshit how about this then.

      In the middle 1960s the Prudential Assurance Company's entire data processing system was transferred from punched cards to an IBM360. A single machine with no backup facilities, and it remained like that for about 6 months.

      And the reason for no backup? Not in that year's budget!

      I have no qualms about mentioning it, because the person overseeing the transfer (very much under protest) was my dad - who is long since beyond any kind of retribution.

    6. Doctor Syntax Silver badge

      Re: I call bullshit

      "Well, I do, and haven't."

      Find a dictionary.

      Look up "hubris".

      1. Kiwi
        Pint

        Re: I call bullshit

        "Well, I do, and haven't."

        Find a dictionary.

        Look up "hubris".

        I'd suggest adding 'karma' to the list as well... :)

        1. A.P. Veening Silver badge

          Re: I call bullshit

          I don't have to look up Karma, that lovely bitch is sleeping in my garden at the moment ;)

    7. Anonymous Coward
      Anonymous Coward

      Re: I call bullshit

      I work with major clients at the moment that have no test/dev environments - government organisations that really should know better.

      Any changes that we make have to be tested in live production environments that hold extremely critical data.

      the real world is a bloody scary place, full of stupid decisions that bite people on the arse more often than you might imagine.

    8. Mystic Megabyte
      FAIL

      Re: I call bullshit

      "That's like saying an air traffic controller gave the wrong instructions to an aircraft because their watch broke and they didn't know the exact time."

      I can't find a link to the original story but the last paragraph here will give you the idea.

      "When the two planes collided, both the main and the backup telephone were out of order, radar software displaying flight coordinates was in a restricted mode and Nielsen’s only colleague was on a coffee break."

      https://www.reuters.com/article/idUSL12514386

  13. Interested Party

    So close...

    Surely the tag line should have been as below?

    "Having bank problems? I feel bad for you son: I've got 25 million problems, but a BATCH ain't one"

  14. codejunky Silver badge

    Ouch

    Never for a bank but wherever I work I insist on 2 copies of the system. One live and one dev and both as close to identical as possible. That way there is no switching of environments, just use the right one.

    1. Sgt_Oddball Silver badge

      Re: Ouch

      We tried that then had to make a fbb* to remind users and testers (for they were often the same) which system was which.

      There's nothing quite like the feeling in the pit of your stomach when something goes wrong because someone else buggered around with your work and you just know they're going to shift the blame to you.

      On the plus side speaking clearly, and with a sense of authority that you understand the exact nature of the problem also helps greatly I've found.

      * something... Big Banner

      1. mr_souter_Working

        Re: Ouch

        "We tried that then had to make a fbb* to remind users and testers (for they were often the same) which system was which"

        I've tended to use different coloured backgrounds - often with a large note on them, If I am allowed to configure it. but then again, there are plenty of times I've logged on to a system, and then spent 20 minutes trying to work out if I am on the test or live systems.....

        1. J. Cook Silver badge

          Re: Ouch

          We've gone further and air-gapped our test lab. when we need to move a machine from the lab to production (or vice-versa) we clone it to a small NAS appliance (think a Synology or QNAP 4 bay device), and hand-carry the data to where it's going.

          It largely works, although we do have some vendors that are cranky about how they get remote access to the lab. (it's via a IP KVM unit and a handful of desktops cabled up to it; the KVM is on the production network, the desktops are on the lab network.)

  15. Simon Millard

    Remeber Y2K?

    In 1999 I worked for a big blue bank in the UK and was involved in the periphery of Y2K testing.

    We'd set up a ring fenced environment to do our testing.

    It wasn't ring fenced enough. When testing a remote restart processes, the port the instruction went out to was missed in the RFE and as the test machine was an exact copy, it shut down the live service as well! Oops.

  16. JimC

    Very similar

    A *very* large employer in southern england. The days in which payroll transfers were done by a couriered data tape. I wasn't involved, but as I recall I heard it the tape wouldn't pass QA our end and so changes were made. The tape was duly couriered over. The tape wouldn't run their end. Changes were made. The tape ran beautifully. It paid the entire payroll into a single building society account, presumably Mr Aaron A Aardvaak. This was spotted. Discussions were had. Building Society took the credit of Mr Aardvaaks account (so he never saw the money) but refused to pay the money back until all is were dotted, all ts crossed. Well this was a *very* significant amount of money. So no money. The employer had to borrow enough money for the entire payroll at emergency rates in order to get everyone paid on time. Did I mention this was one of the biggest payrolls in SE England. Most expensive cockup in the history of the organisation. Never got to the bottom of quite who's fault it was. I always suspected it might have been deliverate on the part of techies both sides not to apportion blame in case it came their way.

  17. Anonymous Coward
    Anonymous Coward

    Who needs test data?

    Many, many years ago when working for a very large building society on a night shift in operations...

    The task to process the mortgage tape was due. So we started the batch process as normal and it finished as normal. Unfortunately it was not ticked off on the process sheet as completed so when half the shift went to lunch, the other half processed the mortgage tape.

    Mistakes like that are not noticed until the branches open the next day.

    Posted anonymous for very obvious reasons. Human oversight just helped the computer to process those million transactions twice.

    1. Gordon 10 Silver badge

      Re: Who needs test data?

      Exactly! In your face Andy103!

      1. Anonymous Coward
        Anonymous Coward

        Re: Who needs test data?

        agreed - used to work for a financial company - many the time that users entered incorrect clearing dates for import batch routines - or the clearing house messed up and sent the previous days file (forget that there was a user balance check to validate that it was correct) - which then needed to be reversed. It was so bad that I actually programmed a reverse batch function so that it could be cleared without needing to touch the database or call IT...as far as I know they're still using it....

  18. earl grey
    Devil

    Wait, you have savings?

    How the hell did that happen?

    1. Kiwi
      Coat

      Re: Wait, you have savings?

      Wait, you have savings?

      How the hell did that happen?

      I actually have a very active "Freeflow savings account".

      Money automatically flows in each paycheque. It freely flows out again well before the next paycheque.

      The only "savings" about it is it saves me having to say 'I'm completely broke" today - at least till I get the money transferred out of it.

      1. A.P. Veening Silver badge

        Re: Wait, you have savings?

        In other words, at the end of your money you still have a piece of month left over.

  19. tcmonkey

    If you think this sort of issue could only happen in the 80s, boy have I got news for you! The year that a major financial institution doesn’t stuff up and charge customers multiple times is a very bloody quiet year indeed, let me tell you.

  20. Anonymous Coward
    Anonymous Coward

    Payday anyone?

    I remember the day when I uploaded a banks Payroll twice...

    then the manager told me to reverse the entries by running the process twice (with numbers reversed...) so...

    Twice credited, then four times debited, don't ask...eventually we did three more credit runs to get everything back to norm...

    All in one nights work...

    Had to go to CEO to explain what happened who shrugged it off as a bad night. Nice...but trying to explain to the employees what happened and the sorry state of their monthly statement was a very different story...

    1. TomPhan

      Re: Payday anyone?

      I've known some people who being told to run the process again but with the numbers reversed would credit the accounts again, changing £12345 to £54321

  21. Paul 195
    Big Brother

    Times have changed

    Back in the good old 80s, there was no GDPR, and not even a Data Commissioner, and it really was the wild west. It wasn't uncommon to create test data by taking a subset of live data. After all, what was the worst that could happen?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021