Having bank problems? I feel bad for you son: I've got 25 million problems, but a bulk upload ain't one Sunday is gone and Monday is here. To ring in the week, please join us in welcoming the latest addition to the shedload of shame that is The Register's Who, Me? column. Today's toe-curler comes from a reader we shall refer to as "Geoff" who, as a newly qualified COBOL and Assembly-language programmer, was working for what he …
Nope, definitely wouldn't get that in the current climate.
Wouldn't get any of the major UK banks using live data for testing, no Sir.
And definitely wouldn't get any of the major UK banks hosting their test systems on their Production network without isolation from Production. No Sir, wouldn't happen.
#sarcasm
Bit like me in the mid-80's. First time on night shift, get told to run the End Of Day processing by taking 'Option 1' - the pre EOD backup. Unfortunately the terminal I used (conveniently already logged on), had a different Option 1. Their's was to purge that days transactions. Whoops. And management still made us run the EOD without any transactions (cos they're stupid). The following day, the input girls had to enter the transactions again, we ran the EOD again (6 hours), then that days transactions were entered (overtime here we come) and that days EOD was run. Funnily enough, training was provided after that and procedures implemented !!!
I was once doing some electrical work in my grandparents house when my grandmother came in and told me the hoover wasn't working. I explained that it was because I'd turned the power off, and it would be off for another ten minutes.
Ten minutes later I (and the opinel knife I was using) were flung across the room. Moments later, my grandmother appeared to tell me the hoover had stopped working again.
I kept that knife for years. The brown streak of copper wire that had plated itself next to the missing chunk of steel seemed an apt reminder not to give people short timescales...
LOTO indeed - for on-site engineering it is standard practice since whenever, and very rigorously enforced at good sites: you don't call up your mate in the control room to cut the power, you go in person, isolate, put your padlock on, keep the key. When you've finished, roll call, back to the control room and unlock. Some places everyone in the team has an individual lock & has to be there in person to lock and unlock.
A bit of crypto wizardry & the IT equivalent should be trivial. But we'd rather a quick bodge job.
I don't know about trivial - the devil's in the implementation - but achievable, certainly. You'd probably want a N-of-M quorum scheme or some kind of key escrow in case the locker forgot his passphrase or was struck by a meteorite on lunch break or something.
"Indeed - LOTO (lock-out, tag-out) is your friend..."
Don't be so sure!! When I was just starting out as an electrical apprentice in the 1990's I went to the stores with another newbie to get two padlocks (one for each of us) so we could work with the 1500v overhead line safely using the locking out system. I tried my key in my colleagues padlock and it successfully opened!! So much for THAT :-D
Not surprised. I started in production mainframe testing a very long time ago. So we all had our own toolboxes with a padlock. It was a new production site so all the toolboxes and padlock were bought at the same time. Turned out most people could unlock most other peoples padlocks. There were a lucky few that had padlocks that were safe from most of the other people.
Oh and then there are US built RVs (campers). Turns out 60% of the outside storage compartment locks are keyed to "CH751" so all the keys and locks are the same. One of my locks just failed so I will probably switch them all. None are storage compartments but still, it's the principle.
CH751 also happens to be the key used by a certain 3 letter ATM manufacturer to lock numerous ATM's rear access console (note, not the safe - just the rear console) - but still :-)
I remember the first time I realized that my mailbox key could be used to open the ATM console...
There's production padlocks... and there's test padlocks... (That's not meant to be taken seriously BTW).
===
Many years' ago I devised a system to automate the registration of codes that are used in a safety-lock system for a well-known client in that field. Certain codes could not be used as it would weaken the structure of the lock. Pythagoras came in handy for this purpose, and so I can say that the maths I did at school was not in vain (belated thank you to Messrs Hogarth and Kelly).
Lock-out breakers are required for at least some appliance circuits in the US, if there isn't a cut-off switch that's within view of the appliance. At least that used to be the NLB requirement - it's probably changed in the years since I last looked at it.
Basically, the idea was if the installer / maintenance person couldn't see that no one had turned the power back on, he (or she) needed assurance that it wouldn't be. And that's with little ol' 110V US circuits. With higher voltages I'd want lock-out regardless.
Basically, the idea was if the installer / maintenance person couldn't see that no one had turned the power back on, he (or she) needed assurance that it wouldn't be. And that's with little ol' 110V US circuits. With higher voltages I'd want lock-out regardless.
A cheap hasp and lock on your fuse box door? Easy to break in an emergency, but family should know not to damage it unless it's a life/death emergency.
>These days, you can get locks to put on the breaker and/or isolator.
Spent all night hunting a vacuum leak and left the pumps running to pump down.
Carefully locked out the telescope with the official lock out hasp and padlock, took the key, wrote it up in the engineering log and went to bed.
The day shift had to override the lockouts to move the telescope because "they couldn't find me"
Arrived to a vacuum pump dangling 20m above the ground attached to an instrument that was 3 years of work for a whole team.
Idiot proof is easy, bloody stupid proof is harder.
A stern post-it note over the switch helps too. But depends on who finds the note, and whether they like you or not.
Interestingly, there are locks you can get for them. Google "MCB toggle lock". I agree, it's nowhere near as handy as pocketing the fuse, but safer in that someone can't put a spare fuse in. Or a nail.
Yep that situation sounds quite familiar. I'm sure electricity can't be as dangerous as it's made out, I know a large number of people who have had big shocks and waked it off.
I've had a few close encounters and a mate nearly threw himself down the stairs when he was working on a lighton the landing and trusted his (now wife) gf to turn the power off.
I'm sure electricity can't be as dangerous as it's made out, I know a large number of people who have had big shocks and waked it off.
When I was a kid my uncle had a shop selling TVs and radios and offering repair services. One of the repair men was known for his trick of stripping the insulation off wires with his teeth, until the inevitable day came when he failed to spot the other end was plugged into the mains. He came to about 10 minutes later, on the other side of the room from where he'd been, and with a head sized and shaped dent in the plaster on the wall. Afterwards he used wire strippers like the other repair men.
Icon as he probably looked like that for a brief moment.
My second job was an apprentice TV engineer. When we were overworked they'd ask in a guy who'd retired several years earlier to come and help out. He had a habit of checking whether something was live or not by deliberately sticking his fingers on or in it (light bulb sockets were a common thing he'd test in that way). He seemed to be immune to electric shocks or maybe was so used to them that they no longer bothered him!
The issue isn't with short shocks. Had a few in my years, got thrown across a room when the back of the plug separated while trying to pull it out, and a screwdriver in a light fitting not realising the pull switch was on (socket was broken and I was replacing it).
Where it does become dangerous though is if you grip the wires. I remember an incident with a teacher at school once who managed to grab both ends of the wire in separate hands, the resultant electric surge through his body then caused his fingers to clamp shut on the wires making it impossible for him to drop the wires. I believe this was solved by someone turning off the socket that the device was ultimately plugged into, resultant damage was burning to his hands and a new healthier respect for electricity for everyone concerned.
" I'm sure electricity can't be as dangerous as it's made out"
In school, my friend and I used to enjoy sticking a nail in the earth hole to open a socket up and then putting another nail in the live one. Felt like getting your thumb hit by a hammer.
Then we came up with the bright idea of doing this with two sockets:
- Stick nail in earth hole to open both up
- Hold plastic end of 2 opened compasses
- Stick in live holes
- Turn compasses so they touch
- Big bang!
- Electricity trips out in that block of the school
- Teacher tells us we're bloody idiots and lucky to be alive
We use colour coded padlocks..... even then I'm sometimes forced to remove the RCB at the main distribution board... and lock that.
Can I turn the power on? comes the question, and I dont know why they bother because whatever version of 'no fucking way mate' I answer , it seems to translate to "Yes go ahead" inside their brains....
Indeed.
A friend of mine working on a large development in central London had a rather interesting experience.
He actually padlocked the cabinet closed and placarded after isolating the 3 phase supply. Went up two floors to continue installing busbars and starting termination. About 20 Min later he dropped a bar bolt. Which apparently then proceeded to short the bars with much banging, flashing and smoke.
Rather rattled (possible understatement warning) he flew back down to the plant room to find the Architect, site manager and owner had cropped the lock off so they could look at the lighting on a finished floor.
There was a subsequent "free and Frank expression of views and opinions management".
Followed by the culprits having a rather large whipround.
According to him he made more in 5 minutes than he normally made in a month as a third year apprentice.
He still talks about it 30 years later
Dave
ahh yes, I have a huge screwdriver with a bit of foreign metal fused forever on its tip.
in the 60's, living in the south (north carolina), my wife and I scraped and saved up to buy a window air conditioning unit.
The day arrived... we went to Sears, laid down our hard earned cash, rushed back to the apartment. After getting the unit secured in the window frame, looked down and saw that the wall outlet was different. I went down to the basement, unscrewed the fuse for that circuit, went to the hardware store and returned with the proper outlet. down on my hands and knees, removed the cover plate, popped the first wire off, went for the second one... Blinding flash, very loud pop... I am now sitting on my ass two feet away, with my rather large screwdriver welded to the socket.
Wife comes in and asks "Did you turn the power off just now?"
Moral: take little round fuse with you, til you are ready.
yep, still use the screwdriver, could clean up the spot weld I suppose, but it is a reminder of "bad things happen"
Also reminds me of the induction/site orientation training for one of our customers (global multinational semiconductor manufacturer) who is very big on safety that I had a few years back. Up comes the various slides, unless the training lady gets to the one about LOTO (lock-out / tag-out).
I looks at this, and looks around the room to see if anyone else has spotted the issue. Apparently not, so I ask the trainer "what's actually wrong with that image?". She looks and ponders for a moment, then says she can't see anything. At this point I point out that her slide seems to show someone has actually managed to lock-out a breaker in the ON position rather than the OFF one (which I didn't think possible, but they seemed to have managed it). Cue her going white, before commenting that she's been using these slides for years and no-one has spotted that until now.
Had a colleague have to do the training recently and asked him to check that slide out for me. Seems that the image is still in it, but it's now a question posed of "what's wrong with this image" asked to the trainees...
Had a colleague have to do the training recently and asked him to check that slide out for me. Seems that the image is still in it, but it's now a question posed of "what's wrong with this image" asked to the trainees...
That is the correct solution, must be a qualified and experienced teacher.
"what's wrong with this image"
Reminds me of the recent story about a Health and Safety course at the University of South Wales*:
University of South Wales sorry over health and safety course
To quote: a lecturer got "very basic scientific information" wrong - for example he claimed that bleach was an acid when it's an alkaline, says the Times.
He also said that "voltage" was named after Voltaire, the French philosopher - when it's in fact named after the Italian physicist Alessandro Volta.
The inquiry found the lecturer, who was teaching safety and business risk modules, suggested that oil could be heated to 360C - when it can actually catch fire at 250C.
It's claimed he also told students that the "most important thing" they had to do in the workplace was to "keep your job and not be prosecuted".
M.
*Slight declaration of interest - the University of South Wales used to be called the University of Glamorgan (with a glorious sign on the A470 which pointed the way to the "University of Glam" - wish I'd been able to get a photograph) and before that it was the Polytechnic of Wales which I attended in the 1980s and 1990s. Before that it was the South Wales Mining School and there was still a mock-up "mine" under one of the buildings when I was there.
Of course, it is a vastly different institution these days, but sadly it seems that the quality of teaching perhaps doesn't live up to the quality of all the nice new campuses.
One of the kitchen sockets blew in my new apartment
Turned of the breaker and, because Mrs Coward didn't have any stupid children, plugged a lamp in to the socket to check it was off.
Took the cover off and got zapped.
Turns out in Merkin houses the top and bottom sockets on a double outlet are on different circuits, turning off the top one leaves the bottom one live....
Fortunately their electricity is like their beer so only a little tingle instead of the, "fsck me" 240V arm tingler.
"Turns out in Merkin houses the top and bottom sockets on a double outlet are on different circuits, turning off the top one leaves the bottom one live...."
Well, each outlet can be independently wired. It lets you control one outlet with a switch while leaving the other hot, useful for lighting. Different circuits, though? That's diabolical.
Slow is right.
Takes 5 minutes to boil a full kettle's worth (1.7l) - because that's how big my tea mug is. (every time I've made an FCC submission over the last 2.5 years, I include a photo of my drinking from it, from my bigger, better insulated and non-corporate freebie mug, just to maybe annoy Pai)
And yes, the 110 does feel like a tingle, like you've just almost hit your funny bone. now the 230 from a dual phase socket (dryer, AC unit, cooker etc) will make me jump a little. (I'm used to it, from being drunk every single afternoon lab session through uni, at the electrical engineer and electronics department at liverpool uni, which is literally across the road from the union.
"the top and bottom sockets on a double outlet are on different circuits"
The instructions for leccy work of one of my customers state very clearly:
... every electric circuit shall be proved dead at the point of work, just before starting work
Saved me more than once, together with my paranoia.
Years ago I was going to replace some socket outlets for a friend who'd moved into an oldish house. Turned all the circuits off at the consumer unit/switchboard. Checked outlet: no voltage between the contacts. Being paranoid I also checked each contact to earth - tester lights up. Turns out some *##@! had wired the consumer unit up with the single pole MCBs in the neutral rather than the live conductor! So 230 wiggly volts still on all circuits :(
Another time I was going to replace a switch, late Friday afternoon. Turned the power off. Took the cover of the switch and tested it - still live. Then realised I was tired and had only thought about turning the power off but not actually done it.
This post has been deleted by its author
My grandfather, back in the bad old days of the 1950s, used to check all his workers' tool boxes, and if necessary insulate the shafts of their screwdrivers and the handles of pliers himself to make sure it was done properly.
He taught me to work on circuits as if they were live "because one day they will be".
This was in the context of signalling systems where it sometimes became necessary to fix a loose or broken contact, and you can't stop the East Coast main line while you do it.
He taught me to work on circuits as if they were live "because one day they will be".
Lovely fellow that.
I was taught the same way.
Has probably saved me a little bit of hassle. Especially in a place where the wiring was, well, lets just say "incapable of passing any level of safety inspection". Treating everything as live - even when the breakers and building mains were off - meant I only got to cuss about how messed up the site was and suggest we talk with the powerco about a larger level of disconnect than getting taken home in a nice wooden box.
Me mum helped with some of my training as well.. We had a dead TV and I'd been working on it (still only a kid at the time, but had been building and fixing circuits enough to be confident of solving this). I got up and left the room. Mum came in, saw me gone, wondered if I'd been successful, plugged in the TV, left the room. I came back.. "Oh yes, must check that". Touched something live. Picked myself up off the opposite wall, then went and had a little discussion with my mother about not plugging stuff back in without warning people. (given that 240v can kill easily enough, she was a tad upset she'd nearly killed her kid).
From work days, I remember the tale of the company that occupied two adjacent buildings. It was sold off as two going concerns. Each building had its own switchboard, meters and so on.
Came the day when, as part of the sale, an electrician turned up to check the wiring. He turned off the main breaker to his building, and half the lights went out.
The wiring was so confused that it more or less had to be redone from scratch in places. As stuff had been added, holes had been made in the wall and things wired across. It was possible that some machinery was getting two phases from one switchboard and the third phase from another.
In another company sale, power was turned off while the building was checked, and at length someone arrived from the golf club half a mile away to ask if they too had a power cut. The former owner, a keen golfer, had diverted the firm's supply to the clubhouse.
I used to carry a pair of pliers with a notch melted into the cutting section as a reminder to always double (or triple) check I had isolated the correct circuit. I eventually replaced it with one that was more effective at cutting - and with better insulation - but only after the lesson was well hammered home.
Surely everyone knows that you always advise of at least twice as much downtime as you need?
That way, you can either look supremely l33t when you get the job done in less time, or still have some time to back out or fix in the vanishingly unlikely event that something may have gone wrong with your carefully laid plans «coughs».
But, yes, make sure that the person in charge of the "switch back to normal" switch (or asking the keeper of the switch to turn it) is you, and nobody else!
Or better still in this case no time frame should have been given. He should have said I will call you when I am finished with testing.
As for the guy who waited 10 minutes before switching it back to live, I'd say he has some of the blame here as taking things literally like that is either not caring, dumbness or malevolence. If you know a test is being carried out, assuming that it's definitely going to be all over within the given time frame on such a critical system is pretty stupid. Nothing at all would have been hurt by the guy picking up the phone and checking if all was clear.
I would go so far as to say "most of the blame".
Even if he had monitored the transaction and seen that it had finished, he wouldn't have known whether Geoff had actually finished his testing. So an approach of "it's been 12 minutes and I haven't heard, let's phone and check" would've avoided a whole heap of hassle.
Say it will take longer than it actually will, makes you look better when you do it in less time.
I often did that but still went to time. Just wanted to be sure I had plenty of time up my sleeve for unexpecteds.
I did find that a customer will be exceedingly pissed off if you go over a year long job by merely a few seconds, but will be barely happy if you complete a multi-decade job in a few days.
But it is much nicer to say "we're done ahead of schedule" instead of "I'm sorry we're late, we're going to need more time".
If only I could've managed to get co-workers etc to promise long rather than try to please the customer and promise short - so many promises at 10am of "Oh yes, we can scan your 4TB drive and have it all done and back to you by lunch time" (NOT exaggerating! It happened at least once - 2 hours to recover a large HDD!). I always doubled then tripled expected timeframes with HDD issues.
This is great. Insights into cockups in a consumer-facing biz, where we can all see ourselves as potential victims!
Obviously this comes from a different era, when there was inadequate separation of testing and production systems. But ... hang on ... a 56k modem dates it to not so very long ago: if I'd read this story when those were current I'd've thought the same - surely such a story must be from the already-distant past ...
It may be not that long since you owned a 56k modem.
Then again it is almost 20 years since I had 128k to my home office using two bonded 64k ISDN links.
Major computer centres with very expensive 56k T1 leased lines would be perhaps a couple of decades earlier, minimum? Banks spend money. They have a lot of it (mainly ours).
56K leased lines have been available since the 1970's, although I'm not sure if they are still available today given the availability of alternatives and things like ISDN/E1/T1 being decommissioned and replaced by DSL/Ethernet.
56K likely means it was a US-based telecommunications system based around 56K timeslots/T.1 rather than a European system based around 64K timeslots/E.1.
You will be thinking of 56K modems which were available for communication via PSTN lines (V.90/K56/V.92) from the late 1990's.
56k modems of the non dial-up have been around for decades. I remember visiting a bank's data centre in the early 80s that had a GPO (BT) 48kbps V.35 analogue wideband modem that was fed by special cable (co-ax?) to the nearest trunk unit where it replaced 12 analogue voice channels on a trunk circuit. It must have cost an absolute fortune to run, but obviously worth the money. (I think the rest of their branch network was running 1200bps, but then everything was character-based)
Mention BNC...Ruin my Monday..
Back to therapy!
I'll join ya.
Just the other day I came across an 8-bit ISA LAN card with a BNC connector as it's only connector (save for the ISA).
Think I'll need some IPA to forget the ISA, and a whole load of therapy to forget the associated 'network' that went with it :(
Years used to be so long and progress so quick in the late 90s. I distinctly remember having a 56k modem and a large phone bill for ages and ages, but in hindsight, it can't have been more than 3 years., since in 2000 the student housing was renovated and we got 100 MBit(!) connectivity.
That same summer, I was able to build a Duron 600 with a giant 15 GB IBM Deathstar HDD to replace the old Pentium... it's been all downhill from there, I'm sure.
Now that TSB is reducing the interest rate on its current account, I no longer really have any reason to keep my "backup current account" with them. I am wondering how many customers were still remaining with them for that sole same reason, and whether they are now starting to see a final exodus?
Their newish and infamous internet banking site still really is shonky: alarming pauses when moving from one page to the next (I have a nasty feeling that rather far too much of it is 'powered' by dodgy JavaScript, rather than server-side), and it seems to lose track of where it was far too often, and then requires you to login again.
It turns out that I can now beat the interest income on my account just by taking the switching bonus from another bank, so that's what I'm doing.
Early signs are also not encouraging: the application form crashed out with an error message on final submission (but later turned out to have been successful anyway), and the website has some rather flakey and indeed almost phishy-looking grammar, but once I've taken their money, I'll just move my account again to a more reliable bank, if there actually is another one, that is (my main current account provider is very reliable, but almost all of the other banks seem to have flaws of some kind!).
This post has been deleted by its author
Reminds me of the time I tried to help a young programmer (ex-YTS trainee, but with a lot of natural talent) simplify an accounting system he was writing. To save needing two separate routines for credits and debits, I recommended simply multiplying by a value which would be either -1 for debits or +1 for credits, the rest of the calculation being identical. Unfortunately his maths knowledge wasn't great and he didn't trust my solution.
25 years later, he's now CEO of the same firm (its direct descendant, to be exact).
Icon: closest thing to an old fart I could find.
An old IT manager at one place I used to work for (no longer there, strangely enough) wrote a debit/credit system to handle partial payments to customer accounts. Needless to say he f$&ked it up royally to the point that any payment not an exact amount of the bill due resulted in random debits and credits being applied, often for pennies here and there, but in some cases for thousands of pounds. The transactions were all for less than £15 (the monthly subscription the customer was paying).
Eventually got to the bottom of it and once fixed the thousands of pounds that customers actually owed suddenly became known to the client. I had great pleasure ripping into his "expertise" in the disciplinary meeting as he blamed me for his cock up. Said IT manager was the same one who claimed he did 6 degree courses in 18 months (all concurrently), was an Olympic archer, classically trained pianist, friend of Richard Branson, owned a Cray supercomputer that he kept in his barn, ran his own multi-million dollar business selling data to the oil industry, and yet couldn't understand the maths behind calculating VAT.
> Said IT manager was the same one who claimed he did 6 degree courses in 18 months (all concurrently), was an Olympic archer, classically trained pianist, friend of Richard Branson, owned a Cray supercomputer that he kept in his barn, ran his own multi-million dollar business selling data to the oil industry, and yet couldn't understand the maths behind calculating VAT.
He wasn't called Jake, by any chance?
Not that appreciated in banking transactions, though, although I would personally appreciate a fully worked through undo function (could bugger up quite a lot f crime as well :) ). The amount of hassle people have to go through to undo misallocated funds is painful. No doubt that is to teach them to never do that again, but to err is human. To bill for it is banking.
Even worse, I'd say there must be multiple readers and commenters here who have done that. Most probably won't admit it, but I can as I used that to repair some manager's (not mine) screw up. I wrote a program from scratch to create a file with special formatting using the original input file in a couple of hours, made me late for lunch. To make things even more interesting was that the problem was in Prague, while I was working in Amsterdam.
I put something similar together to help a customer migrate to a new printing system. They were used to using the old print submission application where they could set rotations, scale, copies etc... let's just say it was quicker to spend half a day writing a small programme that took the output from the old system and repackaging it so that it would output on the new machine than training several users.
The manufacturer was very interested in my code and kept asking me to implement additional options....
I'm sure there have been occasions when I've seen a trio of entries on my bank statement where a transaction is duplicated, then the duplicate is reversed out. I've never paid much attention because the final balance is correct and the interest implication, if any, is trivial. I'd guess it's probably caused by a duplicated Direct Debit request.
Sorry but every time I read these I think hmm did this really happen? You always get some smart arse replying saying you can't have lived in the real world unless you've experienced this. Well, I do, and haven't. Sure mistakes happen but when you're talking about these environments and such trivial mistakes - time after time - it seems very unlikely. Especially when they all just happen write in to The Reg because they want to discuss it?
Firstly - if it's real - someone name the bank or link to a story about it. If it made headline news then it should be pretty easy to find even if it was decades ago.
Secondly - how the fuck would anyone be employed with that level of responsibility who would let a cock-up like that occur? That's like saying an air traffic controller gave the wrong instructions to an aircraft because their watch broke and they didn't know the exact time. The stories get more bizarre when you're talking about organisations with high levels of responsibility (i.e. banks, airports, etc) which coincide with absolutely trivial errors. Again and again. Really?
Yes people screw up but these just seem like made up stories, as do a lot of the comments where people come out with pub-level bullshit like "that's nothing, back in the 80s I put the wrong floppy disk in and the plant melted down". Did you? Did you really?
My what a narrow world you live in my friend.
Before suggesting I'm narrow minded have you ever considered why:
1. Any time these stories occur people write into The Reg - such that every single week - they can produce a story?
2. People are so willing to discuss them with the World + Dog, even though the stories are always anonymised and about something that the organisation in question would absolutely not want anyone involved talking about?
3. How an organisation has become the "biggest" in their sector, whilst also seemingly having bugger all in place to stop a trivial error? See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.
4. Why you can never find independent and publicly verifiable stories to back any of it up, even when The Reg claims it's a story reported by mainstream media?
These are just a small section of the issues. I don't think it's worth going further if you're too narrow minded to accept some of these stories might just be made up.
I've worked in Banks
Hopefully not in any position of responsibility...
Can I suggest politely that if you wish to post as AC, you remember to do so when you reply?
Well I thought it was hypocritical I was moaning about the stories being anonymised whilst posting anonymously so wanted to pre-empt that. Quite happy to make my point.
I'll gladly be proven wrong if someone can show an independent source that backs the story up.
FTFA: it was "the biggest bank in the country" and the value was in "billions" and "The story even made the front page of the newspapers". So, should be pretty easy to find something that reports a story this big, no?
While being quite a low level bank employee* in NZ I'd get a notification email about once a month that something had gone wrong with the clearing house, typically with multiple deposits. It got sorted in the next processing, so I presumed some sort of batching system.
Generally it was felt that it was better to overpay deposits, since those are often how people are paid, and delaying payment buggers up standing orders, like rent.
For me it meant that I'd get a few calls asking if we were going to sot out their three paycheques, or should they talk to their work, you could tell them that while unfortunately two of them would be gone by tomorrow, the bank had it in hand.
No idea what it looked like on the end of the bank that over paid.
As for the size of it, any bank handling government accounts can have scales of daily balances that are mind boggling. The hot money accounts also move billions in and out of NZ for about eight hours at a time.
That's for a dinky country that gets missed off maps with a few million population. I would expect there to be thousands of banks doing transactions of that size in the world.
* most of the job was tech support for their banking app, with a team of actual bankers we could pass over actual banking stuff. Level 2 phone monkey rather than actual banker. As reflected by my wages :)
1: He never said "narrow minded". Living in a narrow world is very different.
2: These stories happened a long time ago and the reg readership is very large. If you've got 40 years of experience to draw on from a million users, it's entirely possible to get 52 stories a year. And people continue to make cockups today which may well become publishable stories in a few years time.
3: Just because you haven't read something in the media, doesn't mean it hasn't been published. Again, bear in mind that it may have been published 40 years ago in a foreign country.
4: Banks were some of the earliest adopters of computerisation. They had to devise best-practice themselves. A lot of what we consider obvious now wasn't at the time and they will have made many mistakes as part of the learning process. Thankfully the state of the industry has improved and lessons learned spread (largely through stories like this being told to colleagues down the pub) such that major cockups are far less major or frequent now.
And yes, I saw some absolutely shocking practice in the bank I did some work in a long time ago!
a million users ?????
Even Wikipedia says the following of The Reg:
In 2011 it was read daily by over 350,000 users
In October 2013, Alexa reported that the site ranked #3,140
In April 2015 ... the site ranking dropped to #3,430
continues to decline, at 11 April 2016 ... ranking for the site of #4,750
...
By 12 September 2018 ... ranking had continued to fall still further to #7,194.
Oh dear.
A drop in ranking doesn't necessarily reflect readership falls. It's just as likely other sites are increasing readership more rapidly.
Anyway, it's unlikely anyone, even El Reg can quantify the overall man years of experience in the readership but there are most definitely commentards posting here with many decade of life in the IT field. It's approaching 40 years for me so even assuming an average of 20 years across 300,000 users, that gives 6 million years of experience to draw from.
I'd be very surprised if (m)any Reg readers are the sort to have the Alexa toolbar installed in their browsers, and if they also use JavaScript or web bug tracking, a fair proportion of readers will probably block those as well, so I wouldn't really take Alexa rankings particularly seriously…
This really shouldn't need explaining, but when you have an incident such as in this article you want to vent to someone, help anyone else avoid the same mistake, and maybe just tell a good story.
People are willing to discuss because they're anonymised; they wouldn't be doing so if the names were included, and this offers plausible deniability.
I've only recently outlined how to make a change to a testing environment to ensure that its data cannot potentially feed into production if the correct steps aren't followed, because the environments team had decided that non production systems for the Super Special Customer should run on the production system.
Then there's undocumented processes created which go wrong because the team involved didn't fully understand what they're doing, and most importantly didn't run it past anyone else for verification. In one particular case things broke after a year of running this process, because a (legitimate) change was made on the system which violated an (incorrect) assumption when creating the process.
This happens All The Time, although processes are improving. Then the customer gives you grief for development and testing estimates.
See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.
As late as the 2000 or so, I worked with several insurance companies that the idea of having a dev and a production environment was lost on them. We worked on live data. Scary to say the least.
My what a narrow world you live in my friend.Before suggesting I'm narrow minded have you ever considered why:
1. Any time these stories occur people write into The Reg - such that every single week - they can produce a story?
I believe El Reg has at least a couple of dozen readers. Some of us have been in or around IT or engineering or other stuff for decades. Some of us have made or seen mistakes made more than once.
It could be said El Reg is the best IT news site in the business (well, even if just by me :) ) - there is just a slim chance that a few people will congregate here who know of such mistakes being made. And we do have a tendency to blow our own trumpets. Those "atmospheric noise" videos on YT? Just the physical manifestation of an El Reg comments thread.
2. People are so willing to discuss them with the World + Dog, even though the stories are always anonymised and about something that the organisation in question would absolutely not want anyone involved talking about?
I took you up on your challenge and tried to hunt for the case referenced in the article. I quickly gave up because, well, this is NOT a rare thing. This is just one of thousands of results, and I was several pages in before I gave up here.
These sort of errors happen a lot (not just this one but those in other stories). People are right to say you've had a narrow experience of life. A few changes of details to keep things anonymous, but even without that how is a company going to know they're the ones being talked about? These things happen a lot
Just look at the article on BGP taking out Farcebork, clodfool and scamathon. Think that wasn't a bit of 'human error' - or maybe it just didn't happen after all?
3. How an organisation has become the "biggest" in their sector, whilst also seemingly having bugger all in place to stop a trivial error? See also: employing people at multiple levels who lack absolute basic understanding of the difference between testing in a dev/production environment.
Again, these people have been doing this for decades. Some of the people who read here played a big role in developing the internet we know today, and the OS's we use. The idea of separating dev/production - do you have even the slighest inkling of how it came about as a standard?
It's obvious now - and of course now we all have oodles of cash to throw at doubling up on IT and massive numbers of VM's and huge raid arrays.. But way back when, testing had to be done on the same hardware during downtime, and hope like hell backups were there.. Or just be sure you did a good job. I've taken overseas trips on the overtime pay from individual series of tests because they had to be done out of hours, and we only had one plant to test on so that mean working through the night to get things ready, tested, and the results cleaned up and the line back to being ready for the next day's production run. Duplicate kit can be prohibitively expensive (as can mistakes on the only production line in the factory :( )
How do you think locking off and tagging off switch gear came about? IIRC the last straw was when an engineer had her had crushed (fatally) by some robotics, because she was working out of site of anyone else and it was thought she'd left the area (I am happy to be corrected on any of these details), possibly even at a Ford plant or some other US-based car manufacturing plant. A solution that wasn't obvious until enough people died. Other solutions were tried after other incidents, but individual locks for each engineer? This poor lady was the catalyst for something that keeps the rest of us safer.
Some of us have been working far longer than the current standards have been around.
4. Why you can never find independent and publicly verifiable stories to back any of it up, even when The Reg claims it's a story reported by mainstream media?
Take a look at MSM over the last few decades. For this story alone you have thousands of candidates (and I even wrote one very sweary post as, well, if this happened in NZ then I was a victim of the screw up and it actually made quite a mess of my finances that took a long time to clear - yes I was a customer of our largest bank, yes it made front page (and TV) news, and yes it was some filthy scoundrel at a bank double-dipping "accidentally" - but the post was too much even by El Reg standards so I deleted it. And yes, nearly 30 years on I am still very pissed at that person!)
These are just a small section of the issues. I don't think it's worth going further if you're too narrow minded to accept some of these stories might just be made up.
Maybe. Probably. Probably if not 'made up' then at least some alteration of some key facts to keep it anonymous.
I've worked in BanksHopefully not in any position of responsibility...
Actually, I kinda hope with your limited RW experience that you are not in places of any real responsibility. "You're not a real engineer until you've made at least one $50k mistake". With time, you'll make the sort of screwups that teach you how to manage serious problems - maybe a new 'never again' standard will be developed because of you. God willing, you'll survive the experience and in 20 years some 20something will be telling you it was made up, 'how can someone have made such a mistake when we have the Andy103 Law to prevent people doing such things?'
"You're not a real engineer until you've made at least one $50k mistake"
Only $50k? I've known people make £500k mistakes and still somehow keep their jobs. I've also been on the receiving end of some engineer's mistakes and had the pleasure of phoning the CEO on a Sunday to tell him I was shutting down a £50M a day production line until the next day when every single device hit test and started failing because of said engineering cock ups. Not once, but twice.
For those interested: IGBTs that were supposed to be 5V rated. On silicon these should be tested to about 5.2V, the waferfab set their pass limit to 7V because they did such a bad job on them which resulted in a 90% fail rate at final test. The second time the IGBTs were built with a design flaw that meant current leaked across the gates. Ironically the design flaw meant that any that didn't fail the leakage test were nigh on indestructible and would pass avalanche testing at a spec way above anything they were designed for. From memory these ones had about 70% fail rates.
Sorry but every time I read these I think hmm did this really happen?
If you think this story unlikely you should hear some of the ones an ex-RAF friend of mine told about working in the missile division of a major UK deathtech company in the 70s and 80s. The one about the constipated 4 star US general and the anti-helicopter missile always got a laugh.
I recall the extremely dirty dual-fuel engine on a certain British tank, and how the crews would attempt to start it up if a suitable MoD official was standing in the right place to be covered in soot. I've seen it with my own eyes and watched a major trying to control his amusement. Yet other people have told me it couldn't possibly have happened.
I wasn't thinking of andy103. A few years back I noticed that every once in a while an entire thread of comments, none of which were particularly bad or controversial, would receive a single downvote each. As a suitably amusing conspiracy theory I concluded that the Phantom Downvoter lurks amongst us, unknown and unseen, waiting until he(*) spots a potential victim thread, at which point every remark in the thread gets mercilessly downvoted over the next 24 hours, although only by one vote as the Phantom Downloader is but a single man(**).
(*) she, it, they, whatever …
(**) woman, robot, sentient hive mind, whatever …
I for one would like to hear that story about the constipated 4 star US general and the anti-helicopter missile
OK, here goes. It dates back to the early 80s and is second hand so some details are a bit hazy, but the basics are true, even if andy 103 doesn't believe them.
As the UK is a relatively small and well occupied country and thus not the sort of place to fire off experimental missiles willy nilly, my friend and his colleagues tended to do their testing on US Army facilities in the south western desert, occasionally alongside other firms. IIRC on this particular occasion they were having a first firing of their anti-helicopter missile while another company was having a major demonstration of an anti-tank system, hence the presence of the 4 star general.
A key point is that unlike the usual heat seeking technique and explosive warhead used by most anti-aircraft missiles, AH missiles work by detecting the radar signature of the rotating blades and simply flying into the rotor to smash the blades off. They're tested by putting a mocked up helicopter on a tall stand a distance away. (You can probably find videos of such tests online.)
A couple of other key points are that the south western desert can get a tad warm (and for some bureaucratic reason these tests were usually scheduled for the summer), and the test facility latrines were not terribly salubrious at the best of times. Therefore the general had pulled rank and had a spacious and well air conditioned portable latrine block shipped in for his own use, and ordered it sited well away from the normal facilities.
The test the general was there for was taking a while to set up, so he took the opportunity for what was intended to be a leisurely enjoyment of the air conditioned facilities just before my friend's group had their first test. They got the mock helicopter running, checked everything and fired the missile. Unfortunately the radar aperture was fairly wide and it picked up two sets of rotating blades and headed for the nearest, which was of course the AC fans on the general's current place of abode. The missile went straight through the aircon unit and smashed its way out through opposite wall, half demolishing the place. Luckily it was large enough that the general was out of the direct line of flight so avoided a face full of missile exhaust, but when he appeared from the wreckage in an understandably irate state he was a little smokey. Much shouting and blame storming ensued which took a while to sort out. However, one of the general's staff had a quiet amused word with the British team to tell them that since the incident the general had at least stopped complaining about his constipation.
I have a story from (most likely) the same testing facility. An engineer fired some sort of shoulder-launched heat-seeker at a target. Something went wrong with the missile as it flew: it fell, hit the ground, then bounced back up into the air, flipped over and came straight for him.
After it flipped over, the hottest thing the missile "saw" was the launcher tube. Fortunately, the tester was an engineer and realized what was happening, so he threw the tube as far as he could and hit the ground - and lived to tell the story. (Fortunately, the test missile didn't have an explosive warhead.)
If you've worked in any reasonably busy environment for some years, you will have seen numerous examples of this sort of thing, and may even have caused a few yourself. I'd imagine that El Reg's postbag is bursting at the seams with examples people send in, and the ones published are those that are most impressive/alarming.
Just off the top of my head, I can think of a couple that have happened in my workplace (hence the AC post here...) that I have seen first hand:
Thankfully, I've never done anything to accidentally destroy a client's data or interfere with their operations, but this is mostly because as a dev, I'm protected by a support desk who do most of the donkey work. Never assume that you can't make a mistake yourself though, a cock-up is just a slip of the keyboard away.
I know someone who did a very similar trick, first ever Citrix server logged in as full admin and did a shutdown rather than log off, I don’t think it was in production at the time.
In the first week at a company I caught the hair trigger (less than 1mm of movement) bypass switch on the ups and powered down the comms room.
And a colleague who demonstrated an as400 command line with pwrdwnsys *immed and pressed enter......
Oh and I turned on vtp on a nexus 7k and then spend the next 12 hours resetting an entire site overnight....
"A colleague doing support on the client's network and when they were done, rather than disconnecting their RDP session, accidentally shutting down the client's server (due to the version of windows at the time having the shut down and log off options next to each other, and the client not having the forethought to remove the shut down option)"
Been there - done that - cue an immediate call to the customer to let them know that their server was about to shut down, and if he wouldn't mind very kindly toddling over to it and pressing the power button.
very small company - only a couple of staff, and the servers were in the MD's office - he was fine with it, after all, we all make mistakes from time to time - and was very happy with the service that we provided.
I heard of an 8-bit microprocessor controlling conveyor belts at a strip mine. The machine dropped a bit and one of the conveyor belts stopped. Another belt didn't and overflowed the stopped belt. When large thumps were heard coming from the parking lot they discovered the problem and several scrap cars.
The first landing on the moon was almost aborted during the final descent when the guidance computer ran out of real time. The error was eventually traced to a mis-thrown switch whose position was specified incorrectly on the check list.
The initial ARPANET had an error/misconception about routing and caused the entire network to crash hard on a single bit error. The network could not be restarted until new routing code was loaded into all of the IMPs.
At the end of WW2 a soldier fired a bazooka at a pile of ammonium nitrate to start it burning. It didn't, but rather exploded, making a huge crater.
Some mistakes are less visible than others.
Well, if you think that's bullshit how about this then.
In the middle 1960s the Prudential Assurance Company's entire data processing system was transferred from punched cards to an IBM360. A single machine with no backup facilities, and it remained like that for about 6 months.
And the reason for no backup? Not in that year's budget!
I have no qualms about mentioning it, because the person overseeing the transfer (very much under protest) was my dad - who is long since beyond any kind of retribution.
I work with major clients at the moment that have no test/dev environments - government organisations that really should know better.
Any changes that we make have to be tested in live production environments that hold extremely critical data.
the real world is a bloody scary place, full of stupid decisions that bite people on the arse more often than you might imagine.
"That's like saying an air traffic controller gave the wrong instructions to an aircraft because their watch broke and they didn't know the exact time."
I can't find a link to the original story but the last paragraph here will give you the idea.
"When the two planes collided, both the main and the backup telephone were out of order, radar software displaying flight coordinates was in a restricted mode and Nielsen’s only colleague was on a coffee break."
https://www.reuters.com/article/idUSL12514386
We tried that then had to make a fbb* to remind users and testers (for they were often the same) which system was which.
There's nothing quite like the feeling in the pit of your stomach when something goes wrong because someone else buggered around with your work and you just know they're going to shift the blame to you.
On the plus side speaking clearly, and with a sense of authority that you understand the exact nature of the problem also helps greatly I've found.
* something... Big Banner
"We tried that then had to make a fbb* to remind users and testers (for they were often the same) which system was which"
I've tended to use different coloured backgrounds - often with a large note on them, If I am allowed to configure it. but then again, there are plenty of times I've logged on to a system, and then spent 20 minutes trying to work out if I am on the test or live systems.....
We've gone further and air-gapped our test lab. when we need to move a machine from the lab to production (or vice-versa) we clone it to a small NAS appliance (think a Synology or QNAP 4 bay device), and hand-carry the data to where it's going.
It largely works, although we do have some vendors that are cranky about how they get remote access to the lab. (it's via a IP KVM unit and a handful of desktops cabled up to it; the KVM is on the production network, the desktops are on the lab network.)
In 1999 I worked for a big blue bank in the UK and was involved in the periphery of Y2K testing.
We'd set up a ring fenced environment to do our testing.
It wasn't ring fenced enough. When testing a remote restart processes, the port the instruction went out to was missed in the RFE and as the test machine was an exact copy, it shut down the live service as well! Oops.
A *very* large employer in southern england. The days in which payroll transfers were done by a couriered data tape. I wasn't involved, but as I recall I heard it the tape wouldn't pass QA our end and so changes were made. The tape was duly couriered over. The tape wouldn't run their end. Changes were made. The tape ran beautifully. It paid the entire payroll into a single building society account, presumably Mr Aaron A Aardvaak. This was spotted. Discussions were had. Building Society took the credit of Mr Aardvaaks account (so he never saw the money) but refused to pay the money back until all is were dotted, all ts crossed. Well this was a *very* significant amount of money. So no money. The employer had to borrow enough money for the entire payroll at emergency rates in order to get everyone paid on time. Did I mention this was one of the biggest payrolls in SE England. Most expensive cockup in the history of the organisation. Never got to the bottom of quite who's fault it was. I always suspected it might have been deliverate on the part of techies both sides not to apportion blame in case it came their way.
Many, many years ago when working for a very large building society on a night shift in operations...
The task to process the mortgage tape was due. So we started the batch process as normal and it finished as normal. Unfortunately it was not ticked off on the process sheet as completed so when half the shift went to lunch, the other half processed the mortgage tape.
Mistakes like that are not noticed until the branches open the next day.
Posted anonymous for very obvious reasons. Human oversight just helped the computer to process those million transactions twice.
agreed - used to work for a financial company - many the time that users entered incorrect clearing dates for import batch routines - or the clearing house messed up and sent the previous days file (forget that there was a user balance check to validate that it was correct) - which then needed to be reversed. It was so bad that I actually programmed a reverse batch function so that it could be cleared without needing to touch the database or call IT...as far as I know they're still using it....
Wait, you have savings?How the hell did that happen?
I actually have a very active "Freeflow savings account".
Money automatically flows in each paycheque. It freely flows out again well before the next paycheque.
The only "savings" about it is it saves me having to say 'I'm completely broke" today - at least till I get the money transferred out of it.
I remember the day when I uploaded a banks Payroll twice...
then the manager told me to reverse the entries by running the process twice (with numbers reversed...) so...
Twice credited, then four times debited, don't ask...eventually we did three more credit runs to get everything back to norm...
All in one nights work...
Had to go to CEO to explain what happened who shrugged it off as a bad night. Nice...but trying to explain to the employees what happened and the sorry state of their monthly statement was a very different story...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
