Congestion or a Christmas cock-up? A Register reader throws himself under the bus Let's take a step back in time for today's Who, Me? with a trip to the dying days of manual credit card imprinting. John, our latest confessor, was working on one of the UK's very first Electronic Funds Transfer at Point Of Sale systems (EFTPOS) during the early 1990s. His customer was a then-major electrical retailer, with a …
"The retailer's network guys were ecstatic about how fast their network was, and convinced the suits that the incorrect timeout was an honest mistake, couldn't have been foreseen etc."Even better, they were fine with deploying a patch that included the proper timeout, despite the fact that it was during the pre-Christmas lockdown."
Looks like they deserve some Credit (Card)
I remember putting in a change request to fix a performance problem. It went through all the reviews, and as I was the expert they accepted my comments.
It took a week to implement, and after the IPL the performance was worse! I tracked it down and raised a problem saying the person who misconfigured it needed to be shot!
Of course that was me... the change needed to be at both ends of a connection (or to be more accurate, both ends of all connections). I learned to be a little more careful when saying 'execute the guilty'.
This sounds so familiar. Been there, done that too...
Nowadays there are only two people messing about on servers I'm responsible for, me and my not-so-PFY. At a stretch, a sysadmin for a sister service might be drafted in. But it's usually me who's cocked it up.
Mea culpa, mea maxima culpa.
... OS/2 is alive and well and still doing good and useful work, world-wide. You can purchase a brand new, shiny license to run OS/2, complete with support for (some) modern hardware. And telephone support. I've used two different versions in various places over the last couple years ... Serenity Systems has sold eComStation since 2001, and Arca Noae LLC has sold ArcaOS since 2017. Both with IBM's blessings. Wiki for more (and links). Recommended.
All this IBM talk reminds me of when I was looking through an IBM repair manual (I forget what the device was), but one of the trouble-shooting steps was to replace the AMD. I had to look that up in the glossary - it was short for an Air Movement Device, which normal people would call a 'fan'.
Another IBM part name: IIRC the early PS/2 machines had the power switch round the back, but people complained to IBM about this, and the later versions appeared to have the power switch at the front. In fact the real switch was still at the back, and the control on the front of the machine merely moved a steel rod which poked the real switch. That rod was called the "ferric power transfer bar".
Reminds me that, among IBM business partners, the joke went that if IBM had to market sushi, it would be as IBM Cold Dead Fish.
I never found out whether it was real or a pisstake that people were spotted at an IBM conference wearing IBM Certified Website Navigator t-shirts, given how legendarily simple and navigable IBM.com wasn't.
Honestly though, it's a coin toss as to whether you will get roasted or not.
In this case it was about money, so there may well have been someone up the chain blowing his lid and getting all vocal about it. The fact that the hierarchy chose to look at the positive side, aka network performance, is just a stroke of luck.
As I read it the “hierarchy” wasn’t involved. Basically someone in development double checked before “escalating” to operations. Ops appreciated said behaviour and took dev’s back. And thus it should be always. Unfortunately, especially in larger outfits, the blame game is too often played even within IT departments.
In such cases, the simple question "would you rather I hadn't admitted and mentioned it?" usually suffices.
And if it doesn't, then it's time to consider whether the manager has ever made a mistake, and whether they admitted it or were just found out. Those are the ones to avoid working for whenever possible.
A lesson from outside IT. If there's a quantity decision always have a backstop person who can look over your shoulder and say "Did you really mean 300 sacks of manure". A (further) sign of a crap manager is that they won't allow an underling to check their data is correctly entered into an order or report.
Any single point of failure decision needs error checking- we all make errors.
When ordering a lot of small parts make absolutely certain that the vendor actually means "per 1000" and hasn't helpfully modernized the use of "M" to mean one million. If you don't, you'll wind up with an argument over who owns 10,000,000 3" nursery pots ... I got the shipment stopped at about 1,000,000 delivered or in transit. We settled out of court, with me purchasing the shipped lot at manufacturing cost. I'm still using them about 15 years later ... the vendor told me later that it took them 18 months to shift the other 9,000,000.
If you want manure, come and get it. 30 sacks, 300 sacks, 3,000 sacks ... it's all the same to me, but I recommend a truck, not sacks. You load.
No, not terracotta. Cheap, black injection molded plastic. Some folks call 'em 4 inch pots. They are 3.25 inches square at the top, and 2.75 inches square at the bottom. 18 fit into a standard 10x20 tray. It was the first year I sold heirloom tomato and hot chili starts. It's quite profitable, if you don't mind getting your hands dirty ... and you can figure out how to automate most of the repetitive work.
Just don't purchase a million pots when you only want ten thousand :-)
Back in the distant past, when I was an apprentice at Ferranti Computer Systems, one dept I was seconded to (SSG I think) they had boxes and boxes of very wide rainbow ribbon cable all over the place. It seems somebody had needed 100mm for a project, but had ordered (from RS!) 100 boxes (each of many meters.)
I'm still chipping lengths off the portion I was allowed to take, nearly 40 years later..
It was regarded as a universal panacea in the Mechanical's development department of London Underground too, when I worked there.
We also had a few Microprocessor ICE test systems too, where a high-spec "umbilical cord" (looked like a braided hi-spec ribbon cable IIRC) sat between the cpu and its socket, in order to do real-time debugging, hence ICE - In-Circuit Emulation. Holy grail stuff, when one had worked out what was, and what wasn't possible to emulate.
The problem often occurs when the fingers are still typing one thing, but the brain has moved on to something else, especially if this has been triggered by an overlap or association. e.g. You're ordering 3 sacks of manure - but the number (3) reminds you that you need to get £300 from the cash machine. Your conscious brain wanted you to type 3, but your fingers have picked up the thought "300" lying just below the conscious level and are now following that thought instead.
A company for which I worked needed some ball bearings to fix a ballrace in a specialist camera. The purchasing department was given a requisition for "12 .3 inch ball bearings, surface accuracy 1/10th of a thou". The failure to write the decimal as '0.3' resulted in loss of the decimal point, which led to the delivery of a dozen 3 inch steel balls, with an amazing surface accuracy, and an astronomical cost.
The supplier wouldn't take them back, so they were raffled off as 'decorator items' to the staff.
A neighbour of my parents decided to enlarge his brick driveway. However he did not realise that when the house was built, imperial units were used, so when he ordered new bricks they came in metric sizes, and he had to have the entire driveway redone, at considerable extra cost.
This post has been deleted by its author
Back in the 1970s, my late father was a school headmaster (hence the icon) in Leicester. At the time, there was a considerable debate about the merits or otherwise of comprehensive education. And there was a strange situation in the local education authorities.
The City of Leicester had its own council and education authority, which was Labour-controlled and was one of the last hold-outs to retain grammar schools (against the Labour party's national policy). The County of Leicestershire (Conservative controlled) was one of the first to introduce comprehensive schools (against their national policy). And then the two authorities (together with a third authority covering Rutland) were merged.
Before the merger, supplies for City schools were organised by a lady with a spiral-bound notebook. If a school phoned her, she could immediately tell them how much budget they had spent, and how much budget they had committed but not yet spent (and hence how much budget they had remaining). The County schools used the newer computer system at County Hall.
Custom and practice differed between schools in the two (former) authorities. County schools were in the habit of ordering supplies "little and often", whereas City schools tended to order a complete year's supply in one go. So the first time an order from a City school bubbled to the top of the queue, it cleaned out the supply chain. So for several weeks no school supplies were available at all.
The other issue was that the two (former) authorities used different units for supply orders. In particular, County schools ordered pencils by the number of individual pencils while City schools ordered by the number of boxes. Similarly, City schools ordered paper by the number of sheets, while County schools ordered paper by the number of reams. And so it didn't take very long for a City school to put an order in (failing to read the instructions carefully enough), and for the County Hall supplies people to fail to sanity-check the results. The school duly received a delivery of a hall-full of paper and one pencil.
"Fortunately they were sequential so it was a little easier to search through."
For me it was a Texaco station, also in the 70s. We pretty much ignored the list because it took far too much time back in the days of Full Service. (Oil, water, air, windshield, brakes, power steering, automatic trans, battery if asked, no extra charge for topped up fluids ... and occasionally freebee new wiper blades. The good old days were, in fact, good in this case!)
Fill 'er with Ethyl.
I remember working at a catalog showroom in the 70s. We had those nice, well sorted booklets too. But I think those came out monthly, and each week we'd get an update list: page long columns of numbers in what must have been chronological order. Attempting to check numbers against that part was painful. My favorite days were those when the new booklet arrived!
We also had to verify checks over a certain amount - by phoning the bank. Because this took time, we were allowed to choose which large checks to validate this way. After a few somewhat emotional situations, I decided to only check the folks who looked liked like they were going to be able to pay. I learned a lot of about human nature in that job - it seemed like I was finding more rubber checks that way than just validating every 3rd or 4th check...
"The number of typos in printed material are going up too.
And it's weird stuff, like viscera being swapped to vitriol..."
That's not typoes, that's the evil monstrosity erroneously known as auto-correction.
"And don't get me started on the improper usage of inflammable"
That's just corporate ignorance, apathy and illiteracy.
If the same happened today, would the suits allow such a quick patch and without thorough regression testing? You'd have to watch things like components of the stack having been updated since the last time you compiled the code, potential for new bugs or security holes.
The same shouldn't happen today, the value for that time out shouldn't be hardcoded but loaded at runtime together with a lot of other "constants".
As a PM I regularly have to ask for changes to be implemented during change freezes for my projects to continue on schedule. A change to a new server during a year end change freeze which isn't going to affect the rest of the estate is easy to get authorized. A change to a a feed into the corporate SAP system due to a year end freeze is (rightly) significantly harder. The should always be an exception change process during a freeze and in fact an emergency change process to allow business critical incidents to be resolved such as a national energy gateway accidentally changing the interface spec for registering new clients. I was able to push through a change to meet the new file format in an hour so we kept registering new customers, knowing that in a few days time we'd probably need to revert the change when The portal was fixed.
I have even owned up before my cockup was even found.
I was once tasked with improving security in our product. Got the requirements and produced a design which was approved. Implemented the code which passed the code review and went on to pass test.
Anyway, for some reason I was looking at the design probably patting myself on the back for what a good job I had done. That’s when I saw it, a massive great security hole so bloody obvious a 12-year-old would spot it. Worse still it could not be fixed the design was fatally flawed and would never work.
This was pre agile days, so our release cycle was yearly not every time you take a dump, so the feature had not been released. So, cap in hand I had to tell the project lead I had dropped a bollock and the changes needed to be rolled back. He agreed as bollocks go it was a big one. Next stop we are in the development managers office explaining why we are removing the feature.
Code rolled back but no repercussions not even ‘how the hell did you miss that’. Probably because both the project lead and manager were two of the design reviewers and the project lead reviewed the code.
Well one repercussion I was not give the job of producing a new design or code it. :)
I remember those days, way before I landed in tech. Looked up a customer's credit card on the naughty list printed every month on newsprint in a booklet format. Found a match. Called it in. They offered $50 for collecting the card from the customer (for context, minimum wage at the time was $3.35/hr). The card in question happened to belong to a loosely stated buddy of mine.
He surrendered the card and we split the reward.
Sadly, it wasn't me but someone in the Off License (booze shop for those on the other side of the pond) spotted someone being a bit shifty with their card and did a call to check it. As soon as he picked up the phone the shopper legged it leaving a wallet full of plastic. The colleague collected a couple of hundred pounds as I remember from all the stolen cards.
I owned a few clothing shops in the time of "call over $X for authorization". There was also a "code 10" authorization - if you thought the circumstances or the card presenter looked dodgy, you could call for authorization and say "code 10" which alerted them that something seemed off. you either 1. got the authorization code or 2. the card was declined but you gave it back to the customer or 3. you were told to keep the card if you could without endangering yourself. One lucrative year, I got enough from turning in dodgy cards just myself (if my employees caught it, they got the dosh) to mostly finance a nice vacation to Cozumel.
I was IT manager at a tourist attraction in the north of England and I was tasked with keeping the POS POS system going (the first POS does not stand for point of sale)
It was written as a Foxbase system and when I arrived the database was stored on a Windows 95 PC whith file sharing enabled - supporting about 25 POS terminals running on DOS 3.2. After migrating the database to a proper server, I mostly kept the thing running apart from a few odd foibles where I ahd worked out problems and could cure them by tweaking of the back end data using MSAccess as a data entry tool - the management were very cheap!
One day I was looking through the logs for the credit card processing system (Barclays PDQ IIRC) and noticed something very strange. One customer had received a refund of about £2.5K to their card. When charging them for entrance, taking the money on their card, one of the front desk staff had apparently manqaged to drop something on the keypad, entering a negative amount. There was no front end validation to prevent this from happening.
We had to contact our bank, who then had to contact the customer's bank and ask them to contact the lady, asking if we could have our money back. She kindly agreed after a week or two, and we ended up giving her a year's free entry to placate her (and she didn't get charged for the visit when the cock-up occurred).
The software house got a bollocking and rolled out a fix ASAP.
I’ve done the MS Access/VB Classic Integer 2 byte database autoincrement field (mis)assignment (32,767) instead of 4 byte long Integer (2 billion). Everything works up to the 32,767 record; then everything dies - Usually after the system has been running OK for a while and the punter has signed it off…
... to promptly admit when I've made a mistake. Even better to personally report the mistake before anyone else finds out it exists. Extra bonus points for completing that report with "...and I've already fixed it."
Then when I say "looks like the error occurred at [x] step" (which meant someone else was to blame), it's nearly always taken at face value.
When reviewing someone else's design or code, I try to follow my PhD supervisor's lead and point at wherever the mistake is and say "I don't understand that bit". This allows the miscreant, sorry, highly regarded IT professional with much experience, to avoid embarrassment by saying 'well, so-and-so couldn't understand it as originally written so I've done it simpler way now'.
I once bought petrol at a garage where they still used the old embossed sheets and signatures for credit card transactions. Presented with the chit to sign I crossed through the empty cells to the left of the charge and the garage owner was most insulted. "If I wanted to rip you off I wouldn't do it like that" he said to me. I just smiled all innocent like. I heard of a chap who bought a hand made suit in Hong Kong, had signed the chit and the tailor had added a "1" to the left of the price. He did not notice until he got home and got the bill ... One thing those EFTPOS terminals do is prevent the vendor changing the price charged.
Also, throw out SQL and go within something No-SQL because complete data validation and integrity is for dinosaurs.
Not that No-SQL solutions don't have their place, and a hybrid SQL/No-SQL solution could work very well for some things, but pretending that columns and data structures are an unnecessary inconvenience is just reckless.
When someone screws up, as long as it isn't something tooooo drastic, if you 'fess up as soon as possible after you realise then they're generally OK about it. This has happenned twice to me, once I accidentally deleted the global security database in ISE - the response I got from that Infrastructure was 'Hey Fred (name changed) - you know that button in ISE that you should never press..." (I reckon they were just grateful it wasn't one of them who pressed it). Second time I was trying to remove an Azure group from a user who'd left, managed that bit OK, but I removed the group from everybody. As soon as I spotted that one I found the person responsible and told them, in both cases some minor disruption but not as bad as it would have been if I kept schtum (and tbh they'd have found out who it was from the logs anyways). Also in both cases it led to improvements in procedures that benefitted all of us.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
