Dear Stupid, I write with news I did not check the content of the [Name] field before sending this letter Ah, gentle readerfolk, welcome once again to Who, Me? in which Reg readers much like yourselves regale us weekly with tales of technical disasters of their own making narrowly averted – and sometimes not averted at all. This week meet a reader we'll Regomize as "Stan" who once worked as the subscriptions and IT manager (an odd …
No, that would be : Ah, a while back I decided to create a situation that could possibly destroy the company since I didn't think for a second about any possible consequences of my decisions.
It's called Data Management for a reason. And doofuses like that are the ones who enabled the book to be written on the subject.
I've told this story a couple of times, so apologies if you've read this, but..
A few years ago, we had a need for an equipment and inventory tracking system. None of the existing commercial solutions quite fitted out requirements, at least not for the prices our management was willing to pay. The problem was that we needed users to be able to book some of the equipment, and any equipment worth over £500 required the user to fill out a risk assessment, and get it approved by a manager outside our team. The booking system needed to be able to manage all communication with the approver and user. We did not know about any bookings until it was approved.
So, my boss initiated a project where we would build our own. It wasn't a massive system (although it was much more heavily used than we anticipated, loaning more than 100,000 items and taking hundreds of bookings. Thankfully, we'd worked out our requirements for hardware resources, and doubled them, to enable the system to expand, so the system coped well, mostly. (there was one area where it was too slow initially, but we resolved that).
There were three of us on the team. One colleague designed the backend stuff. Basically a Java Web service connected to an SQL server based Database, and a small Java based application to manage it. I designed the booking website (for the users) and designed some administration pages that were accessed from a central Administration website, that managed various systems we used at the time.
I can do basic web design, but am not good graphically, so my other colleague did the graphic design for both sites, and I wrote the HTML/Javascript required to implement functionality for them.
It worked brilliantly (mostly, there were users who had bypassed the rules before that complained they were suddenly forced to follow them, and there was the aforementioned slow part we corrected eventually) until one day when our counter staff suddenly reported it wasn't allowing them to loan or return anything. Then I started to get notifications from our bookings website that it was failing (I'd built error checking into the code that emailed me any errors as and when they occured).
The emails told me that calls to the webservice were failing, and the webservice was reporting it could not find the transactions table. The way the system worked, every action we, or a user, performed generated a transaction. This was done deliberately, to ensure the system could be audited, and for debugging purposes..
The colleague who designed the webservice opened up SQL Manager. At this point, my colleague who designed the UI disappeared saying he had meetings all afternoon, so needed to go for lunch, and vanished.
It was then we discovered that somehow, he had renamed the transactions table as a full stop. Not sure how he did this, because I am certain SQL Manager does not allow that as a table name (although I don't currently have access to an SQL Server to try it).
My manager didn't take any action against him because all three of us had access to the database, and it could not be conclusively proved who did it, at least not the way that server was set up. So, after the manager give a stern talking to to all of us, our DBA removed both my access, and the access of the technician who had broken it. In fairness, we did not need direct access to the database anyway, so that worked out better, IMO. I'm quite happy at work to not have access to stuff I don't need access to. I can't be held responsible when something goes wrong..
Absolutely as it should be everywhere. Minimum privileges for the required task.
Not "We gave everyone domain admin access to make this application work. Don't worry though, it isn't a security issue as we didn't tell the users." Yes, I have actually had this one and of course, the real fix was a simple permissions change. That client was a "special" case in so many ways though.
A few weeks ago a pole outside our residence demolished by a speeding car - the pole was splinters as I imagine was the car.
After the mess was cleared away the next morning, the two things left were two metal digits from the pole number that had been flung off the pole by the impact.
On the ground where they lay they formed the number 42 :) which I took to be rather auspicious.
A long time ago in broadcasting I heard a tale about a radio station using the RadioText feature on DAB. Now whilst you can do artist and song in RDS I believe this was the station’s first use of it, and it was on DAB. So the playout system was spewing the metadata out of which track was playing etc. the problem was they hadn’t checked all the fields the system was drawing from. They also hadn’t looked at the scheduling database since moving from CD to hard disk. Nor had they checked what fields were being used from it to send the text. Artist 2 being a field used for comments on the version of the song, the CD state etc.
As a result certain songs had some very interesting metadata attached which duly appeared on listeners radios. So comments like “Is the radio friendly version with the F*ck bleeped out at the start.” Or “F’ing thing skipped on Saturday when last played” to “live version, recorded before she stopped being able to sing in tune” etc. I’m told not a UK radio station but I can’t be certain.
It's about incentives.
The regulators like the threat being available - makes the reprobates fight less, and results in less push back from politicos saying "you're endangering jobs in my region" or "you just fined my mate". They get to say "we know, so we low-balled the fine".
The CEO who just walked into the lariat of GDPR gets to say "yes we got hit, but due to my Leadership and Negotiation, we rolled with it" and that might save his job.
You know if anyone ever eats a big fine, that's someone who's got no important friends left.
(and for companies that haven't been hit yet, and don't know how bad it won't be - the unrealised threat is still there, hovering menacingly. I've seen plenty of internal discussions at my place end in "it can go up to 4% of turnover. We _cannot_ risk this.")
> The CEO who just walked into the lariat of GDPR gets to say "yes we got hit, but due to my Leadership and Negotiation, we rolled with it" and that might save his job.
This is akin to what I've been seeing from several major tech company CEO's lately, as they have announced substantial layoffs;
- "I take full responsibility for the decisions that has lead us here" (Sundar Pinchai, Google) or
- "[I take...] full accountability for the moves that got us here" (Daniel Ek, Spotify).
If you're so hell-bent on whole-heartedly accepting the blame, how come YOU get to keep your job while you're letting thousands of the people doing the actual work go?
Database Master? Now, there's a Master Database... which would be a problem if the CFO went messing around in that.
But not GDPR bad.
If they had System Admin access, or they were Database Owner of a database with sensitive information... that would be a potential GDPR issue.
Yes, I know... I'm a pedant. But I'm a DBA so these things matter :p
I've seen worse.
A database with several dozen tables, but no foreign keys. Table relations were managed in code, not enforced by the database. Some tables had in excess of a hundred fields, several of which were duplicates of each other, and which one of those duplicates actually got the data depended on which bit of code was filling it in.
There was actually a function in code called "check if column exists and if not create it", which might have gone some way to answer why there was a "color", "colour" and "coloour" field.
Even worse, significant bits of that database had been duplicated out to other systems.
Despite all of this, the company in question did pretty good business and their system worked pretty reliably.
>A CFO with database master access.
Well what do you expect when you sell tools such as dBase, Paradox, FileMakere, Base and Access as part of desktop Office suites and being suitable for users to build DIY databases. Wouldn't be surprised if the original address database was based on a 1990's hands-on/getting started PC magazine article...
I used to work at a company, where an important tool for the finance team was an Access DB, that pulled data from the actual accounting system, and spat it out into an Excel spreadsheet. The only person who understood any of it was the CFO (other staff having moved on), and they only understood about 85% of it.
None the less, it was deemed 'business critical', so I had to occasionally wave a dead chicken at it to keep it running. Keeping it under the 2GB limit for Access DBs (iirc) was the main problem.
As the lowly IT serf I couldn't force it's replacement, and as far as I know it's still an integral part of the finance systems.
I used to work at a place where Filemaker had quite a good workout - with a key one having been built by the definitely not too technical FD.
Eventually, it became my responsibility to work with it and I spotted some interesting things. I spent a few minutes working out WTF a fairly long string of functions performed and next time I saw her commented along the lines of "Filemaker has a 'round up' function you know". It din't go down well !
I used to work for a major telco and my job at the time was to give customers tours of a fancy showcasing setup where we talk about how wonderful the place was. When a salesperson has a customer group ready for a tour, we ask them to give us names for name badges, and whether each individual was an advocate, a detractor (of us, said telco), if they were a decision maker, an influencer or the like.
One day, I did a tour and the receptionist of the showcase was, as always sent this list. This particular customer group turned up with names including “Influencer” “Detractor” “lEconomic Buyer” “Big Boss” and “Subordinate”. I saw the “names” after I commenced the tour. By this stage it was too late.
I had to think of a way to sort this out. I texted a colleague and said “bring a six pack of beer, and a bunch of post it notes with “influencer” “detractor” “big boss” etc on it. At the end of the tour I showed the customers that we were having a lucky draw to win a six pack of beer, and if your “random title” gets drawn, you win. It was Detractor’s lucky day that day.
I worked on a system for a Housing Association which was set up to encourage tenants to pay their rent in time. If their account was up to date at 1st December they would be sent a prepaid Visa card loaded with £25 which they could then use in any shop at Christmas. I was tasked with setting up a mailmerge from their data to send out a letter to prompt them to clear any arrears and inform them of the scheme. I sense checked the data and found quite a few instances where the tenant had obviously died and they noted this by tacking the word "Deceased" onto the surname. I managed to warn the client before they got lots of complaints from people who got letters for their dead relatives.
I used to work at a pension company. One of the fields on the customer data screen was called "Alias", which was always left blank. One day, after a heated complaint from a Welsh customer, one of the call centre users decided to type something derogatory into the Alias field for that customer. He thought it was funny that when you looked up the customer details on the system it displayed that alias. What he didn't know was that the intended use of the field was for performers etc. You have to have their real name on the contact, but all their mail gets addressed to the alias. So a few days later, the customer got a letter addressed to "Dear Taffy Bastard".
I am unfortunate enough to have an account with British Gas for my electricity supply (?)
Suddenly, last December, I received the third of my £67 vouchers, but addressed to Mrs. Pauline Purvis, someone of whom I have no knowledge. I phoned the helpline on another matter, and asked why my account had been changed. They replied that they had no idea why, but they knew exactly when the change had taken place, and promised that they would take steps to correct the mistake.
I have since received the fourth voucher, still addressed to Pauline, whoever she may be.
Good luck getting anything corrected by British Gas!
They screwed up my gas readings last year, eventually got it straitened out only for them to do a 6 month "summary" statement that had all the (same) wrong readings.
As it is in my favour I am not going to go through that hell to correct it again.
When I were a lad at Uni I was living in Yorkshire and my electricity was billed by the local firm. They wouldn’t accept me as a customer because I had no credit rating so my dad signed as the customer. Come the end of my 3rd year and I’m leaving, sent final meter reading paid the bill and thought no more of it. That was until my dad got sent a bill for 26p, well almost sent to him. It was actually addressed to Mr Previous Occupant, at his address with the supply listed correctly as Coalminers Cottage, The Lane etc. He said phone them and explain they’ve already spent more than that on the stamp. So I duly did and the bloke I spoke to was unhappy that I was refusing to pay the remaining balance.
Two days later another letter/bill arrived and I called them again this time I spoke to a lady, i thanked her for the second letter as she was finding the details on the system. She said that the notes on the account mentioned that we were refusing to pay an I said yes….and we’re prepared to fight this in court. Then she saw the amount, burst out laughing and said she was terminating the balance as they had wasted more than double that on postage. She then said “Thank you for your call Mr Previous……do you mind telling me your actual name please?” We heard nothing more after that.
I heard a tale of a patient going to a doctor's office. They asked what the patient wanted to be called, and the patient flippantly replied "The Chosen One." The patient didn't think anything more of it, until at a much later visit, they overheard one office member say to another, "Please inform Dr. ___ that The Chosen One has arrived."
...of the one where office admins see a bunch of "unused" fields in the database so decide they can keep notes in them. "I've been looking for somewhere to keep the reminder that this shithead always makes a fuss if we don't have the right kind of tea on hand whenever he comes in to give a lecture - hey look, this 'accessibility' field never has anything in it, I'll put it in there!"
Next thing you know the mailshot with this weeks events goes out and at the bottom there's a footer: "Students with accessibility issues, please note: Get twatface his stupid poncy tea."
Sounds like databases need a field explicitly labelled: "Offensive notes".
Many years ago, doctors used secret acronyms for things that it would be awkward to explain to patients. CAAC was "Crazy as a coot".
Several dictionaries and acronym finders seem to think CGSM is real. I thought it was fiction. Anyone know for sure?
Whilst working in the Civil Service a while ago, I can lay claim to having invented the COSCC system (Circular Object Supply Chain Codes). These indicated the likelihood of acquiring a round tuit, from imminent to possibly never. It wasn't actually *used* (as far as I know), just an exercise in fun.
AC, just in case anybody remembers.
Late to the party but that reminds me of the US military, who spent a lot of time partiucularly in Korea/Vietnam era (although much later too IIRC) with quartermaster-types shipping non-existent products to each other when they'd lost items (from a box of pens, to a tank or a couple of Jeeps). This was necessary as there was no formal acceptable way to state that a box of items had been misplaced or stolen without it being a big problem.
The game was to trade them off sufficiently so they could be "used", noting that one unit probably wasn't going to use a shipment of ~10 Jeep axles, but 10 different units could reliably have used them, or until they ended up at a unit that did a lot of flying. If the flying unit had a flight crash, it was an opportunity to posthumously load a large quantity of missing items on the flight and claim them as lost also. I recall a story of one largely empty Huey/C130(?) that had been recorded at something like 8x maximum-weight when it went down, such was the load of "items" it was carrying.
I remember that one from a stint working with medics. There was a common one that showed up in the post-birth notes "FLK" because the delivery room staff just thought the Kid Looked Funny and should be checked over by a doctor just in case. The follow up was often "FLP NFA" after the doctor met the parents.
"Funny Looking Parent. No Further Action."
I worked in a place with a call centre, taking calls from people with problems. Shortly after I joined, I noticed that customers could be flagged as "special". What it turned out to mean was that they are stroppy bastards but had they ever requested their data under the then DPA, they would've just seen "special".
Apparently this sort of thing happens from time to time.
I was starting to get a little bit worried at having to scroll this far down the page before seeing reference to this old classic!
(OK, it was actually chronologically an early comment (Whew! I was pretty much expecting it to be the first one!), but it has only appeared way down the page for some reason…)
I went for a job interview in the 90's with a small UK-based company that made a bespoke database that was used for direct marketing, and the interviewer spent a lot of the interview telling me anecdotes.
One was "Dear Rich Bastard" (told in the context of it happening there). I have never known if it was true of not, but the link by Pete 2 suggests it might actually be. It may even have been that company as the description and timing fit.
They also apparently had to fire a developer as he was going through the unanonymised database running queries as his own dating / matchmaking service and making unsolicited contact with the women that fulfilled his criteria. Obviously they started anonymising the test data after that.
(I don't actually expect anyone to believe me, but all of the above is true; I am not making any of it up)
Reminds of the time I once worked for a telecoms company some decades or so ago. There was one lad who was working there while finishing university. He was studying film making and working on his final submission/dissertation which was a film about vaginas. You guessed it. It was a video of various vaginas and yes he brought it to work to show off. We were lucky no one requested a copy of their notes from the accounts he worked. Such classics as "Customer sound like they have had too much coffee and not enough sex" to "My supervisor is a think cunt who has absolutely no idea how to do her job" with lots in between, those are the tame examples. He also had this weird serial killer vibe. I sometimes wonder what he's up to these days.
If he had hair like a long haired monk (the circle at the top missing any hair), was not allowed to drive (for various reasons) and was thrown out of the local 'war games society' (for being too weird even for them), then I know who you are talking about.
Brilliant developer, if you could find the right job to press his buttons.
There is a place for everyone in this world, and a good manager should find it.
Die by the shoddy business practices.
I would have done nothing. If their most valuable asset is in such a shape, they deserved to fall over.
One of my first jobs had a small AT&T phone switch & voicemail (Merlin?) running off a small AT&T branded PC-AT with some proprietary boards. One day it died, and the horrible noises from the hard drive told us why. AT&T support was really insistent on following the script and did not like being informed that we knew why it was dead - the support ticket said "customer thinks he knows more than we do"
Alas, sometimes the customer DOES know more than support. Like me calling because my DSL didn't work - "Both the DSL and Internet lights are off on the modem." Their reply: "Have you rebooted your computer?" Or very recently, support asked "Have you tried a different outlet?" Me: "It's plugged into a UPS. Nice, clean power 100% of the time." Them: "Try a different outlet. It could be an outlet problem."
I know your pain. When my ADSL modem died a few years ago, I was unable to convince my ISP's tech support that a problem with the line would not explain why it wasn't able to route local traffic - or why the power light wasn't coming on any more. It took a technician visit (who confirmed that yes, the line was fine) before they would accept that the modem needed to be replaced.
Stupid little estate agent, half-assed database in FoxPro (the DOS one). Fields for title, name, surname. Title was supposed to be Mr or Mrs (etc).
Scarily many had commentary like Twat, Pisshead, Eunuch, Wanker...
...and or transpired (from copies of letters in the archive) that letters had actually been sent out formatted as <name> <surname>, <title> so it was probably no surprise that the return on letters sent was abysmally low.
Company owner? Narcissistic megalomaniac with so little talent he failed at door to door sales. Suffice to say, I didn't hang around.
We had a vendor supplied database with an insert-only comments table. No one knows why but the design was PrimaryKey, ForeignKey, DateTime1, Comments1, DateTime2, Comments2. The insert code would (randomly?) pick either Comments1 or Comments2, the other one was always null. The table worked fine for display in the UI, but any reporting query of the type (where comments='foo') performed very poorly. That didn't stop the business from "extending" the application functionality by having the users put in the comments: a prefix code followed by the relevant data. Once word got out on this genius method everybody was doing it. That's a terrible idea even with a properly designed comments table, but needless to say expressions of the form (where case when comments1 is null then (case for comments2) when comments1 like 'foo%' then comments1 end is not null) are the opposite of sargable. The application ground to a halt but rather than telling everyone to stop the madness we just replaced the system with another vendor's. Out of the frying pan into the fire.
You never know when it will get into production by mistake – I’ve heard of instance where instead of setting up clean database for client, someone just copied and “cleaned” a test one to setup a telephone booking system, trouble is the cleaning missed some audio clips and when CEO of client was first to use the telephone booking – over the speaker phone came “This is sh1te, I’ve no idea why you’d want to ...”
Typical old app written by tech enthusiasts with self-taught PHP and SQL skills:
There's a CUSTOMER table with personal information. Do you use that? No. Data quality is poor because a customer may be multiple people. It's old.
That can join to a USER_INFO table with PI. Do you use that? No, somebody read that the contact information should be normalized to a new table.
That can join to a USER_ADDRESS table with PI. Do you use that? Yes, but not everything does. Check the USER_INFO and CUSTOMER tables too.
<desk flip>
This post has been deleted by its author
Having worked for a Stockbrokers, I can confirm that some of the dumbest ideas come from the coloured crayon department. Once they got us to export the whole database to a csv so they could use it for a mailmerge, but they neglected to remove all the "Deceased" entries before running their merge. Over the next few weeks, there were multiple phone calls from outraged clients who had received letters addressed to the dear departed.
When this was raised at the board level, the IT manager repeated that he had warned that most of the disabled accounts were for deceased clients but this had been ignored because, "there are lots of accounts that are disabled simply because the client has gone elsewhere".
It took me 6 months to get Spectrum (Charter) to stop sending me mail. Apparently opt-outs require the name of the person the mail is attached to - at which point they put the name of a different current or former resident on the mailing instead of stopping. Per the Spectrum representative responding to my BBB complaint, the marketing department said this was to prevent loss of marketing opportunities.
Guess what company I'll only do business with if they are the one and only ISP that is physically capable of getting internet to my house?
I think 'Stan' has some culpability here. He inserted rows into a table without running the same SELECT into a table on screen first, for at least a glance and maybe a quick chat with the boss? "Because he told you to," is just not an acceptable excuse. Stan was working at a trade publication, not a Minuteman silo.
FileBreaker is the most egregious piece of software unleashed on business owners and people that fancied themselves IT people. It allowed amazing, epic, disastrous collections of data to accrete over time and be leveraged wtih the most horrible network and sharing hacks I have ever seen added to a product over time. It ended with the most inefficient, horrific results that never scale over time. I battled one such business owner for years, as the business grew, I tried to point out that it is not a real RDBMS and would not scale and it did not. Every year the seasonal company had to start over with a fresh, empty database and as the season moved along, it got slower and slower with more and more complaints. ISP bandwidth and RAM thrown at it did not help. God help them if they are still using it with someone at the helm that poo-pooed relational databases.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
