The reluctant log trawler: The buck stops with the back-end Everyone's favourite day of the week, Friday, has waddled into view. Grab a steaming mug of schadenfreude and settle down with another tale from those Register readers saddled with the On Call phone. Today's story comes from a reader the Regomiser 9000 has elected to call "Ed" and concerns the fun that comes from bodging …
"I guess he's got his plate full with the IRS."
Nah, My CPA has his hands full with the IRS.
I'm good at some things, other things not so much, and dealing with taxes not at all. Major changes happen every year, and who has time to keep up? So I delegate. it's cheap insurance ... Especially when your holding company owns 18 different businesses, and all of them sell goods & services to each other.
I wasn't complaining about my job, AC. In fact, I love it. Positively thrive on it, in fact. And it's even quite lucrative. Sorry to burst your bubble, bubbie.
Remind me again, where exactly have I said anything about IR35 (except asking you where I've said anything about it)?
The obsessive stalker/fanboi won't answer, of course, because it can't. I'm fairly close to declaring Formosa's Law on this one and ::plink::ing it.
Obviously there are many 'bodges' in all our pasts, but during my time on-call at a financial institution, more often than not it was coding correctly for someone else's half-assed code that had made it to live and, on several occasions, because there was some data missing from a product configuration causing the system to try and do a divide by zero. Sometimes I've just flagged the problem account and product (and checked for others that might fail in that run) for admin to fix the next day, with a note to get the config screen fixed, other times I've fixed the config screen there and then, middle of the night, to prevent zero values being entered. I've often referred the issue back to the original developer, but almost always I ended up being the one who had to fix it because they were now working on 'something new'...
The solution to that problem is booking all code fixes until one year after release on the original project. One or two massive overruns because the original creators aren't "available" for cleaning up their own mess and they will be available next time. And once they know (through harsh experience) they will have to clean up their own mess, they will start writing better code.
That is no problem, new staff you tell they will be responsible for maintenance and bug fixing of their own work the first year after release (before they sign). That should deter the worst offenders, especially when you tell them the release clock start again with every bug fix release.
Yep. I started out in support roles, and once you've graduated past password-reset monkey a good chunk of it is just "do what the developers should have done the in the first place". I've still only worked at one place that had decent integration between the support and development functions and I'm five jobs into my IT working life so far. It was probably so good in part because for most of my tenure there were less than twenty people in the company so you couldn't help but be integrated due to physical proximity.
When we were writing the Dartmouth Time Sharing System there was one programmer that churned out voluminous amounts of buggy code. There was another programmer that didn't like to write code, just find and fix bugs.
When we discovered this we just put the second programmer on the trail of the first and produced lots of debugged code.
To be fair here, I think there's a measure of blame at both sides. The web interface should handle multiple tabs, but similarly the iSeries back-end should validate what it is receiving - after all, someone nefarious could otherwise try and manipulate the submissions from the web form.
Still, Kudos to Ed for his troubleshooting skills for this one.
I couldn't agree more.
Some 20 years ago I've learnt the lesson that the back-end should validate what it receives and even more so when the front end is not under your control. It was the case of an online arts shop where you could simply change the quoted price of any item to whatever you wanted. (No, it was not my mistake and neither did I stock on artwork on the cheap.)
If you write a back-end that trusts input from the front-end, then you might as well just let your users type directly into your database. Actively assuming that untrusted input is hostile should be the default. Untrusted input being anything coming from anywhere that you do not have complete control over yourself (and which is properly encrypted if any sort of comms layer is involved).
I very much agree. I was expecting to read that a user had realized that, by getting a quote with a different ID, they could trick the backend into purchasing stock at a different price and that a heist by a black hat was on the way. Verify all input from users; they are not to be trusted.
"Ed" here
I absolutely agree that the backend should validate what is received from web forms - because anything done in the browser can be manipulated by a clever user. The frontend was a separate java-based system that was managing the web session and performing user input validation. It would construct and send specific requests (like quote request and place trade) over the backend via a messaging layer. The guaranteed price ID didn't pet passed to/from the page content, it was kept in the java environment. If I remember correctly, it was determined that if a data item in the quote response message (from backend to frontend) was blank, the java environment treated it as "unspecified" instead of literal blank and could, in this circumstance, be ignored - leaving whatever the previous value was (from the previous quote) intact
And usually if you code defensively for stupid * users, you've done a sizable chunk of the work to guard against those clever users.
* okay, less stupid and more the type that when you look at what they've actually managed to make the system do, you take them to one side and ask them if they've ever considered a career as a tester
Sessions in web applications are evil. Dictates a bottleneck on the server side with either server affinity or excessive reading of state from persistent storage. Push state to the client, ensuring it cannot be tampered with in any nefarious way. In our apps it's serialised data in Base 64 that's then encrypted - just make sure you only serialise the minimum of data to establish state. Many frameworks support this approach.
Many, many, years ago, I was asked to "white hat" a financial transaction system. (Most definitely not my normal job of assembler programmer!) Multiple video cameras pointed at screen, keyboard and me, institutions representative also watching (actually reading crime novels most of the time), specified account to place syphoned money into, pile of video tapes (that gives the age) collected daily by secure courier, etc.
I only managed to transfer funds twice. The first was very obvious social engineering to get passwords and left an audit trail . The second was using a similar (multiple concurrent transactions) technique to this story (but obviously not web based that long ago). I managed to get the money into the correct account AND my special syphon account without any of the checks and balances noticing that I had 'invented' £10M! It was fixed when I was allowed to try again 2 days later.
I got £1000 (two months wages) as a bonus from my companies owner. I now wonder what percentage of what he received that represented, but I was very happy at the time.
I did some white hat testing back in the early 00's.
"You have a SQL Injection vulnerability in your eShop."
"Not important, it works."
"I could insert orders without payment."
"I don't believe you."
"I could disrupt the site."
"Couldn't happen!"
>clickety<>clickety<
"Hey, where has our site gone?"
"Oh, did I just inject 'DROP DATABASE;'?"
(It was on the test system, but still left the devs a little red faced.)
I'd had access to all the backend source code and had actually listed all the places I could find, where they had unescaped SQL queries, but they just didn't want to listen.
I then tried a few simple things to show them that it was a problem.
In the end, I just decided that a DROP TABLES was the most obvious way of getting my point across. A permanent DOS attack, as it were.
They should have been summarily sacked after the "Not important, it works." line. In many professions, the equivalent attitude would be considered to be criminally negligent.
ANY competent programmer who works with something that talks to a database knows about SQL injection flaws, and anyone who has ever learned anything at all about security knows that injection flaws are consistently number one in the OWASP top ten.
Had a team member who wouldn't use prepared statements, just concatenated user input onto strings to build SQL statements. Despite repeated demonstrations he wouldn't change his ways and management didn't give a shit since he was their most prolific coder - not difficult when he wrote no validation code.
The answer was to wait until a small system entirely written by him for management reporting went into acceptance testing. Then during the testing I did a classic "DROP DATABASE" SQL injection, because he also didn't bother with limiting DB permissions and ran everything as the superuser. There was also no logging in the application, so no blame could be attached to me...
(Edited to note that big_D had the same approach as me).
Well there was the time, around 15 years ago, that the Canadian subsidiary of a rather large engineering company let its domain expire. We had a project with them at the time, so naturally my users had a lot of bounced e-mails. This, of course, was my fault (by definition, anything that goees wrong with e-mail is my fault) and the messages could not wait until the domain registration was fixed.
I added an authoritative zone for the company's .ca domain to our internal DNS, which stopped the questions about when I was going to fix someone else's domain registration.
Posted anonymously on the off-chance that my Reg user name might identify the guilty party (seems unlikely, but best to be safe).
I learnt long ago that trying to do two things in parallel in tabs is *dangerous*.
Bit of a windfall, not sure what to do, so of course I wanted to open one fixed-rate fixed-term saving account, and one variable-rate to spread the risk of inflation.
Filling in many tedious details in two windows next to each other. Disaster; neither opened successfully, but at least neither debited my money either. (TBH it could have been so long ago that "apply online" meant "fill in a form online and we send you forms, you sign and return a cheque" - I can't remember)
At least Amazon seems to get that right these days.... I'd still be scared to try it with real finance.
You're a lot better with money than me. The only time I had a minor windfall it was gone within six months of purchasing Lego and inebriates. I'm still not sure I understand what money is, let alone inflation.
I've similarly learned not to trust multiple tabs/windows on websites in any day or age, though a lot less scarily than your encounter must have been!
Close, ...
Nice try but no.
It's as close as possible with respect to the article, which did not refer to anything but the late 2000s.
The 2000s run a range that goes from the first one of the series (2000) to the last one (2099), after which they become something else. ie: 2100s.
The OP did not make reference to anything in between 2000 and 2099. 8^P.
So, late 2000s is that, late 2000s.
Cheers,
O.
Why do you assume that "the 2000s" by default refers to a century? Because for all you know, it refers to a millennium. The only significant figure there is the 2, so any smaller chunk that still includes multiple years is valid, including 2000-2999, 2000-2099, 2000-2009, or for some pedants the 1-offset century and millennium as well. I choose to believe that this happened in the late 2000s, sometime around the year 2978, but the various changes in human culture since now have made it not as funny. Fortunately, they also invented time travel so someone could report it to us.
"We feel the cause of, and the solution to, this issue is on the iSeries."
This is, of course, code for; "We'd like to sort out the front end, but the webbie types will all leave if they have to fix their shit rather than being off with their underpants outside their trousers fucking up the next release with just announced, bleedin' edge, whizzbang shiny."
If I had a quid for every time...
"Ever found yourself doing some hurried hacking with the "unthinkable" happened? Or been called out to bodge your way around someone else's cock-up? Share you story of unexpected weekend working with an email to On Call. ®"
s/with/when/;
Well, yes, actually.
This sounds a lot like working (with|for|despite|in utter exasperation of) HMRC, "Making Tax Digital", some REALLY ropey software that I could mention and the HMRCs very nebulous concept of a "Digital Link".
I won't be sending you an email about it, though, because Google will just reject it.
Whenever I send email to The Register, it gets rejected.
The DSN says my Gmail account is disabled.
I *have* mentioned this before.
I've never had a Gmail account.
Share dealing systems again. The company will have to remain nameless but the name begins with B and it's quite big.
I was using their share dealing system in the early 2000s, and they changed their password algorithms.
Amongst other things the new algorithms were supposed to pick some random letters from a password.
Only trouble was, it wasn't random. It was horribly non-random.
I told them.
They said I was wrong.
I told them again.
They said I was wrong again.
This went on for weeks.
So I proved it to them. I won't tell you how I did that.
Not long after that, they fixed it.
Very soon after that, the Chief Executive of this British bank telephoned my place of business to thank me.
Our receptionist thought he must be a scammer, and wouldn't put him through to me...
A customer used the default short-INT auto-incrementing key on a Microsoft Access table they had created to import a CSV back-up file that our software generated at month end. Obviously it was our fault when it stopped importing after 4 months when we had generated 32k rows. So a quick un-billed site visit fixed that. Were they grateful? Of course not - It was obviously our fault for not making sure that their database, that we did not know about, was "incompatible".
"Some customers in the past have found that their bill is inversely proportional to how grateful they were"
A bit off-topic, but reminds me of my (long deceased) great-aunt who a loooooooong time ago worked as an operator in a manual telephone exchange in a country town. When you were connected to a subscriber, to charge them you pressed a button on the switchboard which clicked one unit on their billing meter.
If a subscriber became abusive the operators were always very nice - because they just kept clicking on the billing meter button as long as the subscriber kept yelling.
Fantastic story and so true of manual exchanges.
Very sad I guess they went automatic in the early 70's. I visited a manual exchange (Morecambe) in 1974 just weeks before it was "cut" to automatic.
I still correspond to a friend who was on the switchboard.
Her male counterparts "manned" the switchboard at nights, they were less tolerant.
A warning, never cross her....she will push your meter button all day long!
She had a posh voice because the GPO trained her to sound posh..(I reckon)
"Number Please"
Toodle Pip.
Mark.
For years this had not been totally compliant, as it would accept XML with a blank line at the start (which apparently the spec says is a nono for some reason). Eventually they decided to clean it up, without bothering to tell anyone, and our code that had relied on the library exclusively suddenly failed to load any files.
It turned out, the devs hadn't noticed that their library was itself generating these blank lines on saving, and we had to do a hurried check for, and remove them when loading. I believe they eventually responded to the howls of protest.
P.S. We still make the check - just in case.
Admit it, we all love to show off how smart we are. It's often a mistake.
I have found that the last thing you want to do is put out an e-mail about what you found that might impact money or security. First option is to keep quiet. It's not your bottom line, after all. Second is to "let it slip" to someone, but choose carefully. If it's someone who will steal the idea for themselves, so much the better. The third option is to leave some clues lying around, but given the cluelessness that abounds, this rarely works.
Let's face it, if you stop them screwing up one way, they will find another.
At one company the Head Office IT Manager put out an email announcing that it had been a month since she had been on a security course and now everything "was locked down as tightly as Gnat's chuff". Except that she and I knew that someone else must have sent it. ;-)
Coat, because I had to let my whole team go from that company, and when I returned from doing this, my manager told me to get my coat as I was also made redundant. Funnily enough all the computers had been removed from our desks, just in case!
This might be a legend. Sounds believable, as a Real Life "multiple tabs open" bug.
It appears that the British had developed some fancy hardware/software that was the top of the top at the time (the kind any of us could perhaps replicate today with a few R-Pi, some Python and a software radio) to pick up ionization trails and do some trigonometry.
If you know anything about mortars in the field, you need about three rounds before you can do real damage. First shot gives you a rough idea, second you adjust the screws in your mortar but likely either too much or not quite, in those days (no idea now) that was a manual operation with some rough settings to guide you and printed calculation tables. By the third one, if you are really good, you will be almost on target, and from there on the poor sods on the other side are out of luck, you're dropping ugly welcome packages right on top of them. Of course you need to have an observer doing good telemetry from the explosions, and you have to really know what you're doing. A more standard crew that is not so good (think kids right out of an Argentinian Secundaria, freezing and famished, maybe 7 shots before they can get anything, if very lucky.)
Well, works pretty much the same from the other side, IFF you can pinpoint where the rounds are originating from. Tricky at best by eyesight. But not that bad if you can get data from the ionization trail, "simple enough" with a radio receiver array. Even if this is 1980s tech, let me not go into detail, who knows how "secret" this kind of things are supposed to be. Whatever, the legend says that the landing British could pinpoint exactly where a mortar was shooting from, after three rounds, and tell their colleagues manning the naval guns, to kindly deliver some of their own into the offending mortar.
Once the Argentinians noticed they were blown off with European precision after their third shot, they really were in a bind. See? to do telemetry when you're shooting, you have to space out the timing of shooting, from your outfit, or else you have no idea if the explosion on the other end was yours, or your mate's shooting from 50 meters away. But, if you had to decamp and set up somewhere else after firing just two rounds, or else, your chance at hitting anything in the other side was nearly nil.
Who knows what happened, perhaps those freezing and hungry kids got frustrated and just starting shooting with no order. Then they realized the British were no longer able to find them...
This didn't help the Argentinians much in terms of hitting anything, because their disorderly shooting didn't give them back good telemetry opportunities. Perhaps at some moment there was one of them bright enough to realize that what gave the British precision came from the mortar trails.
The legend says that from then on, the Argentinians were using bottle rockets, shot from random, different points than the mortar that was testing its parameters. Creating that way fake ionization trails. Bottle rockets might have been too much to expect, though it brings in a low-tech, David vs. Goliath kind of image, but would expect too much from other officers to accept the word of someone in the field, and then have those sent from terra firma or even Stanley. If there were some bright ones at the front, I wouldn't expect any but idiots all the way to Stanley, Buenos Aires and back, it is widely known their supply lines were borked from day one and just got worse later. I would think what was used was just plain standard Mil flares. More prosaic than romantic cheap bottle rockets, but equally effective in confusing ionization trail telemetry.
Yup. That was like "too many tabs open" sending data, that then gets confused in the back end. Blame Ed and his back end, will you? Possibly/probably the software/hardware was developed under ideal lab conditions, go figure, where testing was nice and ordered, not assuming the enemy would misbehave...
If any of you are connected to Falklands or whatever gunnery or other groups that might care about the story, feel free to pass it along. It would be interesting to find confirmation, and also that the story doesn't get lost - this is the kind of "interesting" tidbit that would not make it into standard military history books. Got it through oral transmission, following up after a story regarding on how it was possible to watch Moscow TV from Prague in the late 1950s when reflecting the signal from meteor ionization trails (?!?!) According to that tale, the kid that was doing it got in trouble. But, as we say, that's another story...
It was a teams style project in COBOL class. My team had the backend processing which had input from a VAX and an IBM 360 both batch and interactive. The front end teams thought they could unload all the tedious/hard stuff into the backend and we would fix it. I came up with a very nice fix. When bad data was detected, I returned error codes and messages which appeared to come from the system itself. The project manager was the teacher and even he was fooled at first. When he figured out what was going on, he allowed my fix to remain a secret and directed those teams to the system manuals for troubleshooting and fixing their issues. I did have to fix the one message which revealed my secret to the teacher, the real equivalent system message had a typo of some sort, mine did not. My secret was revealed to other teams at the end of the project.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
