A Reg-reading techie, a high street bank, some iffy production code – and a financial crash Though there has been much handwringing and finger-pointing over what caused the last financial crash, one Reg reader experienced at first hand the type of muppetry that no doubt played a part. The reader, who requested anonymity and so shall be called Bob, was a bright-eyed new recruit at an unnamed UK high street bank. His …
"checking the outputs are sane should be mandatory for any banking software"
Having watched the news for the past BIGNUM years, I'm convinced that banking is the very definition of insanity. Thus, the software was working correctly and proving an appropriately bullshit value (since it's adding up, not giving an average, the result should have been comically off the mark).
A code analyzer pick this up? Why?
The snippet looks sufficiently like C for me to generalize, and what is written is quite valid code, just not doing what was intended.
A code analyzer like lint will recognize things like the argument or argument types being wrong, code that will never be run, or integer/pointer/data object size mismatches. And when it comes to lint, most modern C compilers and their optimizers will do a better job than lint if the correct options are turned on.
Unless you have a meta language that you code the requirement in separately to check the code, you will not pick up logic errors like this, and if you have such a meta language, firstly, the problem has to be correctly coded in this meta language, and secondly. if it could check the code, it could write the code in the first place, so why employ a programmer?
what is written is quite valid code, just not doing what was intended.
but so is "if (a = 3)" and lint will pick up on that.
A code analyzer like lint will recognize things like the argument or argument types being wrong, code that will never be run
In this case it is code that won't be run. The loop is meaningless since all but the last iteration will be ignored, which a good code analyzer should be able to spot. I'll try it later.
The code "if (a = 3)" is not valid. If statements take a boolean condition. a=3 returns an integer (in most languages). Yes, C will read it and interpret it as a boolean because C will do anything you tell it, no matter how obvious it is that it won't help, but it's a type clash and the more intelligent compilers/interpreters will notice it. You could argue that a loop that always sets x=[iterator] is incorrect, because the end value will always be the latest iterator, but you can't always know that. Even if that was the only code involved, the compiler would have to know that op= (set) is a function that has no side-effects while any other function might. If the loop actually read anything as advanced as or more advanced than
foreach account in list {
exposure=account.exposure();
total_exposure+=exposure;
}
then the compiler would be out of luck. Maybe calling exposure on an account does something. No code to remove, no warning to give.
If(a=3) hmm... in assembler CLC A,=H'3' (forgive the lack of spacing) versus MVC A,=H'3'.
Assembler, it does what it says in the instruction mnemonic.
Assuming A is a halfword value, defined as such and not a halfword in the top byte of a larger field, so the implied length is taken, or a halfword over two smaller fields, so you're only comparing one byte.
A is assumed to be based correctly with an appropriate register pointing to an unprotected core area referencable by your ECB/ACB.
Further terms and condition codes apply.
YMMV.
Yes, C will read it and interpret it as a boolean because C will do anything you tell it, no matter how obvious it is that it won't help, but it's a type clash
C does not have boolean types, at least not before C99. Even there a boolean is just 0 or 1. A conditional expression is evaluated, and if non-zero is treated as true. To quote K&R (ANSI 2nd edition) "the conditional expression is indeed an expression, and can be used wherever any other expression can be.". When using a single '=' instead of '==' in that situation you may get a picky compiler issuing a warning about "possible unintended side-effect", but it is perfectly legal C.
You could argue that a loop that always sets x=[iterator] is incorrect, because the end value will always be the latest iterator, but you can't always know that.
The compiler is entitled to assume it, unless iterator has been declared with the volatile qualifier.
This code is not a no-op. It will change the value in the TOTAL_EXPOSURE variable each time it runs around the loop. Thus there is no need for lint to pick it up.
And even though it would not be efficient, the code does leave the value of the last POSITION.EXPOSURE in the TOTAL_EXPOSURE variable. I can't see why someone would code this, but it is possible (especially if the variables were less meaningful) that this was the intended result.
This code is not a no-op. It will change the value in the TOTAL_EXPOSURE variable each time it runs around the loop. Thus there is no need for lint to pick it up.
Fair enough, I suppose it depends on the scope of TOTAL_EXPOSURE. If it's local, or thread-local, storage then it can't be read by anything until after the loop exits, so the loop is effectively a no-op since it could be replaced by a single instruction to store the last value of POSITION.EXPOSURE in TOTAL_EXPOSURE.
If TOTAL_EXPOSURE is more globally accessible then I agree, other threads/processes could see the incremental changes during the loop.
Programmer, QA, Audit...? It's not unheard of that the specifications that the techies get do not entirely reflect what the business needs or wants - unit tests might not pick that up. Someone on the business side, however, must have tested the bloody thing and signed off UAT results.
Although, having worked at the UK headquarter of one of the major banks at that time (later known as G-SIBs) I'm not surprised by Bob's experience - both the code and the manager's resonse.
A little over 30 years ago I encountered similarly-obvious errors in production code, for controlling satellites in Earth orbit.
I fixed an error. Unit tests blew up: my fix caused a fail. Uh-oh, trouble. If I go fixing the unit tests to accommodate my code fix, I'm jeopardising the whole framework: marking my own homework! And when you're the young grad just doing the work, you're not expected to tell your seniors the whole edifice is rotten.
I concluded in retrospect, it was unit tests that effectively caused the problem. Programmer goal had shifted from "get it working" to "get it through the tests". And as the tests were more complex than the code itself, so they were also more error-prone.
Yep. A while back we read in these pages of a flight simulator that inverted the virtual aircraft when it flew into the southern hemisphere's sky. The thrust of the article was in the quick fix approach supplanting the figure out what's wrong and fix that approach, and was written after one of those "inches/metric" screwups on a Mars shot (as I remember it).
If the test suite is bebuggered then all hope is lost in the face of willfully blind management.
a more plausible scenario:
Low level code monkey: "Look here, this loop doesn't add up anything, it just sets the value to the last one parsed."
Low level PHB: "Hmm! Let's see! You're right"
LLCM: "Why didn't QA catch this?"
LLPHB: "I will have to take this up with my superior."
....
LLPHB: "One of my code monkeys caught this (pointing out the mistake), and wants to know why QA didn't catch it"
Mid level PHB: "We don't have a QA department any more"
LLPHB: "Why??"
MLPHB: "The C-suites needed to make some aggressive quarterly numbers, and had to cut costs. QA was one of them."
LLPHB: "Well, then, what do i tell the LLCM??"
MLPHB: "FIRE HIM!!!!"
LLPHB: "Why?"
MLPHB: "Somebody has to be the scapegoat, do you want to be?"
LLPHB: "NO!"
MLPHB: "And nor do I. And you know the C-suites are not taking the fall, either!"
...
LLPHB and a couple of security guards approach the low level code monkey.
LLPHB: "Your services are no longer required, security is here to escort you from the premises."
LLCM: "W-T-F!!!!???????
Well, in the context of banks and economy, it is technically correct that the total exposure is only the last transaction. Traders, banks and society in general only have short-time memory and that means only the last value can be of importance.
Think of it as the modern version of the Marshmallow experiment. Better eat your cake now when you have a big one in front of you. You will never know how big the subsequent cake will be or how much indigestion the previous cake has caused.
In tech we call this SNAFU IIRC.
This post has been deleted by its author
Better than a company I know where someone scheduled their GDPR email for last friday, and then went on holiday.
Unfortunately for them, the "please let us keep spamming you" email kept firing off, every day, so people came in on Monday to find that they'd spammed all of their users four times each over the bank holiday...
Eagle-eyed readers will note a critical character missing, namely a "+".
I spotted that, but this is potentially easy to miss on examination by Mk.1 eyeball because it's just a single character that's missing....and a relatively easy mistake to make by simply missing a key.
I have long been an advocate of writing slightly more verbose code, which gives a clearer indication of the code's intent, e.g. in this case I would have written...
TOTAL_EXPOSURE = TOTAL_EXPOSURE + POSITION.EXPOSURE
...as opposed to...
TOTAL_EXPOSURE += POSITION.EXPOSURE
The compiler will optimise both of these to the same thing, but if you write source code like you're expecting your mum to read it, it's generally easier to (a) spot these errors, (b) avoid making them in the first place by a single miplaced keystroke.
Bottom line though is that this is the sort of goof that should have been picked up by testing.
In my experience, there are three main types of bug which I've seen in later stages of development...
- Some complex real-world condition which code doesn't take account of, leading to unwanted result - designer/developer oversight
- Some stupid sort of schoolchild error - developer lazyness/stupdity
- Code which has been written to demonstrate how well the developer knows the language, at the expense of readability and reliability - developer arrogance
...and I have found that these tend to occur in fairly equal measure.
"- Code which has been written to demonstrate how well the developer knows the language, at the expense of readability and reliability - developer arrogance"
I may have posted this on El Reg before many years ago, but anyway ... In my comparative youth my boss mentioned an anecdote, whereby the Chairman of the Gas Board promulgated the Edict "Sack all clever programmers ! " :D
The shortcuts ++, --, += and -= were designed to allow the code to map on to the instruction set of the PDP-11 (and probably the PDP-7 before this), because there were auto-increment and auto-decrement instruction modes on these processors.
This made it possible for a skilled programmer to write code that would generate fewer instructions (and thus be faster), rather than seeing whether the compiler would spot the possible short-cuts.
Remember, when B (forerunner of both BCPL and C) was written, the machines Ken and Dennis had were only just capable of running a compiler at all, and code optimization was completely out of the question.
The systems were really slow. When I got my first UNIX Version 6 and later version 7 system to look after (a long time after UNIX was first written), compiling the kernel took over 4 hours (and I had relatively fast disks), and I never did get around to recompiling the tool set, just used what came in from the distribution tape. It got to the point where I would touch many of the .o files and the libraries and bits I had not touched just to fool Make into not going the whole hog and compiling everything.
This direct mapping of high level code to machine instructions is why many people used to refer to C as a two-and-a-half generation language, and suitable for writing efficient code for operating systems.
Nowadays, where the systems are so obscenely fast as to make compiling code a relatively trivial operation, adding optimizers into the compiler such that these short cuts are not necessary is a no-brainer, so they could be deprecated, but they're written into the standards, and C has spawned a huge number of C-like languages that have taken much of C syntax into themselves verbatim.
My rule:
"Program in a way that the compiler will bitch at you"
The += thing leads to problems, especially if one side is actually a pointer (either you end up overwriting pointers if you miss the + or you might end up doing pointer arithmetic by mistake).
The x = a + b thing means that if you mess up, generally the compiler will bitch about something missing or being of the wrong type, or the assignment not being possible, etc.
We have these all-signing-all-dancing tools but if you program in the way of most convenience, they won't know what you're doing is wrong.
Programming is about instructing the computer in a language it understands. You're already coming down to its level. So you may as well come all the way, especially with modern IDE assistance.
Also... compile with EVERY WARNING YOU CAN. Force yourself to fix them all (-Wall -Werror). Still no substitute for proper testing (and how can you not realise that you only added up one of a whole list of numbers?), but at least it gives you a ton of things and gets you out of the habit of possible ambiguity.
I spent my early coding years as a kid trying to turn off warnings and errors and using clever tricks. It was only later that I realised that I need to train myself to avoid them in the first place.
I don't think I have an active codebase, personal or professional, that doesn't compile cleanly on all the architectures it's aimed at. And there are numerous constructs where I just got tired of silly errors and went "the obvious way" in terms of coding it, rather than "the quick way" to solve them.
---
The best one I had in terms of a baffling code problem was caused by memset(), filling memory with '-1', on ARM, with a particular version of glibc. I was porting some long-standing code and the same code worked perfectly on multiple platforms. Worked perfectly on other glibc versions. Worked perfectly with non-negative numbers. And memset() is designed to work with signed integers by default. For some reason, the memset on that particular combination only memset every 10th memory byte or something similar, which screwed up all the future code.
When you see an explicit memset, you expect it to work. Literally, I narrowed it down to the line and then REFUSED to believe the memset could be at fault. Until I overrode it with a macro with a for loop that did the same thing. Turned out, it was a known bug in the library version for that architecture. But drove me insane tracking that down and I didn't really believe it even when someone else narrowed it down for me.
But even something as simple as a basic memset can go wrong.
(P.S. credit to Simon Tatham, whose code it was in, and who happened to work at ARM at the time).
You don't need verbosity with a better programming paradigm
Smalltalk...
totalExposure := positions inject: 0 into: [:pos :sum | pos exposure + sum]
Java / More general OO approach...
totalExposure = map.values().stream().reduce(0, Integer::sum);
F# / General Functional approach...
totalExposure = [1..n] |> List.map exposure |> List.sum
NB indicative pseudocode, but the general point is that few people writing business software need to be explicitly coding a loop these days
No, thats the sort of I'm a skilled coder language use that can be unreadable to someone else.
Long winded loops are easier for someone (who may be less proficient in teh language / from a different language background) to analyse in code review.
Using someone with limited knowledge of a language in code review can be good as it means code has to be very clear and verbose
Will someone please tell me why I am wrong to say that no-one using a modern language (of a higher level than Assembler or C) needs to explicitly code a loop to sum [attributes of] the elements in an array?
NB I'm not objecting to downvotes, I'm just interested in discussing it with people who know more than I do.
"For each X in Y" is pretty clear, no? Why shouldn't that be explicitly coded? -- David Nash
Ah I may have been unclear. Of course "for each" is a much better way of writing a loop than "for i=0; i<total; i++"
My problem is explicitly coding a loop to sum a collection. Most language / library combinations will already have a method to sum a collection of elements.
For instance, in SQL, surely it would be a bit weird to use a FOR EACH loop instead of a SELECT SUM()?
Every time a coder manually bashes out a loop to sum elements they run the risk of getting it wrong. Reusuing a fully tested age-old function (or Collection class method if we're talking OO rather than functional programming) is surely preferably to cranking out a brand new bit of code?
Sure, running sum(list) is fine, if you already have a list. However, consider that the data might not have been in a list, such that the code actually looked more like:
foreach (account in accountsList) {
(login,password)=db_login_fetch(account);
account.access(login,password);
exposure=account.exposure();
total_exposure+=exposure;
}
Sure, you could rewrite it. The other option, using sum, looks like this:
exposures=[];
foreach (account in accountsList) {
(login,password)=db_login_fetch(account);
account.access(login,password);
exposure=account.exposure();
exposures.append(exposure);
}
total_exposure=sum(exposures);
The code is longer. It requires more memory (perhaps quite a bit if there are lots of accounts). This code is assuming a nice list data structure with its own append function and memory management. If this is C, that's more complicated. Storing in a structure takes a bit more time, and it will be thrown away immediately. You can also mess up this code by mistake, as well. This would have prevented the += problem, but it doesn't prevent other problems.
I don't think this is really important; given that the data was in the form of numbers, adding them up or summing a list would both be very basic. However, if I had a different type of data that took more memory, was complicated to "add", or could take a while to access, I would prefer incremental addition rather than a list collection and subsequent summation.
The issue with what you said is contained in the term "modern language".
I don't believe that the article said anything about when the error was coded. At one time, C, Pascal, Algol, PL/1, FORTRAN et. al. were all regarded as modern languages, and none of them had a construct to auto sum elements of an array without a loop, but I suspect that you already know something about older languages, as you give a snippet in Smalltalk.
And then, one of the oldest high level languages around, APL, would allow you to sum across a slice of an array in a single operation, to the point where there is not even an explicit loop construct in the language (don't ask me to write the code, it's nearly 40 years since I wrote any APL in anger, and I don't want to work out how to represent the greek characters necessary to represent it here).
Looking at problems with a different perspective often gives different answers.
Will someone please tell me why I am wrong to say that no-one using a modern language (of a higher level than Assembler or C) needs to explicitly code a loop to sum [attributes of] the elements in an array?
I didn't down-vote you, because what you wrote was, in essence, eminently sensible.
However, in real life, the developer rarely gets to choose the programming language, and a lot of business software is written in languages that would make you shudder.
Also, it's worth remembering that the above is pseudo-code, and the loop in question could very well have had several hundred lines of code. In principle, it is often more performant to write set-based, rather than loop-based code. However, if you are doing anything inside that loop that is anything beyond trivial, that code may become utterly unreadable if rewritten as a lambda (or whatever). There's always a balance to be struck between performance and maintainability. Don't forget: Premature optimization is the root of all evil.
There's nothing wrong with what you are suggesting on the face of it, but in my personal experience it's a solution looking for a problem. I don't see errors like this creeping into the codebase.
Patterns which resolve common programming problems, absolutely! I code mainly in C++, and so in the last 10 years various patterns have come in which make a positive contribution - standardised containers which are fast enough to use without worry, smart pointers, std::atomic, ranged for loops, there's a long list of sensible ways to resolve common problems.
> Will someone please tell me why I am wrong to say that no-one using a modern language (of a higher level than Assembler or C) needs to explicitly code a loop to sum [attributes of] the elements in an array?
Not a downvoter but ...
I have seen LINQ used in some pretty bad ways. Here are a couple.
Developer not realising that their method was O(n^2). They simply forgot that behind those magic select or first or find methods is a loop.
Another developer didn't realise that a .Any(a => a == 5) on a Hashset may yield the same result as .Contains(5), but the former is O(n) and the latter O(1).
In other cases, a loop which iterates some collection and conditionally yields another object with properties based on the initial collection item can end up with such a convoluted expression that the minute it would have taken any half competent developer to follow the intent now takes 10 minutes to unpack, and even then you're wondering if you missed something subtle.
Another issue is with debugging it can be hard to set breakpoints when your line may represent hundreds of function calls.
Note I never said you should never use them. Just remember that most of the time it doesn't matter if your class is 5 lines longer or 150ns slower, but it does matter if your code becomes difficult to read. Use them, but with appropriate discretion.
You aren't wrong that the programmer doesn't necessarily need to code a loop, but I think you might have picked up downvotes because none of the examples provided actually makes the intent nearly as obvious as the original code. And certainly nowhere near as obvious as the COBOL snippets in some of the other comments. All of the "no loop" examples require a certain amount of thinking time to unpick.
Tools and their uses. Asserting that declarative languages are not 'modern' is an opinion. Functional languages have their uses. As do declarative ones. And by the way, if you were not aware, C11 is the latest C standard from 2011-2012. Objectively modern. C++'s latest standard dates all the way back from 2017.
Pissing on either's chips because you either do not understand, like it, or know it is not constructive. Why not try to learn the new standards and find out for yourself where they excel and why they are still so popular, over 40 years after their inception in the case of C.
@John H Woods
NB indicative pseudocode, but the general point is that few people writing business software need to be explicitly coding a loop these days
You ought to use the joke, coat or troll icons when you post something like that. Or end it with "/s".
Because you didn't do any of those things, 11 people (so far) took you seriously and thought you really believed the examples you gave were simpler, clearer, and more efficient on resources than a humble for loop.
Let these downvotes be a lesson to you. You should mark your humour lest somebody take it seriously.
Ummm, you were joking, weren't you?
@handleoclast
I wasn't joking, but I was very unclear. Apologies, I was "thinking out loud" - for some value of thinking.
The three bits of code aren't meant to be a replacement for the loop but various ways of defining the operation "sum" which is then written only once.
Elsewhere collections would be summed using sum(array) in a functional approach and array sum (or array.sum() if you prefer C++ type syntax) in an OO.
The central point that I struggled to make is that I believe it to be simpler, more elegant, and less error-prone to use such a method rather than cranking out a loop every time you want to sum elements of an array. The definitions are just to emphasize that both functional and OO paradigms allow you to add such capabilities even if they aren't part of the base language and libraries.
But it was very poorly explained and I appreciate the downvotes!
----8<----8<----8<----8<----
Smalltalk note: Well, I might have used the first one; in a Smalltalk world no one would bat an eye at the inject:into:) construct.
However, I'd have probably have written...
totalExposure := positions exposures sum
... had a method on the Positions class to return the exposures...
Positions>>exposures
^self collect: [:each each exposure]
... and would be relying on a method called sum in the Collection class...
Collection>>sum
^self inject: 0 [:each :sum | each + sum]
"Elsewhere collections would be summed using sum(array)"
The example code looked a lot like C.
Standard C does not have a "sum" function, and if it did, it would likely just be a loop that does what the example loop did.
Yes, it would be more logical indeed to use the facilities provided by the language rather than using old fashioned code to "do it yourself", so long as the language provides such functions in the first place.
You're transforming something fairly simple and obvious into something that is less so and will take an inexperienced person, such as a domain expert/QA, more effort to figure out.
Thus providing a good example of what another poster had to say above:
Code which has been written to demonstrate how well the developer knows the language, at the expense of readability and reliability - developer arrogance
congrats!
Bottom line though is that this is the sort of goof that should have been picked up by testing.
...picked up by unit testing/test cases written by the Developer who wrote it.
And if that code was originally written by a couple of Pair-Programmers, then, both should be taken out and shot.
But in the general case
expression1 = expression2 + expression3
vs
expression1 += expression2
where
expression1 is textually the same as expression2
The second is better because you don't have to eyeball the code to make sure expression1 and expression2 are identical
e.g
badger[cheescake*duck.hamster+womble].goat[stoat+boat]+=banana
vs
badger[cheescake*duck.hamster+womble].goat[stoat+boat]=badger[cheescake*duck.hamster+womble].goat[stoat-boat]+banana
Did I mean stoat-boat or was that a mistake?
His predecessor- as seen in El Reg's stock photo- didn't spot the missing "+" because some git stole the glass from their spectacle frames.
And also because the programmer in question appears to be a pre-school child.
This should have been bleedin obvious to the trade floor supervisors as the position exposure should have been vastly lower than usual behaviour. They also need to take some blame. I would be astonished if there were no other obvious correlations to individuals matching the overall total.
Yes, coders make mistakes, QA testers probably tested with one candidate so issue would not be apparent.
I have seen a similar issue at a different financial services company although thankfully not as unsafe as trading positions.
As anon pointed out, notwithstanding all the valid points of code generation and testing made above, or even the importance of UAT processes, once it got to live this should have stuck out like a sore thumb to any half-competent end user. To the extent that I have to question whether it was actually used in live anyway. I've certainly found some similarly huge howlers in production systems (although not in a bank to be fair), that upon investigation had been sat in the system unnoticed for years because no one was actually using the thing.
I don't write code apart from a very basic batch file maybe, I struggle to read it and didn't see what was wrong. However, I know that you test by submitting known data and checking against the known result to see that the process has worked. Guess testing wasn't sufficiently carried out here.
... in everyone's critique of the code, you all seem to be making the assumption that the value for total_exposure is going to be displayed or visible in some manner to users.
What if the only purpose of tracking the value is to compare against some threshold value and display an alert / send a notification / ring an alarm or some-such? Then that would explain why it managed to sail under the radar for so long.
Making sweeping assumptions on usage, without seeing the full context makes you almost as bad as the original coder.
"What if the only purpose of tracking the value is to compare against some threshold value and display an alert / send a notification / ring an alarm or some-such? Then that would explain why it managed to sail under the radar for so long.
Making sweeping assumptions on usage, without seeing the full context makes you almost as bad as the original coder."
1) The submitter's boss told him it needed fixing.
2) If that was the case then the TOTAL_EXPOSURE variable is totally misleading and will hurt future generations... and the code is still bad.
Part of the way I work(ed) is to include "sanity checks" for both my code and, for damned sure, anything I get from the operating system(s). I incorporate it as part of the contracts running around in my code, most of which are set by the very business logic, especially for auditing.
It's some decades since I learnt to code, as a kid. Almost as long since I did any (amateur) coding at all. But even so - I was taught that you start by getting the logic right, then code to the logic. Each step matched to the outcome that the logic called for. And debugging involved making sure that that section of code did what the logic wanted it to. Passing dummy data to the code and making sure the output==what you expected it to. That (pseudo code) example looks to me like a rather simple piece of logic, something well within my own limited ability. Which suggests that the debugging stage I was taught to use didn't happen. i.e. no one passed any dummy data through that module. Maybe because amateur programmers aren't paid so doing the job properly doesn't become anyone's cost.
Which leads me to wonder if the root of the fault is neither coders nor testers, but rather bean-counters limiting the time allowed to produce and debug code.
UK retail banks don’t make trades, and never have.
They take deposits and lend them out.
The head office risk management unit do make fixed for floating interest swap trades, but they are simply to hedge their fixed interest rate mortgage and loan books.
Only investment banks engage in prop trading and that has more or less ended since the financial crisis.
As someone from Back in the Day lots of High Street/Retail banks also had trading arms. One might do FX, interest rate derivatives, futures contracts and credit derivatives. This was in the days of The Big Short. Sometimes these would trade on behalf of big (corporate) customers and sometimes they'd trade on behalf of the bank itself.
The intent was that these guys would hedge interest or exchange rate as required by the bank, but they'd often go further. They might back out their risk with other counterparties but even then many got seriously burnt when the counterparties failed, leaving them naked. In one case that I saw the traders realised that the head office wasn't bothering to look at the trades that they'd requested so they just added a few points of margin onto all the trades - making the traders look like experts (whoah bonuses for all!) and the PhDs in Risk Management look like numpties who were losing money on every hedge (Why do we even pay you?).
Not a surprise if the exposure figures weren't being properly reviewed. Bad code but worse incentives.
for each byte in largeDataPacket. // megabytes
{
UpdateStatusBarOnScreen() // takes a millisecond or two
SendByteToDevice() // takes microseconds
}
The result was that sending the entire packet took about 30 minutes to progress a status bar a few hundred pixels across the screen, and only about 10 seconds to actually send all the data to the device.
Nah, Win file copy probably works as follows:
- time to compute a progress report on a 3000 file copy: .5 seconds, severely degrading file copy I/O for duration of polling.
- frequency of progress bar updates chosen by MS: 10Hz, because you really need to know that often.
It sounds daft, but I was rather fond of robocopy.exe for suspecting just this kind of stupidity from them. Any coder focussed on performance knows that profiling/tracing can significantly impact the system under test.
This is why the above code should be written more like:
counter = 0;
size = largeDataPacket.bytes;
pixels = progress_bar_width;
increment = size / pixels;
for each byte in largeDataPacket. // megabytes
{
counter++;
if (counter % increment = 0)
{
UpdateStatusBarOnScreen() // takes a millisecond or two AND ONLY UPDATES WHEN THE SIZE CHANGES
}
SendByteToDevice() // takes microseconds
}
When I started as a C developer many moons ago working on a piece of commercial software, the focus was firmly on quality/testing etc.
Especially if you wanted to charge your then customers £5k a seat for the product in question.
Since then I've been through a number of different organisations, most recently they tend to be financial (banks or hedge funds).
Without exception, every financial outfit I've seen that develops software in-house isn't looking for the resulting system to be good (in quality terms), it has to be "just good enough" to be released (escape?) into Production and then spend the remainder of it's lifespan being propped up with manual bodges/hacks to keep things going (semi-)swimmingly.
As for shorthand increment/decrement operators, I have direct experience of the clusterfuck that can arise from using those, particularly if you have them on both sides of an assignment. We had a library that couldn't be built in debug mode due to a change in compiler behaviour which meant it no longer worked unless it was a release (optimised) build.
Finally one day I got so annoyed with it I went through the source, found the offending statements and replaced them with the longhand equivalents. Never had another issue after that with the bonus of making it transparent what got incremented and in which order.
Tl:dr version? Write for wetware, let the compiler do the smart stuff.
Not being a programmer, or indeed having ever worked in any kind of production system, this does raise in my head the question that has popped up so many times when hearing about these sorts of issues. "On time and in budget" does presuppose that the time needed for the task and the funding allocated bare some resemblance to the amount of time the job will realistically take and the cost of doing it. But in none of these or any other discussions have I gained the sense that the people setting the time frame and the costings are actually in a position to make a realistic assessment of how long it will take, how many people need to spend how many hours using how much equipment and how many delay risks there are. "On time" instead comes across as being the bait to catch the client and the budget is generated by working back from how much the client will be persuaded to spend and deducting how much profit the company wants to make. The actual costs being considered insignificant in the tendering and planning stages.
Or have I got this wrong?
Or have I got this wrong?
Nope you have it exactly right. Most of the problems I have seen in various companies system can be traced back to their accounting practices.
This chunk of work never gets done because no one wants it to come out of their bucket of money. This results in games of hot potato until production crashes, customers start screaming at upper management; And IT are then the idiots who should have fixed this thing before it became a problem.
And don't get me started on the bureaucratic hell that most change management procedures become. A Sisyphean rubber stamping exercise that doesn't do anything to manage or control change/risk in systems, but IS the arse covering process so no one (Except the poor monkey at the coal face) is to blame.
If you want to get a idea of how long something might take, you'd perhaps present the folks who are likely to be doing the actual work with some details of what's expected (lets call that a specification) and after they review it, they give you a steer on how long it's likely to take (lets call that an estimate).
But surely that's just crazy talk??
Most inhouse projects I've seen, the timescales have no such "informed" estimates, more like a particular senior PHB has picked an arbitrary line in the sand (perhaps bonus related??) and that's your lot.
One particular investment back I worked in, I saw graduates who came into their development groups, clamouring to leave after perhaps a year or two as it was so soul destroying. There was no mentoring or training in good development practise. More like what's the maximum we can get with the bare minimum, no exceptions.
Corners weren't just cut, they were butchered.
@AOD
>>increment/decrement operators, I have direct experience of the clusterfuck that can arise from using those,
If you're discussing C, calling aspects of a 46 year old language 'a clusterfuck' may sound supercool and down wit da kids, but it's not as clever as you think. And the article is about assignment operators, not inc/dec operators which are indeed more interesting.
Understanding why they were added, (back in the day), requires a good level of understanding as to what happens behind the scenes of an innocent looking line of C.
Just saying, dude. And I agree with your 2 from 3 comments.
@Mike 125
Not trying to be clever, just my 2p worth on when inappropriate use of shorthand operators (whether for assignment or increment/decrement) can bite you hard, as personally observed in real live commercial code.
The code in question was using pointers on either side of the assignment and each of the pointer expressions had a trailing increment, so something like:
lpCurrent++ = lpNext++;
At which point in executing that line each of the pointers would be incremented was dependent on the implementation behaviour of the relevant compiler (MS C as it happened).
Oh and another reason to break out the operations into multiple steps. Some debuggers will treat a single source line as an atomic operation when stepping through so if you want to debug it and see what's happening, good luck.
The age of the language isn't really relevant here, of more concern is how well it's used.
My "write for wetware" comment stands.
@AOD
lpCurrent++ = lpNext++;
If those are pointers, the left side is illegal in C. But assuming it's an indirection via a pointer, that implies a memcpy operation. The left side would normally show the indirection operator: '*lpCurrent++', but it could be masked by some textual trick. And remember that '*' is higher precedence than '++'.
So the statement behaviour is well defined and does what you'd expect: copy the content, then increment the pointers. Your tools must have been very broken!
But of course, the statement isn't threadsafe by any stretch. It breaks down into a complex sequence of loads and stores.
>>My "write for wetware" comment stands..
Indeed, and I completely agree.
@Mike 125,
Since I haven't looked at the code in question in more than 20 years you'll forgive me if it's not a direct cut & paste. Basically pointer objects on either side of an assignment using shorthand increment operations against both objects.
Looks cool? Sure!
Easy to trace through, debug & maintain? No.
Your assertion about tools being broken, well it was Microsoft C++ 1.52 (if memory serves) and I already explained that the optimising behaviour changed between certain releases. Don't forget we're not discussing the syntactical correctness here per se, more whether just because you *can* code something in a certain way doesn't mean you necessarily should.
My first task when I entered software development was maintenance coding. That exposed me to some other utter horrors and gave valuable first hand experience of "I don't ever want to write anything that resembles this pile of fetid dingo's kidneys".
Whoever was reading these reports and seeing the risk exposure change regularly by large amounts (depending on what was the last position evaluated) should have known something was up.
They should have also known something was up the first time they used this tool, based on "wow we have all this risk but our total exposure is that low?"
The worst thing is if management used it as a guide of "we have room for more risk" they would have just kept piling on risk after risk, and never come close to their threshold. They surely should have known something was up then!
Worst code i've ever seen was written in banks. Some of it by me straight outta uni. There was no qa or code review or audit. Just deadlines. And high staff churn. I worked in a few banks.
One line of Java that sticks in my mind...
// events MUST be processed in order
synchronized(this) {
...
For those of you that don't code synchronization does not guarantee order. The person that wrote this clearly did not know the basics. And because of the comment, its pretty clear that I was the second person ever to read that code. Despite being in production for years in a system that regularly failed due to this bug.
None of the code I wrote was ever audited. I had a friend sat next to me shovelling millions of euros around each night in batchjobs and all his code, always, from each day went straight to live with no qa.
I work in gambling now where we know about risk.
He works in an XP team now at a porn site where quality matters.
AC, natch.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
