Programming error created billion-dollar mistake that made the coder ... a hero? Ah, dear readers, welcome once again to Who, Me? in which Reg readers confess the times their reach exceeded their grasp, technology-wise-speaking. This time meet "Francis" who was a bright young thing working on contract as a programmer to a largish organisation that was in the process of upgrading certain systems. Despite …
At a small on site lab, we had a centrifuge that was fitted with a head rated for half the maximum RPM the base unit could deliver. I pointed out that it should probably be hard locked to prevent settings beyond the head rating and even pulled up a manual showing how it could be done. This was left with the lead maintenance technician before I took a much needed holiday.
When I returned, there was a shiny new centrifuge in its place. Apparently, the technician had verified that the centrifuge could indeed be set higher but had then gone off to track down that pesky manual. In the mean time, a sample had been run and the head unit gave way. Fortunately, the shell contained the chunks but it apparently made quite a bang.
In my younger days my employer used a big-ass centrifuge to dry out acetyl salicylic acid in mass.
It had a diameter of 4m, all stainless steel, and was quite heavy.
I would not like to be near that thing should its governor decide to pack up and allow it to overspeed out of control....
Apparently[1], you could learn the history of experimental hardware, particularly the centrifuges, by reading the scars in the walls of the university basement lab: note how the 4 cm deep gash bisects that strange chemical stain...
[1] "apparently" 'cos I never went down to look - mad science is scarey!
I once rescued a trainee who had been backed into a corner by a rampaging centrifuge into which she had placed only one tube (NB my pet TV peeve is people only putting one tube in a centrifuge then not having to chase the beast across the lab.(my second pet peeve are these "naked" centrifuges -- yes I know they're cinematic, but they'd been phased out before Queen's College became Dundee University)). I could reached the power switch and just flipped it off...
--> labcoat
> I could reached the power switch and just flipped it off...
Something is just not quite right about that sentence.
Do we read it as "I could (and did) reach the power switch" OR "I could have reached the power switch (but was having too much fun laughing)".
Given the general attitudes of El Reg commentards, do we think John 110 was more likely to have been a gentlemanly hero or...?
"W0t? You don’t have a self balancing centrifuge?"
Not in 1973 we didn't. It was another 5 years before we had centrifuges that you couldn't open while they were spinning.
(NOTE: The preferred method for bringing a slowing a centrifuge to a full stop back then was to press a rubber pipette teat onto the hub. Not recommended if you want the buffy coat.)
I worked in a refinery and we did "lock out tag out". You had a brass name tag that fit on a pad lock. You used said pad lock to loch machinery in the "off" position. Thereby preventing an intense emotional events in the form of electrocutions or being sucked into large moving machinery. So if working on a system, lock it out because Mr. Murphy.
As a kid we lived next to a 'stately' home with 700 acres of woodland with its own sawmill - lovely old thing all belts and widgets and a 6' diameter saw blade that seemed to take forever to get up to an enormous speed. One day the blade decided to go for a walk and sheered off. It managed to get several hundred yards uphill through densely wooded rugged initially like a knife through butter and then after it struck some rocks and became unstable or bent it just shredded everything in its path for a couple of hundred yards. Its one of the few reasons I'd not advise using rotating masses as energy storage devices!
Well, even one used as a *data* storage device stores energy as a byproduct.
I heard a story 45ish years ago, from what were even then the old days, about just such an occurrence. It seems a multiple-foot-wide disk drive platter broke free of its moorings, and even without teeth it sliced through the drive enclosure and kept going. I wish I could remember more details.
I recall that tale - as a gap year job I started my computing experience at Harwell, and, as an early morning arrival, along with switching on the kettle, I had to turn on the vertical winchester disk as it took 20 mins(?) to get up to speed, before I loaded the paper tape system bootstrap. I was told to run like hell if the disk started making noises...
I heard of one steam turbine in South Africa where the blades decided to leave the building and get encrusted 3m deep into concrete columns.
Considering they were something in the 2 - 5m width range and were spinning at exactly 3000 rpm, their mass was just a multiplier.
When I started on the drawing board at a large steam turbine manufacturing company in 1974, my first assignment was to lay out the installation of a blade wheel in the vacuum overspeed chamber. About a year later, we were all busy at our boards when there was a loud bang and the whole building began shaking, as if we were in an earthquake.
After a while, the shaking slowed down and eventually stopped, during which time most of the inhabitants of the office block had exited and were milling around in the car park. My Boss, one Tom Heath, poked his head round the doorframe and said "That's one of your jobs, Iain, please get me drawing XXXXXXX and bring it into my office". With much fear and trembling I retrieved the drawing from the filing system and trudged reluctantly towards my doom.
We pored over the drawing, recalculating every dimension, stress calculation, and instruction, after which Tom said that he couldn't see anything wrong, perhaps we should go down and inspect the corpse. On arrival on the shop floor, we found that the overspeed chamber had been opened, and its contents were being shovelled out onto a portable workbench for examination. The Foreman was standing by with his arms folded across his chest, his foot tapping, and a face like thunder.
Tom searched through the metallic scrap on the table and suddenly his hand shot out and grabbed a mis-shaped bolt head. He showed it to me and asked "What do you think that is?". I replied that it appeared to be the head of a 3/4 inch black mild steel bolt, which was displaying a textbook cup and cone tensile failure on its underside. Tom unrolled the drawing over the scrap and pointed to a legend which called for 3/4 inch High Tensile bolts to be fitted.
The Foreman suddenly defated like a punctured baloon, and his face went an unhealthy grey colour. "Oh! Shit!" he exclaimed, as the realisation that the failure was down to him and not us, and that he would therefor be footing the bill for a new blade wheel, shaft, instrumentation, and radio communications equipment, all of which had been destroyed by the failure of the drive bracket counterweight.
Tom rolled the drawing up, tapped me on the shoulder with it, and said "Time for a cuppa, I think". We walked away across the shop floor, leaving the Foreman standing dejectedly beside the ruined chamber.
(PS, as well as Tom Heath, we also had a Ted Jones on the section, at a time when Ted Heath was PM and Tom Jones was big in the charts).
... was the one I wrote half a year ago. More or less. It happens less often as I got better. I think it is mostly ok now, except for the bits of code that I know I should have written differently and not under pressure, and I definitely should revise the code rather sooner than later.
I learned a long time ago: if the bug is so obscure that the only possible explanation is that the compiler is generating bad code, well, that just means I've done something *really* dumb.
(Except, of course, for the two or three times in a long career when the compiler really was generating bad code, or the moral equivalent in an interpreted language.)
OP did say "two or three times".
I've found a compiler bug or two m'self, and a number of errors in language-implementation and third-party libraries. But I too have learned to assume it's my error first, and then when I can't find it, assume it's still my error and I just haven't looked hard enough. Only then try to reproduce the problem in some other way to show it's probably not my code.
One C library bug I identified many years ago I demonstrated by instrumenting the large software package that originally showed the problem to log its malloc/realloc/free operations; then I wrote a script that converted the log into C code that just duplicated those operations, after validating them for obvious errors like invalid realloc and duplicate-free. When that program reproduced the problem, I cut it down by bisection until I had a minimal reproduction, which turned out to require only 16 mallocs and 3 frees in a particular pattern.
I used a little USB board that could be hooked up with a PC floppy drive to recover some Amiga formatted disks. These held source code I had written a very long time ago including an unpublished game and a couple of published games that ended up on magazine cover disks.
When I looked at the source I was actually surprised that the code wasn't completely abysmal - it was pretty legible for the most part and I could follow it even though it was invoking Amiga APIs. But I was shocked by how few comments there were in it compared to how many I write now. One of the games had a lot of vector geometry going on in it and it was almost impossible to figure out what was going on without any comments.
almost impossible to figure out what was going on without any comments
One of mine is currently "doing Python" in GCSE Computer Science. I've been impressed with how well it's been picked up, though of course I still get the occasional "hey dad, what have I done wrong here?".
But they haven't been told how (or why) to comment code. Everything written so far is almost completely comment-free. Granted, most of it is self-explanatory at the moment, but a few well-placed comments inserted now would get them into good habits for later methinks.
I have a similar gripe about keyboard skills. Nobody seems to teach "touch typing", despite the fact that most (all?) school leavers will be going in to jobs where typing is a major part of their daily lives. Recipe for RSI, carpal tunnel problems and the like.
M.
That was the problem I had when learning to code. My dad (a university lecturer) taught me first in Pascal, then in C and always expressed the need for comments. As the code I was working on originally was so simple and self-explanatory, plus each exercise was pretty-much standalone, it seemed pointless to me adding comments that explained code which was simple and obvious, and which I had never needed to refer back to so far.
Of course over time, the code steadily becomes more complex as I learnt new commands/functions and more advanced algorithms and techniques. Then we get onto the topic of reusing code when tackling a challenge that is similar to some code I'd written a month or two earlier. Being faced with a fairly complex algorithm and zero comments suddenly made me understand why comments are so useful.
These days I probably end up over-using them if I'm honest, but it doesn't half help when I rip-off some of my old code from previous years.
Then we get onto the topic of reusing code when tackling a challenge that is similar to some code I'd written a month or two earlier. Being faced with a fairly complex algorithm and zero comments suddenly made me understand why comments are so useful. ... Dave K
Would anyone else/many more agree that such is very true here on El Reg which makes the exercising of fairly complex algorithms here invaluable, and indeed much appreciated. It is a service par excellence.
Some years back I landed a job at a large event-management outfit. The role was on a greenfield project which had been running for a year or so, being built by a sizable 3rd-party systems provider. I was entrusted to the care of 'Ada' (for that was not her name) who was a hotshot contractor assigned by the aforementioned provider who had been on the project since its inception. They were in the process of handing-over to internal staff, of which there were few - I was a new hire to beef-up the crew.
After the usual corporate-standard fortnight of sitting around waiting for a desk, a laptop, email and security permissions setup, etc. I was finally given my 'intro to the system' session by Ada. Neither of us expected this to take long - she knew the code well, and I've been getting paid to write code for 30+ years. She walked me through the repo, pulled the source, and showed me some of the code she and her associates had created.
I noticed a distinct lack of comments. As in, zero. Not one, in any source file, about anything, no matter where in the object model we were or how specialised the business logic seemed to be. Nothing at all to give even a hint of what the oftentimes-complex code did, or why it did it, or when. I made an observation to Ada, indicating my surprise. Her reply, drawn from her vast well of commercial experience (2 years out of Uni and in her first contract placement) chilled my blood.
"Comments are a code-smell."
I resigned that day.
It depends. I usually have no problem reading other people’s code without comments in Java when I see a system for the first time provided it is well written. The average C code on the other hand is usually written in a way that requires comments.
But some people’s code is very difficult to read even if it is full of comments in any language.
I did experiment with writing code in HTML documents so the code was in between verbose comments and yet the build process stripped out all but the code before running make. Some editors now can call a URL so you can just add a URL in the code comments to take you to the bit in the documentation related to the following code. I still feel there is some option for hypertext in code so documentation (from sketching out the projects to completion) allows easy jumping to the right documentation is worth exploring but teaching old dogs new tricks is often counter-productive.
I know of plenty of algorithms that are very difficult for a maintainer to understand without additional explanation, even when implemented cleanly in a language with ample abstractions. Individual function points may be relatively easy to grasp, but understanding what they do in concert may not be.
I've read the arguments for "comments are a code smell", and it's an interesting philosophical proposition, but it utterly fails in practice.
The arguments are that code which isn't comprehensible itself needs to be refactored, and that comments don't get updated during maintenance (or even during initial development) and so you get skew between code and comments. More generally, it's that having different documents for different audiences – code and comments for developers, and just code for machines – increases the likelihood of error.
While these are all points with some validity, in the real world, there is always code which is too complex to be sure maintainers will divine its intention, regardless of how well it's structured; failure to update comments should be fixed by process, including code reviews; and humans can't perfectly emulate machines when reading code anyway.
"Comments are a code smell" is a sophomorism, the sort of reductive thinking you get from people who have some experience but not enough, and so still believe in elegant solutions to irreducibly complicated problems.
I learned the hard way. Coming from BASIC on a 32K micro, there were no comments as comments took space and execution time.
I started writing C like that.
Until I came back to revise a project several years later and eventually figured that rewriting chunks rather than try to figure out what was actually going on would be a quicker solution.
Gone are the short 1-2 letter variable names, and hello lots of commenting. It'll help down the line, especially if something is being done in an unusual way and you leave a note to explain why.
When I was doing some particularly involved kernel mode code on a VAX, I ended up with approximately a 80x24 screenful of comments per line of code. I also had a preface explaining what the code was meant to do, why a particular approach had been taken, and why other 'obvious' approaches had not been chosen, with links to extensive documentation elsewhere. When you are doing counter-intuitive stuff, you have to work hard to prevent other people making 'obvious' simplifications to improve what 'some idiot' (including previous versions of myself) has written.
I was working on early UNIX (pre-BSD) at Berkeley when one of the professors bellyached at me for including too many comments, as in his opinion it wasted paper.
"Yeah, well, get me a glass tty and I'll stop using the '33 ... "
I had a second-hand VT52 on my desk the following morning. JOY! :-)
The very same professor still demanded weekly dead-tree printouts. Naturally.
"I was working on early UNIX (pre-BSD) at Berkeley when one of the professors bellyached at me for including too many comments, as in his opinion it wasted paper."
Clearly the massive capital expenditure on the computer wasn't his problem, but looking after the pennies worth of paper coming directly from his budget was a huge concern :-)
In a previous life when I was doing more embedded work, I butted heads with a fresh-from-University co-worker who resisted my suggestion to replace the binary-on-the-wire protocol he was struggling to implement with a text-based one.
His reasoning was that everyone knows how slow it is to parse text and it needed to be binary for efficiency.
So instead of being able to sniff the traffic and dump it for his viewing using a terminal program which he already had, he had complicated the task of not only writing both ends of a new protocol, but using those same pieces to write a decoder to make the protocol human readable.
The problem he was running into was that as both ends were still being developed, the decoder itself was also being developed; he was verifying the tools with the same code that was in the tools themselves.
There are times when every cycle counts, but this wasn't one of them. This was two microcontrollers exchanging messages of 100 bytes or less, several times a second, and doing little else aside from busy waiting. I told him there was no bonus check to be had for having the microcontrollers sitting idle 70% of the time rather than 50% of the time.
I asked him to prove how inefficient parsing the text would be and asked: "Who's time is more valuable; you or the computer?" It's a mantra I still chant...
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
