A ghoulish tale of pigs, devs and docs revived from the dead
"My pages have come alive!" accused one from my pod of guinea pigs, unfeasibly. This (as it turned out) not-so-singular anecdote from my murky professional experience working on large-scale content management projects came to mind this week while reading The Reg. It was triggered by that recent Who, me? story about the little …
COMMENTS
-
-
-
Friday 9th March 2018 11:50 GMT Anonymous Custard
Re: Secret is out
That's swans.
Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.
Although I've known projects where the latter would actually be more of an accurate description...
-
Friday 9th March 2018 14:22 GMT CrazyOldCatMan
Re: Secret is out
Ducks just bob along peacefully and then suddenly disappear completely by diving underwater for no apparent reason before popping up randomly a few feet away.
Err.. not mallards. They don't dive at all (what they do is called "dabbling" - which means their head is underwater but their tails are stuck up out of the water, pointing at the sky). Some ducks (crested ducks for example) do dive but not for very long.
-
-
-
-
Friday 9th March 2018 10:22 GMT Chairman of the Bored
Love it!
You've confirmed some of my deepest, darkest suspicions about what really happens when I grovel and beg for old data to come back.
Sometimes, though, it comes back on its own. Like my night of the living dead:
Supposedly in this one govt agency emails - once purged off the exchange server- are gone forever. Of course that's utter BS because if management wants to pop you for misconduct they can suddenly get not only what you've written, but even stuff you only dreamed of after a few stiff drinks. Mere mortals do not get their lost emails restored.
Imagine my surprise when I opened up outlook after a prolonged server outage and magically found my mail spool transported ten years into the past. Instead of my working set of spam, I was seeing one specific day in 2005 all over again.
Here's the part that keeps me awake: email on the top was in all caps, title screaming: "YOU ARE LATE FOR A MEETING!" Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that.
So where the hell is he, and am I late for a meeting upstairs or down?? Is the weather uncomfortably warm??? Speak, ghost of the server, speak!
-
Friday 9th March 2018 11:53 GMT Bob Wheeler
Re: Love it!
Dear Chairman of the Bored
I have to inform you that not only the 'pages are alive', that 'emails get resurrected' but also there are ghosts in the machines - how else do you explain these damn systems working one moment, then failing then working again without anyone even touching them. I swear the software is taunting us.
In addition to that, it has been known for many years in the highest level of BOFH'dom that the servers also have gods that not only require, but demand blood sacrifice from any mortal working on them in the machine halls.
yours faithfully
IT bod in chief
-
Friday 9th March 2018 12:02 GMT Anonymous Coward
Re: Love it!
"[...] but also there are ghosts in the machines[...]"
A large interactive system in the 1970s. The main administration console suddenly types out "I am the ghost of the machine - I have changed your password". Unfortunately the prankster had made a programming error and the message repeated every few seconds - preventing the admin team from logging in.
He was fired the next day.
-
-
Friday 9th March 2018 13:00 GMT Helldesk Dogsbody
Re: blood sacrifice
We have found a certain degree of success in utilising the blood of users. You may just require more of it thereby creating a win/win situation. The appropriate oath also appears to still work, you know the one: "Why won't you WORK you &*%!ing piece of £$(^!!!" at around 120 dB.
-
Friday 9th March 2018 14:25 GMT CrazyOldCatMan
Re: Love it!
ghosts in the machines - how else do you explain these damn systems working one moment
Mostly, for me, it's the other way round. Stuff won't be working at all until I go and stand next to the computer.
At which point, it'll work perfectly.
Maybe it's my sparkling personality (bought on Ebay) or maybe They Know that in my pocket is an etherkiller and I'm not afraid to use it..
-
-
Friday 9th March 2018 13:17 GMT paulf
Re: Love it!
@ Chairman of the Bored "Problem is the guy who sent it - one nasty piece of work - had died shortly after sending me that."
Was the death, cough, an accident? Did you go all BOFH on him by the top floor windows that have the broken catch as retribution for him typing all in caps?
-
-
Friday 9th March 2018 10:31 GMT Anonymous Coward
Scarily true to life...
I read this as I am trying desparately to build a USB drive with the latest UEFI version for my three Lenovo System X VMWare hosts to unfuck the fuckup that was Lenovo's first iteration of Meltdown/Spectre back in January. Just trying to get the latest UEFI from their site to a bootable device is an effort of ridiculous proportions - the default option gives you an older version of it rather than the current one.
Meanwhile, all my VMs are running on two hosts, and whilst not quite hitting resource limits, I've not got the comfortable feeling I'd like to have...
-
Friday 9th March 2018 14:34 GMT CrazyOldCatMan
Re: Scarily true to life...
desparately to build a USB drive with the latest UEFI version
I spent some considerable time recently trying to install Windows 7 on an old 17" MAcbook Pro I had lying around (my wife wanted a laptop to do Excel-type stuff on but simultaneously didn't want to spend actual money on one..).
Even wiping *everything* on both internal drives and booting from a clean USB or CD install didn't do it. Which is bizarre because (on that self-same Macbook Pro) I used to dual-boot into Windows 7 quite happily.
So, either there's been a EFI update applied to the MBP that borks the boot process or the latest incarnation of Windows 7 won't work at all on a 2011 MBP.
Or, knowing my luck, both. Need to find my *old* W7 boot CD. Trouble is, it was done on a cheap CD-R so probably won't be readable either.
-
-
Friday 9th March 2018 11:28 GMT Fading
Re: Lesson
Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time. This product will lack pretty much all the features that differentiated it from the previous version during the procurement process and if you are very lucky will at least do 80% of what you need it to do.
Then it breaks and the budget put aside for enhancements rapidly becomes subsumed into fixes and maintenance.
Then the cycle begins again - lessons may be learnt but as only 2 people were around from the last time and are still suffering PTSD it is unlikely to break the software cycle.....
-
Friday 9th March 2018 16:38 GMT Doctor Syntax
Re: Lesson
"Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither. Minimal viable product tends to be what is delivered irrespective of development time."
This is a lesson that needs to be learned by manglement. It never happens because when you know you're always right, why would you have to learn anything.
-
Friday 9th March 2018 18:12 GMT Mark 85
Re: Lesson
Back in the real world there is time for testing or time for fixing the bugs discovered during testing or neither.
Usually what happens is "neither". Seems manglement only see the "go live day" as the goal. They'll hammer with "improvements" and "features" to the point that the "testing" schedule gets tossed out the window as no time is left. In an ideal world, if the testing start date slips, so should the "go live date".
-
Friday 9th March 2018 11:50 GMT Anonymous Coward
Re: Lesson
"Test everything thoroughly before you go live, and don't go live until you know it works"?
As Donald Rumsfeld said - there are always "unknown unknowns". In other words - expect the unexpected.
The live environment is by definition different from any test environment - no matter how accurately you believe it to have been staged in a dress rehearsal.
What has often happened in the past is that on the "live" day users start reporting problems as requested. What they fail to tell you is that the problems existed previously but no one had fixed them.
I remember two major switch-overs.
1) A large online system was being moved to the next generation hardware - and running a different O/S. This involved virtual machines and all sorts of emulations so that no customer programs needed recompiling. Several of the user sites had resisted the advanced testing of their connections - but had eventually agreed. On the "live" day the reported problems were all previously known to the users on the old system.
The plan had been to keep the old system as a warm stand-by for a couple of weeks. After three days the customer asked for it to be switched off.
2) a customer had three mainframes - each with their own comms lines. The project was to use one front end comms processor to allow any terminal to switch between mainframes. The weekend before switchover each comms cable and its port were marked with numbered sticky labels. Different shapes and colours differentiated the three mainframes. The cables were then plugged into the new ports.
On the day we were ready to move all the cables back if necessary. Only one real fault was reported - and that was for a new comms port that the end user site had refused to allow to be pre-tested. The senior engineer said "I would rap the knuckles of anyone doing this" - as he took the port board out of the live rack and put in a spare. It worked. The removed board had a repair label saying "NFF" (No Fault Found).
Over the next few days quite a few problems were reported - and they were all pre-existing conditions that the users had previously endured on the old system.
-
Friday 9th March 2018 13:01 GMT juice
Re: Lesson
Ha. Haha. Hahahahahhaaa...
I was talking to a friend recently, who's working for a company which is about to switch some Very Important financial infrastructure into the cloud.
However, the people doing this work have decided to not tell anyone about this, to prevent the usual "the sky is falling" claims which tend to abound when any changes are made. After all, they've cheerfully proclaimed, everything should Just Work after the changeover.
Oddly though, the list of people who haven't been engaged in the handover process includes the internal QA team. Nor is there any evidence of a contingency or rollback plans.
Overall, it does seem like an appropriate time to order some popcorn and get comfy on the sofa...
-
Friday 9th March 2018 14:37 GMT CrazyOldCatMan
Re: Lesson
Call me naive if you will...
Hi naive.
I presume that you've never worked in a large IT department. That sort of thing is depressingly familiar.
A combination of restricted resources, senior management unrealistic[1] deadlines and general "if anything can go wrong it will"ness will give you it every time.
[1] Surely an oxymoron..
-
Monday 12th March 2018 11:22 GMT Geoff Campbell
Re: Lesson
For the record, I've been working in and with large IT projects for over three decades now.
I understand that testing is frequently done supremely badly or not at all. Which is why the lesson I pointed out should be learned by anyone who have been involved in such a clusterfuck.
Yeah, I'm an idealist. So sue me.
GJC
-
Tuesday 13th March 2018 19:31 GMT Michael Wojcik
Re: Lesson
it is the remarkably simple one of "Test everything thoroughly before you go live, and don't go live until you know it works"?
"Always mount a scratch monkey" is simpler yet.
(N.B. Anyone interested in the corrected version of the scratch monkey story should refer to the first link in the Jargon File entry linked above. Or that link in the previous sentence.)
-
-
Friday 9th March 2018 12:07 GMT juice
Back in the 90s...
There was a demo for some of this newfangled internet technology.
However, as the technology was still a work in progress, the demo involved a certain amount of smoke and mirrors. And the person doing the demo had been given clear instructions about where the smoke and mirrors were placed.
Alas, said person was nervous and strayed from the script. Cue some furious hackery in the background as the tekkies supporting the demo had to try and fake up the S&M elements in realtime.
The main tekkie in question is not a small chap; he'd have made a fine rugby forward or viking berserker. As such, when the demo-presenter cheerfully came over afterwards and proclaimed "well, that all went well, didn't it!", he very nearly got throttled...
-
Friday 9th March 2018 12:21 GMT Anonymous Coward
Re: Back in the 90s...
Same thing in the early 1970s. A roll of pre-punched papertape in a teletype in the mainframe data centre. Ready to be switched back-to-back with the one in the remote press/bigwig hotel venue. A very experienced operator at the demo end - who knew not to deviate from her script.
For the first time ever the system ran without fault - until a few minutes after the demo ended. It was several weeks before such reliability became standard.
-
-
Friday 9th March 2018 12:29 GMT Voidstorm
This is what happens when...
..."Management" publicly promises a live date in six months, before any problem analysis has been done. Or even asking the IT department how feasible it is.
Having worked for a major UK Telco in their development department, I got really sick of that shit; it precludes anyone doing a decent job, and leaves you with "the only thing we have time for is the shittyest quick fix possible".
-
Friday 9th March 2018 13:03 GMT Anonymous Coward
Re: This is what happens when...
I have a bit of code here at work with a comment before it that says (and I quote!), "A seriously egregious hack, but we have to do it..." Of course, I'm the one who wrote it... and it really had to be an egregious hack in order to function properly.
(AC 'cos actual comment in actual code from $employer)
Of course, my favorite is "TEMPORARY, I HOPE HOPE HOPE" in the final version of the Apollo 11 sourcecode. Apparently it worked.
-
Friday 9th March 2018 14:07 GMT GrumpenKraut
Re: This is what happens when...
Here it is: LUNAR_LANDING_GUIDANCE_EQUATIONS.agc (link is to github.com). See lines 179 and 180.
-
-
-
Friday 9th March 2018 15:46 GMT GBE
Allways Educational!
I have to admit that I always end up learning something from Dabbs's columns. Often it's a slang phrase or pop-culture reference unfamiliar to us USAins. This time it was "baked bean wrestling". Somehow, I've spent more than a half-century on this planet without knowing that baked bean wrestling was "a thing".