Word 97's ghost in the machine: The crash that vanished the moment anyone looked at it

Veteran Microsoft engineer Raymond Chen shared another fireside tale – one to strike fear into the hearts of engineers everywhere: the bug that can't be replicated in the debugger. The story begins near the end of the Word 97 release cycle, when a defect surfaced in the test labs but vanished the moment engineers reached for a …

[theOtherJT]

Bug hunting really was bug hunting in those days...

...can you even imagine anyone going to those lengths now? "Eh, just ship it, we'll patch it later"

[Jay 2]

Re: Bug hunting really was bug hunting in those days...

Patchees? We don' need no steenkin' patchees!

Maybe if enough people publicly shamed them on social media they might patch such a thing post-launch. You know, working as intended etc...

[phuzz]

Re: Bug hunting really was bug hunting in those days...

Why fix it? It's clearly a hardware problem :)

[Someone Else]

Re: Bug hunting really was bug hunting in those days...

More to the point, can you even imagine anyone even knowing how to go to those lengths? Especially at Micros~1??

[BasicReality]

Back when software developers had talent.

(yes, I'm in the industry)

[MiguelC]

They still have, but most were forced to move to other roles....

[Notas Badoff]

Permutate until pow

In the 90s we had to bring in an ICE to catch a number of bugs our side because Apple couldn't write according to standards. Their AppleTalk Remote Access product implemented v.42bis compression in software, and obviously their programmers thought standard docs were bathroom reading only.

Wonderful bugs like sizing the data compression dictionary not quite like the standard said, but one or two slots shorter or longer. So bugs only appeared after completely filling the compression dictionary through long runs and then happening to hit a reference to old and somehow invalid dictionary strings. Permutate until pow.

As we warped our validated software further and further to match Apple's take on standards, the bug stops got rarer and rarer until finally the test harness coughed only every 12 to 16 hours or so. We had to have the external tracebacks the ICE gave us to have any clues.

Came in on the morning, got printouts from the ICE hardware, then spend hours trying to figure out how to understand the garbage Apple had sent. Crinkle the code some more, set up another test run, then return to our normal work load.

And yes, the managers thought the ICE an unneeded expense. "What, you can't debug your code?" No, we can't fathom the non-standard insane crap Apple is sending us. Reverse engineering intensely ego-ridden code hurts!

Perhaps these days AI can be used as an ICE to reverse-engineering ego-ridden management thinking?

[Someone Else]

Re: Permutate until pow

Permutate until pow.

Isn't that one of those legendary (perhaps mythical) machine instructions like Punch Operator or Halt And Catch Fire?

[shodanbo]

Re: Permutate until pow

We could only wish there was an instruction to automate this away!

Nowadays of course we call this "vibe coding" LOL

[Bebu sa Ware]

Halt And Catch Fire ? Mythical ?

Curiously I encountered this undocumented instruction (XHCF) when looking up the Motorola 6809 instruction set.

As a result of nostalgia^† on my part - the 6809 was the first micro I programmed in assembly and probably the nicest instruction set of all the 8 bitters (never dealt with the 2650 or the TMS 9900… arguably 16 bit.)

Presumably 'halt' because only exits the test mode with a hardware reset and 'catch fire' because it is thrashing the address bus with incrementing addresses which presumably generates a lot more heat than just sitting in some sleep state awaiting an interrupt or reset.

_{† the pain (-algia) of foolishly comparing our aged jaundiced view of the present (nost) with those the rose tinted view of our misremembered, juvenile past}

[Dagg]

Re: Halt And Catch Fire ? Mythical ?

I started on the PDP 11 specifically an 11/10. In those days the CPU was all TTL all the way down. The later versions used a micro code emulator. However in the PDP 11/10 there was a specific self destruct instruction 014747(oct) which was:

01 MOV instruction

4 double decrement before read accessing memory pointed by register

7 Register 7 the PC

4 double decrement before write accessing memory pointed by register

7 Register 7 the PC

MOV --(PC),--(PC)

So when the CPU read the memory pointed to by the PC that contained this instruction it would then double increment the PC then start executing the instruction

MOV -> move instruction

--(PC) -> double decrement PC and read contents from location pointed to by PC

--(PC) -> double decrement PC and write contents into location pointed to by PC - AND LEAVE PC unchanged so that it pointing to the word before the previous instruction.

The result of this instruction was to start at the initial 014747 and fill the whole memory below it with 014747

[Kevin McMurtrie]

Re: Permutate until pow

I feel like Apple sank into proprietary implementations of everything because interoperability testing was beneath their levels of ego. iPhones are especially bad. Just try to transfering files with one without an intermediate external service.

[Gene Cash]

Re: Permutate until pow

Jesus. I was retarded enough to buy an iPad so I could read PDFs at lunch without it turning into postage stamp theatre.

Then I tried to transfer said PDFs from my Linux box without any Windows machines anywhere in the vicinity (and I didn't know about VMs at the time)

What was amusing was trying to explain it to the Apple Genius and being told to contact my Linux manufacturer.

Said iPad ended up being used for target practice.

Edit: This was back when iPads were pretty much the only tablet-anything. Later I was even stupider and bought a Motorola Xoom.

[bazza]

Re: Permutate until pow

They couldn’t avoid interoperability testing with their own mobile phone modem development. Though they clearly took shortcuts in this in the early days, Apple’s modems couldn’t do cell tower handovers

[Stevie]

Re: Permutate until pow

But ... Apple just works ...

[Excused Boots]

Re: Permutate until pow

Mostly....

[BartyFartsLast]

Re: Permutate until pow

Sometimes

[Anonymous Coward]

Re: Permutate until pow

With enough market mind-share, you can redefine "works" as anything you do.

[shodanbo]

In embedded development we commonly used ICE's until JTAG made it possible to take more direct control of processors and do debugging without an ICE.

Devs still have talent, but most of us have moved so far up the stack away from the silicon that we don't do stuff like this any longer. I miss those days but there is more work to be done higher up in the stack and you have to go where the work is.

[ravenviz]

In laypersons terms, who is doing the ‘lower down the stack’ work?

[bazza]

Hardly anyone.

Intel a few years ago published an open letter pleading for microelectronics engineers. They said that the US jobs market essentially had none to offer, and no one was doing the subject at university either. The irony was that this was after they’d had a firing round after some chip taped out, so they’d not exactly encouraged anyone to take it on…

And when it comes to manufacturing the chips, we’ll, as we all know that’s got to the “one deep” stage long ago, which is why TSMC et al are so important.

[Anonymous Coward]

It’s those fucking turtles again isn’t it? Turtles, all the way down, belting out machine code with their stupid flippers. It’s not called shell access for nothing.

[Bebu sa Ware]

those fucking turtles again

An novel explanation for global seismic activity: rutting chelonians.

[Bebu sa Ware]

who is doing the ‘lower down the stack’ work?

All a bit arbitrary in architectures where the stack grows from higher addresses to the lower.

But the answer is probably Sam Altman's legions of the damned (and damnable,)

[BartyFartsLast]

ICEs *were* the processor so I'm not entirely sure you could have more control but JTAG is damn good

[trevorde]

Reported a bug in an open source VOIP phone many years ago. It was something threading related and I couldn't repro it with the debugger attached. My conclusion was the debugger upset the threading and timings. Don't know if they ever fixed it.

[FeRDNYC]

Microsoft veteran recalls how a CPU errata

*cough* That's "erratum" in the singular, my good sir.

[Jou (Mxyzptlk)]

Now the Richard Speed has to write that one hundred times on the wall. Fast, before the sun rises and Romans catch him doing it.

[Anonymous Coward]

You're suggesting that there was only one of them? ha ha ha ha cough aaak.

[Taliesinawen]

CPU erratum and hardware defects :o

FeRDNYC:

>> Microsoft veteran recalls how a CPU errata

> *cough* That's "erratum" in the singular, my good sir.

CPU erratum are to the rest of the universe would be known as hardware defects.

[anothercynic]

Re: CPU erratum and hardware defects :o

CPU erratum are to the rest

CPU errata are.

Erratum = singular. Errata = plural.

[FeRDNYC]

Heisenbug, my old nemesis. So we meet again.

[MiguelC]

It's a classic, things that break while running live run flawlessly while under the debugger microscope.

I've had one like that, that I never found out cause or how to correct - it was probably due to events timing or concurrency that stop being a problem when run on a step-by-step basis. Somehow, after a seemingly non-related change, it went away.

[Brave Coward]

'It's a classic, things that break while running live run flawlessly while under the debugger microscope.'

I heard that Volkswagen once had the exact opposite problem.

[Paul Herber]

I think we've nearly exhausted this thread.

[Uncle Slacky]

> things that break while running live run flawlessly while under the debugger microscope.

Happens a lot when you have an optimising compiler - in debug mode the variables were stored neatly, within expected boundaries, but when optimised (i.e. stored into exactly the space required) bad things happened when you tried to reference them at the expected positions.

[Herring`]

Even throwing a printf in could make it work in a release build. Ah, those were the days.

[running with nukes]

Heisenbug

Schrödibuger, my old nemesis. We have never meet.

[Anonymous Coward]

Re: Heisenbug

... because I could never figure out both where you were and where you were going at the same time.

[ravenviz]

Re: Heisenberg, I’m not so sure.

[Ken Shabby]

Lucky it didn’t reveal itself as a Mandlebug

[trevorde]

Latency ... matters

Worked on a product involving VOIP-ish packets from peers and a remote SAAS endpoint. Inhouse testing was perfect until it went to the customer test site, where it promptly fell over. The issue was out of order messages confusing the system. Only one of the devs could repro it. Turned out he was on ethernet whereas everyone else was on wifi. There was just enough of a difference on wifi to mask the bug. Guess what the customer site was using?

[Paul Herber]

Re: Latency ... matters

Bluetooth.

[David 132]

Re: Latency ... matters

Token Ring? Econet? Wait, wait, don't tell me...

[Paul Herber]

Re: Latency ... matters

Token Ring? Wasn't that a network of Hobbits?

[Someone Else]

For various values of...

But without it, Word 97 could have been distinctly wobbly.

Uhhh, Word 97 was distinctly wobbly. (How soon we forget....) I suppose it would have been more wobblier if they hadn't bothered to track this down and fix work around it.

[Anonymous Coward]

Re: For various values of...

Rather than patching the binary, maybe those cowboys could have recompiled with the new compiler that had fixes for the erratum.

[Phil O'Sophical]

Re: For various values of...

Did you not read the part of the article which said that they couldn't change to a new compiler version that late in the cycle because they would then have had to restart the whole test and certification process in case the new compiler introduced other, new, bugs?

[ravenviz]

I definitely had a negative physiological response seeing Cl*ppy.

Can we have prior warning next time please, along the lines of, “Caution: This post includes an image of Clippy, which may invoke a trigger response in some readers.”?

Thank you.

[david 12]

Word 97 introduced ,,, VBA

Word already had Word Basic. Word 97 and Excel got a common language engine and as part of that, Word 97 adopted the 'Excel Basic' name. Not really a headline feature, because Word Basic, Access Basic, and excel basic were all variants of the same language, and the move move to a common library object was import internally for Microsoft, not externally for users.

[Gene Cash]

Oracle had it last month

So you probably already realize that Oracle patches are done with a Java executable.

And of course you usually have to use the LATEST and GREATEST version of Java. WHEEE!!!!

One of the recent middleware patches ran into an situation where the optimizer ended up issuing misaligned data accesses, which pissed off the RedHat libc that's picky about such things, which results in SIGSEGV applying the patch.

Oracle's solution? Run opatch with the -debug option, which disables the optimization.

[hugo tyson]

Bloatware

So the whole app still worked fine with a NOP patched in, instead of the offending instruction(s)? Proves M$ware is bloat....

[hugo tyson]

Code-motion bugs

I once found a long-standing bug in a BlueTooth chip codebase: one particular build consumed a lot more power, like 100x more, than it should. The change that "caused" the bug couldn't possibly do this.

I proved it by editing the faulty binary, to undo the one code change that apparently "caused" the fault, by replacing its test and branch with NOPs. The fault was still there. So it had to be to do with the code *orientation* wrt page boundaries or cache lines or something. And so it was. Someone had removed an "unnecessary" delay sometime in the deep past. It was necessary again, with the code in that particular orientation, which just suited the caching and page boundaries and suchlike. FFS.

This, IMHO, is the only reasonable use for editing a binary with NOPs. It's a powerful technique. I earned greybeard-wizard points :-) But you don't ship it!

Boring detail: I found the mechanism by normal debugging: the chip should, every 10ms or something, read its own temperature, and if it's changed by more than some amount, recalibrate some clocks and oscillators. It was doing this every single time. Because instead of a temperature, the read-temp routine was returning "Failed to read temperature" so its caller did the recal every time just in case. And all because the code to read back the temp was hitting the HW register too soon after kicking off the "get a temperature" AtoD converter.

[Bebu sa Ware]

Debugger Shy Bugs were pretty common in the early days.

I must have seen dozens under MS Codeview and MSDOS. I was never sure whether it was due to differing memory layout, timing, or the whole int3 patching thing. Generally used binary search and embedded writes to localise the bug or more commonly, coding error.

I have seen similar timidity under sdb and dbx with various Unix versions especially in the early days - not so much with adb but then you are usually desperate at that point.

The one I didn't expect was the C compiler using the wrong instruction for byte (or half word) swapping which was inexplicably obscured by the debugger. Only spotted it after generating assembly output from the compiler. Lesson: don't trust anything or anybody (and with those qualifications become a BOFH instead.)

[G.Y.]

Q: Why debug in hex? Re: Debugger Shy Bugs were pretty common in the early days.

A; That's where the bugs live!

[Mcins4nd]

Tangent—rogue e-mail in the 1990s

This reminded me of an email in the Windows 3.X days. There was an email that a number of us had fun with in our building. To the best of our knowledge, the corruption was an accident, but any IBM PS/2 that scrolled to the end crashed. The user could reboot fine, as long as the user didn't scroll through the email again. We had a lot of fun with that one.

[marhor]

ICE - you know what this really is (do you?)

Of course, in reality a real ICE (Intrusion Countermeasures Electronics) just blasts away the users/hackers, thus protecting the software.

(No idea what this is about? Go for W. Gibson & Neuromancer; it's worth it, really!)

[Steve Graham]

I used an ICE to debug software on an OS/2 Warp PC which was communicating with an IBM mainframe. This was while I was a group manager. The task was delegated upwards to me by one of my team leaders who said that his team members were "too stupid" to do it. (Not a people person.)