USB Cart of Death: The wheeled scourge that drove Windows devs to despair Microsoft veteran Raymond Chen has recalled a time when Windows developers encountered the trundling terror of the USB Cart of Death following an unfortunate mistake live on stage during a Windows 98 keynote. Speaking to another Redmond old timer, Dave Plummer, Chen reminisced that while the USB Cart of Death did not actually …
You may also find malicious beta testers deployed by the competition.
Their goal is to waste as much development and support time as possible.
For instance, they report bugs that don't exist.
They can give steps to reproduce, doctored screenshots, videos how they encounter the "bug". But the bug doesn't exist
Micros~1 has competition... ?
(Apple and Linux and others really are not that much of dent in their plans for world domination, nor would there (likely) be sanctioned resources to disrupt a Windows beta test. Reverse engineering would be a smarter use of human power)
"that is literally a part of what the beta channel is for"
That may be the case with Microsoft. What it should be for is to ensure that the development and integration testing has been done properly. In fact, it's arguable that that's the role of the alpha testers. There should be relatively few issues making their way through to the beta testers.
Agreed. I was always told that a "beta" was when the developers couldn't break it but it hadn't received comprehensive testing. So a really good beta with no issues found during that further testing could be the same as the eventual production release. An "alpha" release is what I think of when most people talk of beta releases nowadays - it's got the features, but is known to be buggy
My own recollections
Alpha - known to be broken, but it runs well enough to start testing.
Beta - known to be buggy, needs widespread testing to find them.
RC - Mostly bug free but needs a few more tweaks.
Release - How the hell are there still so many bugs? Didn't you guys test this?
Debatable. The beta channel is for "we think we ironed out the bugs, can you verify and try stuff that we haven't thought of", not "well, we added support, but didn't really test what happens when you plug in 64 devices at the same time and yank the cable out three seconds later"...
Just saying. Beta testing is to make sure your production-ready thing is... production-ready. And based on my experiences with Microsoft a decade ago, that was definitely not the case then.
No, they don't. It's entirely optional. If someone chooses to get betas to play with then they are volunteers. They want the chance to play with and break the latest shiny from Microsoft. If you don't want to donate your time, don't. If you want to be paid then get a job. It's purely user choice.
True but: Their process or evaluation the feedback is messed up. Else my pet bug, broken shadowcopies / previous versions for local drives, wouldn't be in there since about May 2022 and unfixed in 23H2. Despite a huge number of insider feedback from me. But on a different forum someone from Microsoft noticed my constant complaining, so I may be lucky to get it fixed by Q1 2024. Would be nice if others from Microsoft would push it as well, it is so easy to reproduce - you can use the "create restore point" way too, it is the same as creating a shadowcopy using powershell CIM and wmic.exe methods.
To be fair: I have not seen a USB-induced blue screen for more than a decade, not since Vista SP2 / Win7 Sp1. I don't even remember seeing one with Windows XP SP2. Please note me specifying a service pack level for a good reason :D. As for all later Win-OS-es I've never seen one USB induced either.
To be fair part 2: Linux and Mac were not safe from USB induced crashes either.
The only things which were able to blue screen recent Windows, where I actually experienced it, were Intel network drivers (including crashing Server OS-es) and some versions of nvidia drivers. And those are not USB...
And that is currently more the problem than ever: Lots of emotional propaganda screamed around by lots of Dunning-Kruger-Infected. Even some former normal relatives switched to the "Blinde Wutbürger". The tiniest deviation from their "this is right" thinking and they switch from lovely to "rabid bloodhounds of Baskervilles are nothing in comparison" mode. Why should it be better here...
This is STILL a problem!
However, around 15 years ago, I was at a ETSI conference somewhere: Was talking to a colleague in the public area of the centre when, all of a sudden, there was a collective, "What the f***!" arise from many people. Not sure what happened (probably a manformed packet on the network), but those with a particular Lenovo all bluecsreened all at the same time
Ah, yes. the Packet of Death. Fun times. We ran into that with a pen test where the company we had paid to run the test decided to use the "not safe" tests against our production network, forgot to tell us, and took down a few things before we called them up and screamed at them.
as far as USB crashes? I have a bit of an issue with my home workstation forgetting to process USB insert/removals in a timely manner, but then, there might be a grounding issue in the room somewhere...
I'm not sure that Windows 2000 suffered from USB-induced BSODs either.
What is much more recent though is Windows just being dumb with USB configuration and deciding that a device is no longer working even though it works perfectly fine. Unplug from one USB port and plug into another and Windows detects it and uses it just fine. It was the point that a good few years ago I wrote a small application to purge dead USB device entries so we'd stop running out Windows caused unusable USB ports. Hard to remember, but I think that stopped being so necessary from Windows 7 onwards.
In the mid-2000s I was involved in developing a USB powered radio peripheral that was using the generic MS USB driver for XP to talk to it. One day I managed, via the peripheral's firmware, to crash the XP box so badly it screwed the MBR. Yes, that serious a flaw in their own driver's code. The IT department had to re-image to box. So much for post-Win98 USB support being OK!
Maybe I should have got in to penetration testing instead?
I had a USB DVB-T TV adapter that managed to corrupt a Windows XP SP2 system that I kept around as my sole Windows system just I could check things.
I was trying to check that the device worked, because I was having driver problems with it on Linux, which is really where I intended to use it. I had plugged and unplugged it successfully several times, and just repeated the operation, and the system blue-screened and would not reboot. Filesystem was so corrupt that I was unable to rescue much off the system at all. Had to re-install the system to get it working again. Never understood how it damaged the system so badly.
PS: That reminds me of an OnCall Amiga story from 2020, where the reg forum users were more precise about the actual cause than the article... And yes, a driver/DMA/shared memory/unprotected memory issue.
This is why Windows 8 or higher is recommended. Whaaat? Recommending WHAAAAAT? YES! Simplified history: From XP to Vista/Win7 a lot was done on the kernel side to protect against havoc running drivers - one of the reasons why Vista was not as fast as XP until SP1 came out (as long as you had 8 GB quad-core, of course). With Windows 8 and 8.1 the protection against bad drivers got to the next level: A gfx driver can crash, and most of the time Windows recovers from it - and believe me, nVidia does take advantage from that. I have quite a number of "Event ID 4101, Display, "nvlddmkm" not responding" as proof, though the last one was from 10th of September this year.
But...but...but...we are always told that the NT kernel was a micro-kernel and so did not suffer such driver problems?
Specifically, a hybrid kernel with the graphics subsystem in the kernel. A design decision on the basis that, if you were doing things that crashed the graphics subsystem, you were running WinNT as a Personal Computer, and a crashed graphics subsystem was a crashed system anyway.
The main (technical[1]) performance flaw of Windows Vista was a shocking storage bug which, depending on your kit, might have never been solved.
As for the gfx subsystem, that should've been the case in 7 (or at least, I remember hitting "your system has recovered from a GPU crash" messages in 7), if not Vista, although it did change again on Windows 8. The good bits that Windows 8 brought to the table
On one hand yes, it's quite the shame that the (comparatively) revolutionary Windows versions are so universally panned (with the exceptions of 95 and XP[2]), and with good reason: Vista started taking security seriously and "ruggedized" updates, while 8 put some necessary finishing touches on the update front to finally make "upgrading your windows install" a viable, respectable path instead of a joke in bad taste[3], and began the saga of the new-style control-panel which, while awful in its time, in Windows 11 is almost ready for prime-time.
[1]: The main performance flaw was the social/human failure of the "Windows Vista Capable" labelling for computers that very much weren't
[2]: For readers of this fine news site XP may be Fisher-Price Windows 2000, but for home users 'twas the first taste of NT outside corporate kit
[3]: On a personal note, when I was a kid I bought a Windows 98 upgrade CD and successfully upgraded our family's PC from 95 to 98 (it worked! for about 6-12 months!)
Peter:
Just exiting the same situation myself, but with recovery. Here's what works (all Linux tools):
* Data/filesystem recovery: TestDisk. Tip: Advanced Scan is the big boy, but only available after Quick Scan completes. Save yourself the 3-4 hour wait by Stopping the Quick Scan: Advanced then becomes available immediately. Rewrite partition map. 100% fs recovery at this point.
* dd 1st 512 bytes off disk, note partition map (now) good but the rest junk. Use hexeditor on copy to rebuild it:
* BootLoader #1 (where Grub is bootloader #3 in the sequence & BIOS is #0) can now be pulled out of the middle of diskpart.exe. Search for the key byte markers, or see TheStarman site for offsets. About 3-400 bytes: copy it over the corrupt junk; zero out rest.
* Make sure the 3 magic# bytes pre disk signature are OK. They're not normally written to, so were ok on my disk; if not, just re-enter them.
* Disk-Signature probably also scrambled. You can pull this out of the restored partition's Registry files with a reg-reader. From memory: /windows/config/sys is the hive file, and TheStarman has the precise key. This gives you a long list of disks (I had 60). Use spreadsheet and regex to split the list values out into Sig, StartOffset, & SizeOffset. Use the latter 2 to identify your physical disk: cross-check via Gpartd that the byte values match up precisely (512b blocks in Reg). NB: 1st partitions should all start at block 63 so that's an easy timesaver for boot disks. NB: boot disk upgrades will show as multiple entries for a DiskSig. Type that disk sig into the 4 bytes after the magic #s pre partition map.
* dd result back over the first 512bytes of the disk
* Welcome back.
Notes:
* this is for pre-Vista disks (XP in my case). I did run across a suggestion that there's a _further_ layer of endian switch in the disk signature for Vista+: something to watch out for. Ie, that the Registry disk sig might need to be endian switched on disk.
* when cross-checking diffs via cmp, the apparently mad #s are NOT yet another endian wrinkle requiring yet another deepdive to resolve: as I took way too long to realise&confirm, cmp is resolutely bytes, not words: 8 bit: you're looking at Octal. (That was a nasty moment then a great relief)
* I'm just typing this on the phone over the morning's first coffee by the river. I'll be near desktop later and try to remember to post some timesaving links. This was the tiny outcome of a LOT of deepdiving and blind alleys...
Rightio this'll have to do, can't quickly find the other low-level MBR stuff but guess I've summarised above the key lacunae of the below links:
Key background info:
* crawling the byte-structure of the various MBRs: https://thestarman.pcministry.com/asm/mbr/Win2kmbr.htm
* files embedding the Stage-1 bootloaders: https://thestarman.pcministry.com/asm/mbr/WTC.htm
Watch out for the endianess swaps.
DiskSig:
* 7E00 was what I was trying to remember this morning:
if looking for single-boot-partition disks, they should start at Offset= 0x7E00 (dec32,256 bytes; 63 sectors skipped)
* the lists (duplicated) are in Registry Key: HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\STORAGE\Volume
On a booted Windows, it'll also be in other key(s) but in RAM-only.
I made a mistake earlier re which File has the Hive; it is actually "\WINDOWS\system32\config\SYSTEM"
Tools which actually WORKED (without crashing, being useless, etc):
* Testdisk for partition recovery
* chntpw for Registry reading
* wxHexEditor, being the only hex editor that wasn't catastrophically buggy, useless, or both
Thanks for all of this, but the system is long gone. As a indication of how old it was, it had an AMD Athlon XP 2600+ Thoroughbred processor, so got ditched when my kids upgraded their gaming rigs, handing down (up?) their replaced mobo, CPU and memory for my one deskside system (it's currently running an Intel Core Quad Extreme with 16GB of memory), but as they carry their Windows licenses on wherever possible, it now exclusively runs Linux.
I do still have Windows partitions on the 2nd user ex. corporate Thinkpads that I buy for my main daily driver, utilising the license that comes with them (even though the MS T&Cs probably prohibit transfer of license, although they do activate), but they are very rarely used. Annoyingly, they generally use 2 primary partitions, limiting what I can do until the time I decide to use GPT partitions instead.
Never understood how it damaged the system so badly.
Power supply irregularities. SATA disks are susceptible to power supply interruptions/spikes/brownouts on either connection, as I've learned to my sorrow. And USB devices connect to the MB power supply rails -- as I've also learned to my sorrow.
In a perfect world, the USB power would be sufficiently isolated from the SATA system that you couldn't cause problems, but this world is not perfect. The processor power is normally well isolated, because for ?? 20 years ?? there have been very specific specifications for x86 power requirements. Peripherals, not so much. So you don't get processor resets, you get device glitches.
And who knows about the quality of USB devices. For whatever reason, I've junked a lot of mother boards that have had USB power failures.
USB is still misunderstood by many.
If you want to create a device, the common way to do it, is to buy a similar device that is working fine and then copy its USB descriptors, then modify to suit your project.
I mean that is just a fraction of the hoops you have to jump through.
The actual knowledge is still guarded like a cat over its kittens.
Heh; that's pretty much how Cisco does the USB 'console' port on their current line of switches- you plug in your baby-blue USB A to USB mini(!) cable into the switch, and it enumerates as a serial port.
I still have a SeaLevel USB to serial converter with an old school baby-blue Cisco Serial to RJ45 console cable in my connector kit, though...
I've not noticed that being an issue on FreeBSD. What i DO find disturbing is that Windows seems to have to not only re-enumarte a device but re-load the drivers if you unplug a USB device and then plug it back in to another USB port and it takes a noticeable amount of time to do that. If I switch my mouse or keyboard into another port on FreeBSD, it just works, instantly.
This is because your keyboard and mouse are not reporting a serial number. Windows binds devices by serial number, or, if that is missing, the port it was plugged into.
For example, I have a USB flight simulation yoke, which does not report a serial number in its USB descriptor. I have one dedicated port on my USB hub that I plug it into, and it always works there immediately. If I unplug it and plug it into a different port, Windows will re-instantiate the driver *and* give the device a new name (like "Foo 2", then "Foo 3", ...). I then have to re-calibrate this device all over again (null zones, sensitivity, center point, etc) because as far as Windows is concerned it's a completely different device. I got so fed up of this that I put a yellow zip-tie around the cable just before the USB connector, and a piece of yellow tape above that port on the hub. Likewise red zip-tie and red tape for the throttle quadrant which also doesn't report a serial number. You see this all the time in the Network & Sharing Center list of network adapters when you're using USB to Ethernet dongles for laptops and tablets.
FreeBSD (and Linux) don't care what port you plug things into, or whether it has a serial number or not, because the device driver is only loaded once, the first time any device that uses it is enumerated. Plug in any other keyboard into any other port and it just works immediately because the USB HID driver is already present; it wasn't unloaded when you unplugged the first keyboard and there is no device to driver binding; the list of device types that the driver handles is in the driver itself.
Which, on linux, makes it impossible to identify multiple printers and (in our case) multiple laboratory power supply units. So, laborious process of sending print commands to multiple printers, then ignoring ones that error out with "wrong paper", and user intervention for physical disconnection of other units.
Along with ignoring the serial number, the distributions we're using take the position that, if the system doesn't care about the serial number, neither should you. The information is not available at the application level. I think there is probably some simple method of configuring permissions that would allow application access to the hardware (and hence the serial number somehow), but if so, nobody here has ever been able to figure it out.
On Linux, I've come across devices that when plugged, unplugged and then re-plugged will be installed as a second instance of the same device. I think the last was a Gigabit USB3 network adapter. Not sure whether the problem is in udev/systemd, or the USB implementation for the device.
"The USB standard emerged in 1996 as a way of powering and connecting devices on a single bus."
A bit like the Atari 800 did all those years ago. Serial connector on the back and everything daisy chained along the serial bus.
And, IIRC, the guy who designed that went on to be on the USB standards committee or helped design it or something (not sure now, I read that quite some years ago)
> .Even the slowest USB connection is faster in any way
Damning with faint praise, eh?
USB is faster than the speeds that could be managed by an 8-bit 6502 clocked at under 1.8MHz - pointing that out is really not disparaging the Atari 800 in any way, nor is it saying a great deal in USB's favour!
"Old hands – such as your writer – well remember the days when printers, keyboards, and even the newfangled mouse all had their own connectors. USB promised a way of resolving the mess."
Uhh, well... not exactly. AT connections were probably the most "exclusive"... mice would be connected using a standard serial port, which is not exclusive at all. PS/2 was probably the 2nd most exclusive, a mini-DIN connection developed by IBM and subsequently used on many motherboards for mouse and keyboard connectivity. Back long ago, most devices used either serial or parallel connections... and all computers had those. Mice were serial (as were modems and other devices), printers were parallel... but it's not as if each single device type had a different connector. USB is just much smaller and can support all these things, meaning a single port can do much more. It's also hot-swappable.
Ps2 was a pain, two identical (almost) sockets for different devices, yes they were colour coded but trying to refit them behind a typical installed base unit was an absolute pain.
As to printers I had a citizen dot matrix years ago which you could swap the interface from a serial to a parallel as the interface was on a self contained card. It might had had all the printer processing on the card as well…
Ps2 was a pain, two identical (almost) sockets for different devices, yes they were colour coded"
PS/2 connectors were never color coded in IBM PS/2 computers, and at the beginning the ports were just marked simply as "1" and "2".
Several years after IBM killed PS/2 line of computers did MS + Intel come up with the PC-97 design recommendation which included the PS/2 color coding.
A PS/2 mouse was immediately addressable whereas with a serial mice autodetection wasn't granted - you needed to know which physical port one was connected. Computers (way) back then usually had two or more serial ports with no visible address at the back of a computer, so bit of guess work at times to set up.
"but trying to refit them behind a typical installed base unit was an absolute pain."
You mean inserting without seeing? I can't say the AT style DIN keyboard was any easier, and inserting a COM cable by feel wasn't easy either since they weren't always oriented the same way at the back of the chassis. At least some PS/2 mice and keyboards had a notch to indicate which way was "up".
Then again serial cables had a benefit in that they could be fastened with screws - as long as you didn't screw them too tight.
And USB Type-A still usually requires 3 tries to get it right. :-)
A PS/2 mouse was immediately addressable whereas with a serial mice autodetection wasn't granted - you needed to know which physical port one was connected. Computers (way) back then usually had two or more serial ports with no visible address at the back of a computer, so bit of guess work at times to set up.
The PS/2 did mouse enumeration in the BIOS. Serial mice were (and are) enumerated by the mouse driver. The original "MS Mouse" had a dedicated mouse card, so Windows 2 probably didn't have a problem finding it (By choice, I was not a Win 2/3 user). Serial mice were originally third party devices, and I can't remember when automatic mouse enumeration became a thing: certainly Win95 identifies your mouse port by detection.
Although mouse port detection was automatic with serial mice, that was not the case for almost everything else. Finding your modem remained laborious.
Wish I could upvote you twice for the Citizen dot-matrix reference. Care for this ----> instead?
Those were the days of IBM DOS 2.1 and the original Broderbund Print Shop making our own greeting cards and calendars...
Fast forward: Our Citizen eventually blew a fuse, and my replacement must not have been quite right because it let out some magic blue smoke with a small "BANG!" and a spark. It had served our family well.
The article is referring to each device having it's own connector, because you couldn't [1] daisy-chain stuff, it's not trying to suggest that each connector was unique to any given peripheral. So yes, you could have a "generic" serial port into which you could plug a mouse. But you couldn't then also plug a scanner, or a modem, or any other serial port device into *that* serial port, you'd have to have multiple such ports if you wanted the ability to connect multiple serial port devices at the same time.
So it's not merely that USB is much smaller [2], can support multiple different types of device, and is hot swappable. The key point here is that a single USB port on the PC can support simultaneous connection of multiple such devices, hence the cart of death with its myriad of things which would all get connected to the test/victim PC *at the same time*
[1] not always true in the old days - the ZX Spectrum edge connector supported daisy-chaining to some extent, for example.
[2] depending on which type of older connector, and which type of USB connector, you're comparing, this isn't always true either.
I sincerely hope that wasn't Microsoft's primary mode of testing their USB stack. Then again, given that they made a fetish out of releasing stuff late and buggy and being macho about it, it seems plausible that their stone age cut the victim in half to see if it bleeds approach was the best they had.
Now, how about that unit testing to ensure those BSODs don't happen in the first place - and maybe we get some more helpful diagnostics instead ?
I love these kinds of things that give people who have zero experience with programming a small peek behind the curtain at just how much work it actually is. People think it's like writing an essay for a class, but it's more like trying to teach a newborn baby, with absolutely zero natural instincts, how to do everything. From making sure the heart beats to pump blood to remembering to breathe, to literally every single tiny step to perform even the most basic of actions, many of which we don't even consciously think about.
It's also noteworthy that not every USB host port provider was entirely consistent with their reading of the OHCI standard. The Intel UHCI was better (as you had to buy a license to use it), but it wasn't until about the time of the EHCI (USB 2.0) that everyone kinda figured out how to do xHCI. And of course it was Microsoft's job to fix the problem caused by NEC/VIA/whoever producing something a bit different from Intel's version.
When Borkzilla had a testing department and bothered itself enough to make sure everything worked as well as possible. Bill Gates having demonstrated a BSOD on stage must have been quite the incentive.
Unfortunately, Nadella on stage with a BSOD won't change anything these days. What a shame.
...perhaps whilst the scars were still healing from the earlier use of the term when attempts were being made to rid the world of the horrors of having to manually juggle IRQ settings etc. when fitting new expansion cards.
In a previous era I was asked to reverse engineer a box that had a GPIB/HPIB interface on it and use it for bespoke purposes. My employer at the time was a heavy user of HP instrumentation, plotters and pc's. I dare say that if HP manufactured a teapot there would be an IEEE488 connector on it. What I did worked reliably for the low speeds we used, but the size of the connectors and cables would have been unwieldy for consumer usage.
A bigger problem which I didn't encounter was the non-standard commands used for/by different devices on the bus. I would have thought manufacturers would look at the mistakes made with GPIB and make sure history doesn't repeat itself. However, the solution to yanking peripherals with GPIB connectors on was simple: everything is screwed in tight.
With the ( to me) recent switch to usb-c connected docking stations which drive multiple monitors of different resolution, at least one nice, sound, a bunch of legacy USB Shit from sometimes literally the previous Millennium, mice with built in color changing leds bought for less than 5 ucu from some Chinese Ali baba clone website, connected to a USB hub in one of those monitors and then plugging in usb3 disk drives in the same slot that a microsecond ago was syncing iTunes to their phone, yes, USB induced bsods are making a comeback.
Esp If the aforementioned docking station was being powered by the power brick from their first laptop from over a decade ago which was already plugged in and nicely cable tied to the bottom of the desk.
Let's just get something clear: In 1998, no home user would have even wanted a beta build to use on their machine. Today we live in an entirely different place and I would argue: we cannot and should not compare them to one-another due to the massive changes in use and stability. If you are old enough to remember Windows 3.1 through ME, you understand how buggy the OS was. Even sneezing at the wrong moment could lock up the kernel and need to be rebooted.
Today, our live builds and beta builds are not far off from each other. Don't agree with me? That's fine - just remember how today on the insider builds Microsoft can actually re-write and deploy updates of entire subsystems that were unable to be fixed/updated on those early consumer systems.
Today it takes folks 30 mins to download 10 GB of updates, extract and install those patches. Many of the subsystems in W10/11 are able to be updated independently of the entire kernel. Our internet connections are way faster, our NVMe storage is insanely quick, and our processors have more total clock cycles than 10 first gen pentium 4 chips.
Games come out that are in early access and halfway done so studios can collect valuable data that improves the game when the final build is released. If you asked me to play commander keen beta/early access, I would have said no immediately. Today I would want to lend you my spare time so I can help make the game that much better in the end.
Stuff changes. It will change again and people will complain about that as well.
I don't get the downvote here, because you are right. The stability of the OS has improved to a tremendous level, especially after switching to the AMD-x64 codebase as base.
The only thing that, up to now, manage to blue-screen servers are Intel network drivers. Last example was a "Set-Switch-Group" instead of classical teaming for Hyper-V, which made Server 2022 crash two minutes after starting. Others are nVidia drivers, in 2020/2021 some versions could cause "hard crash", i.e. mouse freeze total stuck on Windows 10. Most of the time the OS recovers from nvidia driver crashes, but it did not always manage that in 2020/2021.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
