OS/2 was dead by design, because it was hard coded to the 80286 segementation and security model
I remember that period very distinctly, because I had just sunk the equivalent of a used Porsche (or a new Golf) into am IBM PC-AT clone with an EGA graphics adapter, basically two years savings from freelance programming work while I studying computer science.
I then wrote my own memory extender so my GEM based mapping application could use extended memory, while GEM and DOS were obviously tied to x86 real-mode. It basically switched the 80286 into protected mode for all logic processing, and then reset it into real-mode via a triple fault to do the drawing bits.
It worked, but every PC had its own little differences on how to trigger the reset or do the recovery because the mechanism might have been IBM intellectual property (who used the keyboard controller to toggle reset).
Anyhow, having worked with PDP-11 in the form of a DEC Professional 350 and with VAX machines, I was utterly bent on overcoming the CP/M feel of my 80286, and also ran Microport's Unix System V release 2 on the machine, which included a free Fortran compiler that unfortunately produced pure garbage as code.
It also included a working DOS-box, long before OS2 could deliver that. Using the same reset magic, I'd exploited for my personal extender. I ran a CP/M emulator on that with WordStar inside just for the kicks of running CP/M on a Unix box!
Then the Compaq 386 came along. I even got one coming to my doorstep. The dealer who I had purchased the 80286 from came to my house, rung the bell and told me he had a 386 for me.
You see, when these machines were the price of a new car, house deliveries up the stairs and setup of the machine were actually part of the service...
Can you imagine just how painful it was to tell him that I had not ordered it? And finding out that in fact my father had ordered it for himself? Including a full 32-bit Unix that actually worked like it would on a VAX?
BTW: that Compaq wasn't slow. Perhaps that ESDI HDD wasn't super quick, but the EDO RAM was 32-bits wide and way faster than anything on my 8MHz 80286. And Unix apps don't typically block on physical disk writes.
Anyhow, finally going on topic here:
OS/2 was an OS tailor-made for the Intel 80286. The 80286 was very similar to a PDP-11 and their discrete MMUs, which kept processes and users apart by allocating their code and data into distinct smallish memory segments (16-bit offset addresses) and protecting them from unwarranted access. Unless your program was permitted access to a memory segment, any attempt to load and use it would result in a segmentation fault via hardware and program termination by the OS exception handler.
The 80286 went a bit further yet and allowed for a full context switch between processes via call and task gates, putting almost the entire logic of a process switch into microcode which could be executed via a single call or jump.
That was continued on the 80386 and caused an overexited 16-year old Linus Torvalds to think that writing a Unixoid OS couldn't be all that difficult and fit on a single page or two of code!
It wasn't until Jochen Liedtke of L4 fame carefully dissected just how horribly slow those intrinsic microcoded operations were, that Linux gained the performance which enabled its wider adoption by ditching all those Intel shenanigans and eventually discarded all segmentation use with the transition to 64-bit.
The 80286 didn't have that option, nor did OS/2.
Linus grew great acknowledging, enabling and encouraging others to do better than he did. Perhaps the size of his early mistake burned that lesson in extra strong.
Whatever you say about politics, IBM and their ill fated micro-channel machines has very little to do with the fate of OS/2.
It was doomed by being an 80286 designed OS for 64k segments, very similar to a PDP-11 and its various operating systems.
32-bit CPUs and virtual memory made for a completely different OS design and Microsoft clearly understood that it called for a complete restart.
They snatched Dave Cutler to get their hands on one of the best virtual memory operating systerms availabale at the time, that wasn't Unix.
And the rest is history.
The so called 32-bit versions of OS/2 weren't really a 32-bit OS. To my understanding they were a lot like DOS extenders in that the kernel and many base services mostly remained 16-bit code, but allowed 32-apps with virtual memory.
A re-design of OS/2 for 32 or 64 bits wouldn't have been OS/2, because the segmentation model and its hardware security mechanisms were really at the heart of the OS.
I bought Gordon's OS/2 book when it came out, read it and it spelt out its tight integration with the 80286 on every page and thus its doom. I chugged it into the reycling bin decades ago. With some lingering regret, since I had spent my fortune on the wrong box, but boy am I glad I wasn't in Gordon's place and mispent a career!
I had to read the details on the 80286 architecture to make my extender work.
And I remember reading about those tasks gates and call gates and feeling a pull somewhat similar to what Linus must have felt.
I also remember reading about the Intel 80432. Intel has a penchant for designs that look great on paper.
But by then I had an 80486 running BSD386 and/or various "real" Unix variants, as well as various closed source µ-kernels GMD/Fraunhofer was developing at the time. And my focus was on getting smart TMS34020 based graphics cards to work with X11R4, so I wasn't biting.
I also had access to Unix source code, so why should I settle for something amateur?
After finishing my thesis porting X11R4 to a µ-kernel with a Unix emulator built on Unix source (thus unpublishable), I actually got a job where I was to create a TIGA graphics driver for OS/2 so it could run the PC as an X-terminal. Got the SDK and went diving deeply into OS/2... for a month, after which I was called away to work for my dad's company.
I was glad to go in a way, because even if the technical challenge was interesting and so called 32-bit variants of OS/2 had emerged by then, the smell of death was too strong.
DOS boxes whetted my appetite for VMs and containers and I've built my career on crossing borders or merging operating systems with VMS and Unix lineage and far less µ-kernels than I ever thought likely. Nor did scale-out operating systems like Mosix ever really take off or clusters ever become significant at OS level, except in niches like Non-Stop.
OS/2 to me is the 80432 of operating systems: dead on design.
How Intel then crippled the 80386 to not support full 32-bit virtual machines is another story.
As is how Mendel Rosenblum, Diane Greene and team overcame that limitation via the 80386SL/80486SL SMM (system management mode) and code morphing.
Intel wasn't amused and it's nothing short of irony how Gelsinger came to head the company that destroyed Intel's CPU business case.