One mistake was targeting the wrong hardware
One view that I've frequently heard and find very believable, is that IBM's original intention was to develop an OS that would exploit the 16-bit 80286-based PC/AT. The trouble with the 286 is that it was intentionally designed so that it would start in an 8086 compatible mode ("real mode") but could then be switched under software control to its native mode (Protected Mode) - but not back. The latter allowed for larger address spaces, with memory-protection between processes etc. - all the stuff you want for a multi-tasking operating system.
The problem was that of course IBM needed to provide compatibility with by now well-established DOS apps and so they wanted an OS that would be truly multitasking and able to exploit the PC/AT hardware, but still able to run those DOS apps, and that required switching the processor back to Real Mode whenever such an app was running. As the 80286 was specifically designed to prevent switching back to Real Mode, this required an ugly hardware and software fudge that made the whole thing pretty difficult to implement.
If they'd looked a little further ahead, they could have targeted the 80386, which had a lovely virtual 8086 mode built-in and a full 32-bit architecture that would have made the whole OS easier to design from the start. If they'd done that, OS/2 could have been out around in time for the first 386-based machines and would had a much better chance against Windows 3.0, which also exploited those features - though not as well - but had the huge advantage of having got there first. The massive take-up of Windows apps really only occurred once Windows 3.0 was in the market, so I believe it really was at this point that the seeds of OS/2's eventual demise were sown.