There are middle grounds, you can still rent physical machines for example. In that case hardware and infrastructure is not your problem. A lot also depends on what you are building, not all software is equal.
Personally I've decided that whatever we deploy needs to run on plain Debian. For now on cloud virtual machines (there's still room to scale those up). But it could easily be moved to either rented or owned physical machines if needed. I don't need a huge amount of nines, if disaster recovery takes an hour or two that's fine. I also don't have a big variation in load, so there is no gain from dynamic scaling. If you do need high availability, and/or you have peak loads over short periods you might be better of with a cloud solution. But my guess would be that a lot of people overestimate the benefits of cloud solutions (and underestimate the overhead they bring).