"I've never understood why TLDs are handled the way they are."
To make it easy to determine what kind of site you're about to deal with. With the initial structure, you could determine what country they chose to attach themselves to. Sometimes, this was validated (Australia, China, Saudi Arabia). Sometimes not, but there was still an ostensible reason for doing so. And there were special zones for certain types of validated domains. .edu means an accredited institution of higher learning in the U.S. .ac.uk means the same for the UK. And pretty much every country has one of those. Which helps clarify that it's a real school and not someone making it up. That's why we have a tree structure.
"The whole "level" thing is just a mess and only exists because of history and tradition."
On what basis? Name exhaustion? If that's your complaint, and I have no clue whether it is, that's a little convenient since I can found a company with a name without having to purchase a hundred similar addresses. Sure, I might have to modify my name to get an address that someone's not sitting on, but if they're an actual place, that prevents trademark clashes. We can deal with those who are not by changing the rules on domain parking or resale, but not by making a couple hundred new TLDs and tossing them out.
"It's just an address. All that is needed is to verify that the full address is not already in use, and then add an entry to DNS for it at some point so that the address points at the correct IP address."
And to ensure that people can't easily abuse it. For that reason, I own any subdomains of my domains and you can't have them. And we don't operate certain TLDs that could cause problems (ICANN specifically banned anyone trying to reserve .home and .local, for example). And to ensure there's a dispute resolution process and a method of cancelling a domain used to commit crimes.
"Yes, I know it's more complex behind the scenes, but only because we have made it complex."
Actually, the technical process of DNS servers isn't all that complex. It takes a lot of hardware because we want reliability, but you can explain it to a nontechnical person in ten minutes.
"As an example I'd pay to register @big.boomer but I'm not interested in managing/maintaining all of .boomer and the chances of anyone else doing it is equally minimal, so why shouldn't all registrars be able to register ANY unused domain?"
[Shudder] Because your desire for that domain isn't enough to justify adding another path to the tree. If you want to, or a registry thinks there's a sufficient interest, they can apply for it and get it. It costs $185K. If they didn't do that, then people would reserve every string out there in the hope that people will buy up domains to avoid scammers misusing them. You wouldn't necessarily get the domain you wanted, because some registry you'd never heard of would have already registered TLDs of every word in the dictionary. If I had my way, they wouldn't even let you apply. New TLDs would be assigned if a large group of disparate people and organizations saw a need and requested it. Otherwise, they can petition their national registries for a local one or use existing TLDs.