> For the last 1500/2000 years or so they have basically withdrawn from active intervention
Very well, but it still doesn't answer why they would hang around watching some primitive life form for any length of time. When you see monkeys on the trees, you stop to watch them, make a comment (and a picture) or two, then you mind your own business. You definitely don't spend generations watching them delouse each other, they simply aren't that important.
All right, maybe the invisible aliens are scientists and have been studying us. But in this case a remote camera-like device would be enough, it's not like they need to be here in person if they don't interact with us. They certainly have more important/better things to do than to be present at our pathetic antics.