My 2 cent...
At home I use FF 3.5 on a 7+ year old P4 Dell with 512 memory. I've 13 extensions (not sure if that's a high or low number - a mixture: Adblock+, NoScript, Web Dev, Tree Tab, Greasemonkey...). Never experienced any problems at all (using Ubuntu, that may help!)
In work I'm on XP - a dual core machine with 2 gb - use FF again, same extensions - same sites, give or take, all seems very similar. The extra CPU and memory is probably being used by various development tools that I'd have going.
Also in work I use IE6 (developing for a certain CRM application that insists on using ActiveX controls limits your choices somewhat...) I have an MS Web Development toolbar installed, that's about it. To be fair I don't really have too much trouble with this either, but I wonder is IE6 less featuretastic that IE7 or 8, so better suited to lesser machines/OSs?
Anecdotally, I think I restart IE the most - and the odd time I do need to restart FF (only on the XP machine, I don't think I ever have on Ubuntu) it brings all the tabs back in (not sure if that's FF or the Tree Style Tab plugin) as they were, so no real panic.
So seeing as neither machine is terribly well specced here, but by and large run fine, I think the trick is to keep extensions lean as possible and/or be careful where you point the browser at
Oh, and I use FF (both XP and Ubuntu) for my internet banking.