Speaking from personal experience
I upped sticks from working for a UK company and moved to Amsterdam, where I had got a job working in IT. The job was working providing software support for a UK retail chain, so as a British native it was easy to get the job.
I spent 5 great years in The Netherlands, although not at that one job, as the company took the work in-house in the UK, and I only came back to the UK because of family issues. If we leave in the EU, and I get tied to this island because I can't just up and work in another country off the cuff, I am going to be an angry resident.
People also forget the British migrant workers who will be forced back to this country if Britain leaves the EU. Roughly 2 million+ net British people left these shores, so you'd have those active workers being forced to return to this country.
How do you think Britain would cope with a massive influx of people who don't want to be here, who have become accustomed to a more relaxed and liberal way of living? One thing is for sure, it won't be a happy country.