If we leave, what's to stop the house builders reducing the rate of housebuilding in order to maintain their profits? And who's going to do all the building if most of the builders are as, often portrayed, immigrants?
You're aiming your ire at the wrong target. The real problems are the policies of this government and their phallic fallacy of balanced budgets. That's why homelessness and hospital waiting times are increasing.