Trade tariffs are also influenced by perceived fairness in production and competition. Directly subsidise your producers more than your target country: tariff increased, indirectly subsidise them with poor working rights for employees: tariff increased. It's the underlying basis for the 4 freedoms, letting movement eliminate unfair competition.
The tories will find themselves seriously constrained on their usual behaviours if they really wants free trade.
Biting the hand that feeds IT
