We import a lot of food from across the planet now because we can afford to pay for it
And those who can afford to pay for it can continue to do that.
Supply and demand does suggest that when the supply drops, the price will rise. Oxfam estimate that 2 million people in the UK are malnourished now and half a million reliant on food banks. Those are the people for whom life will get harder, not Tim Martin, James Dyson, BoJo or Rees-Mogg.