I didn't add the nonvoters to either side. I pointed out they're actually the biggest bloc.
Take it to an extreme: if only 1 person had voted in the referendum, whichever side they cast their vote for would have had an overwhelming - 100% - share of the vote, but that still wouldn't constitute a democratic mandate.
Turnouts above a certain level are necessary for the continuance of the current political mess. The only tool we have left when politicians aren't fit for purpose is to withhold our votes - unless we can persuade the turkeys to vote for Christmas and change the voting system.