Thanks, I have now found https://wordpress.org/plugins/disable-json-api/ which has been updated to disable the whole REST API for unauthorised users.
But I can't get my head around why the Wordpress developers haven't made this isn't the default state, If individual users have a use for the API then fine they could switch it on. But then again I don't see the argument for moving the API into core in the first place, rather than leaving it as an addon (where it started life). To me it smacks of a "look at us aren't we clever for doing this" type of thing, rather than something that is genuinely useful to most people.
There are all sorts of things you could build on top of the API, but I'm suggest that for 99% of them you'd be better off doing it a different way.