When I say it doesn't hold up well to scrutiny, I mean in the same way that the concept of race pretty much falls apart as soon as you start looking at trying to define it with any degree of determinism. We use two easy categories which are mighty handy for various purposes, but it's a distinction that we've made because the chromosomal distinctions are far greater (and it can't always come down to genitals, because nature has pretty much done everything you can think of with everything you can think of!).
I was thinking along medical lines but as you rightly point out there's sexual reproduction as well. Male and female is good for those purposes, near as I can tell. There may well be further uses for the finer distinctions within each biological sex but I don't know; I'm not in any way knowledgeable on such matters.