Re: just receive an IP stream, buffer as necessary.
Well, 1ms corresponds to a positioning error of about 30cm: if you move one of the sources by that much, or walk that far, you've changed the timing by that much. So that's pretty clearly not something you need to aim for for a domestic audio system. But as you say the time skew you often get isn't usefully measured in ms, it's measured in whole seconds, and that's not anywhere near OK.
I've just bought a new vehicle which has a radio which does FM and DAB, and has a fancy 'station-following' feature so it knows that it can follow a given station from DAB to FM and back when the DAB (usually) signal goes away (obviously only for stations which exist on both). After a day trying to listen to this strange disjointed mess that it turned radio into I worked out how to turn it off.
Biting the hand that feeds IT
