Why not filter by comparison?
Normally you don't source network time from just one server, usually you set up at least 3 different ones and let the NTP daemon decide which one it likes best after a bit of settling in.
If you source from a number of different confirmed locations, wouldn't it be an idea to draw from the non-preferred sources and see if that time is within some adjustable margin of the current system time as set by NTP? That way, when your preferred source starts to drift out of bounds you could flag it as a problem and reject that server. I assume this is best done after system clock drift has settled to something sensible.
That is, of course, provided you can trust the server list - maybe I just moved the problem to DNS crypto with this :).