Were they properly setup? Generally you want a local time source (GPS), the local CPU clock, and of course anyone you peer with (ideally 2 other peers on site) and of course your servers (ideally 3 as separate as possible from each other).
So the way it's supposed to work is that NTP models the error in all the above services and noticed when a source deviates. So if someone screws with the local GPS you should ignore it, and do the best you can with the remaining sources.
If you trigger NTP drift with a single source something is wrong with the setup.
