Leap seconds is what allows that assumption to work.
Leap seconds exist only in real time, not in historic recorded time.
There are in fact 86400 "calendar seconds" in a day, exactly.
Essentially, when a day is done, we call it 84600, even though it's actually 86400.epsilon.
Only special applications need to know the exact physical number of seconds between two calendar times, rather than the calendar seconds.