A second is duration, but we're really interested in instants. Why can't you can define an instant independently of the observers, e.g. in reference to the rotation of the galaxy (i.e. "this happened at angle 37 from Milky Way - Andromeda line, at distance 403 parsecs from Milky Way center, at 692.3 Milky Way rotations since epoch (Big Bang)").