Government is us, the people. It's not some mythical entity separate from the society it exists in.
How far are you willing to go the other way? If the police knew where every single citizen was 100% of the time, crime rates would plummet! Super easy, fit everyone with a gps anklet, and make it punishable with jailtime for anyone who tampers with them or removes them. Tadaa!
It's a balance between fascism and liberty, between control and freedom. Removing cash moves that balance too far in the wrong direction, which is why we should oppose it, if we want a free society.