Let's look at the chrome unit tests.
https://chromium.googlesource.com/v8/v8/+/3.0.12.1/test/mjsu...
This test checks one case and then does a loop over to make sure various sizes work. This is a POOR test because it doesn't cover the domain of the function and exceptional cases. This test is pointless to have and run every build. Why keep it, it does nothing useful.
My point is that if you are refactoring a function, you are going to have to write a NEW TEST anyway because the purpose of a refactor is different each time. The old tests are going to be useless. This is why contracts are so useful because they will ALERT YOU when the software does something you didn't assume. Unit tests don't do this.
From an information theory perspective, the old tests would not provide you with new information. If you want to make sure your refactor works like the old function, then compare the outputs of your new function with the old one over the domain of the function. Then if it's good, delete the tests because why keep them.
Experience shows that users will do things to your program you did NOT expect and therefore your tests will not cover. And more likely than not, refactoring requires rewriting tests anyway so the old tests are usually always useless.
Now SYSTEM tests ARE useful. Tests that wire up many components and test them are useful. But a UNIT test, which tests one function is often just a waste of time to keep.
I have one question for you that I feel I have to ask: have you actually practiced commercial software development on a team (say, 5+ developers on the same codebase) in this manner that you describe for a significant period of time (say, 2+ years)? and if so, do you feel the resulting software has been robust and successful?