The point of unit tests is to provide an executable specification. Ideally, the specification changes less frequently than the code does. In practice, the specification tends to change a lot too, but you should be able to perform optimizations or refactorings without having to change a whole lot of unit tests.

I know teams that have an "every change should break some unit test" policy, and IMNSHO those teams are making way more work for themselves than they need to. The point of tests is not to make sure that the code does what it does, the point is to make sure that it does what you need it to.