That's the problem here: you're on the honor system that everyone knows and follows every rule every time. Maybe you have a really top-notch team, great code review, etc. but can you say for a certainty that this will always be true, or that it's true of every bit of code you use?
Being able to prove that in advance, especially in more complicated scenarios, has a significant value from checking on every build, especially when you think of the many bugs which have been caused by maintenance code breaking some of the assumptions which the original authors had.
Whilst currently the borrow checker is a bit greedy, it's safe to say that most of the time the borrow checker is checking the code is safe the same way I try to reason about memory accesses in C and C++ when dealing with concurrency or freeing memory. The power of the borrow checker shows itself when reasoning about a local function and no longer requires you to reason about the whole application state as the compiler ensures locality to the extent that shared mutable state is not allowed.