Perhaps both things are true depending on circumstances.

I once borrowed a book, to find a previous borrowers receipt in it, placed as a bookmark. Upon inspection it turned out that the previous borrower was myself(!) (I recognized the library card number), about ten years earlier.

So probably, no one had borrowed it in the time between. I was very happy the book had not been thrown out.

You can find entertaining stuff there. My interests can be really niche. I remember once finding an amazing book in our college library from the sixties or seventies about the use of LSD in treating psychiatric disorders. While I didn't agree with all the suggestions in there, it was a fascinating time capsule (with colour illustrations, many of them by patients). With the microdosing debate, it's probably relevant again.

Yet when I took the book off the shelf it looked like no one had touched it in many years.

What you are saying is especially true for fiction, less so for nonfiction. Many nonfiction topics are important and require a large volume of materials to remain as reference. For example, you never know when it might be important to know how something was manufactured 50 years ago, or what happened in Congress 20 years ago, or what a newspaper reported a hundred years ago. This makes it really hard to judge which items could be culled. I'm inclined to agree that borrow rates are relevant but they are not the only thing that matters. The possibilities of digitization and interlibrary loan make culling less risky, but someone still has to decide to keep unpopular reference materials for them to remain available.