The only real way to find the hidden gems is to have genuine curiosity and be a voracious reader.
Oftentimes you won't know that you're going to get a golden egg from a book until you actually read it, so you find good books by following your curiosity and consistently reading.
The only issue with this approach is that some of the best books out there can be prohibitively expensive, so it helps if you can find them in a library before you buy. But many of the best books will never be in a public library, either.
More generally, the best writing out there either exists in the academic world, or the ancient world. Writing that was written by PhDs for other PhDs, and historical (usually religious) text that was preserved throughout history.
If you're searching for books in a shiny, new bookstore, it's very likely that about 95% of what's in there is going to be disingenuous crap. If you're searching online or in a quality used bookstore you'll have better luck.
