OK, that's likely apocryphal, but the overall point is kinda valid. People don't always know what they want - because they don't always know what is possible.
So the real question requires digging deeper... it's not enough to know "I want a faster horse", you need to know that they really mean "I want to get there faster, and be able to travel much further without having to stop and feed/rest the horse, and I want to travel inside a heated/cooled conveyance, and I want music while I'm in transit" and so on.
Now you might argue "Well, the customer could have said all of that" and in a pedantic sense you'd be right. But the typical customer wouldn't have had the imagination to even think that stuff because it would have seemed like science fiction at the time.
I think that's the real key to entrepreneurship: combining deeper insights about what people really want/need (based on their words and/or observed actions) with a deep understanding of what's possible at the bleeding edge of technology, and using that combination to build something awesome.
Of course that's easier said than done. I mean, I have no idea how to do it myself. But one can keep trying...
What I am saying is that the "deeper insights about what people really want/need (based on their words and/or observed actions)" aren't really that deep when you consider the amount of information available. Its surprising to me that people like you and me rely on people's claims to develop a hypothesis and test ideas. Shouldn't it be easy to get insights on a large portion of the population?
So I'd think consumer behavior data / demographic data / etc. is a useful guide, but I'm not sure it's sufficient in and of itself.
All of that said, Alan Kay said something interesting in a set of lectures[1][2] he did a year or so ago. He brought up the idea of basing new product ideas on "Human Univerals", or universal aspects of human nature that really don't vary. A writer named Donald Brown wrote a book literally titled Human Univerals[3] which catalogs a number of these (a lot of them are pretty obvious though: food, shelter, sex, etc.). Kay suggested reading this book, pick one of the human univerals, and figure out a way to use technology to address that.
[1]: https://www.youtube.com/watch?v=id1WShzzMCQ
[2]: https://www.youtube.com/watch?v=1e8VZlPBx_0
[3]: https://www.amazon.com/Human-Universals-Donald-Brown/dp/0070...
While mountains of data may exist, most don't seek it out, and even if they did, wouldn't know how to process/interpret what's shown.
That is, it's all wide open if you know where to look. While some major offerings are satisfactory enough to make it harder to break in, many aren't that great or there's room for competing products.
I believe you are right. My whole point is that I am surprised that process is not easier.
