Moral of the story is scratching your own itch makes you qualified to talk about the problem, it doesn't make you an expert on a good solution.
0: https://kitemaker.co, the super fast, hotkey-driven product management tool/issue tracker that has deep integrations to GitHub, Figma, Slack, etc.
Just because I (as a manager) had previously been willing to adopt bleeding edge tech with no hoops to jump through to eradicate hundreds of man-hours, doesn't mean that the average customer is.
If you're not representative of the typical customer/user, you need to find someone who is to use as a yardstick.
It doesn't automatically mean you're headed in the complete wrong direction, but it does mean a lot of your initial assumptions (pricing, deal size, sales cycle, etc) are probably way-off.
Does, I think, raise the chance of falling in love with your solution though, which is also dangerous.
In my experience to ensure data-driven product decisions you need both a system and process to:
1. Collect feedback
2. Organize it to be useful
3. Analyze and prioritize customer problems
4. Use your feedback to validate problems and solutions directly with customers
5. Communicate status to your team
6. Close the loop with customers when you build their requests
At the early stages you can do this with a spreadsheet or Trello board. Eventually you'll graduate to a tool or in-house solution.
Here's a piece that dives into the details about how to build your own system to collect and organize feedback so you can eliminate bias and drive product decisions based on data:
https://www.savio.io/blog/product-leaders-guide-customer-fee...
Your A/B test just says “look, they spent more time, do it again, they love that antipattern!”
It’s fair comment, though, that undifferentiated ‘engagement’ is rarely a good metric.
Taking a structure/organized approach to collecting, indexing, and sharing the qualitative data isn't something I covered here but absolutely essential to make sure that a research project scales. My experience has been that the first mile is where founders get the most lost.
That didn't work out so well.
It wasn't what they needed.
They didn't buy the product.
But we kept talking to those users and understanding what they thought they were getting and the problems they were hoping to solve. Those conversations gave us a better idea of what would work and helped build a realistic roadmap.
Executing on that roadmap has worked for us. We continue to evolve that roadmap based on customer demand - When somebody asks for something that's on our roadmap, we count their vote.. and generally prioritize work in order of demand.
So ya, talk to your users... it's not about what they ask you build, but what problems they're trying to solve.
0: https://www.enchant.com - shared inboxes, knowledge bases, live chat.
Consider some of the greatest inventions of all time: the printing press, the telephone, the airplane, the Internet, and the web. Considering the notion that they would have been conceived of this way is absurd, so clearly there's more than one way to make something people want.
Mix in the fact that contrarians often disrupt things the most, and when you see a process like this that itself has become considered the consensus approach, it seems like a high point of potential leverage to just try a different method, especially if that one has also been proven to work or is more in tune with your own talents, experience, and intuition.
The article specifically states that you focus on your potential customer’s problems, not the solutions they want or would use. Ie, you’re not asking if they want a better horse, but you are asking what sucks about the one they have today.
From there better shoes, saddles, or even a car/plane can come out of that discussion.
For example, if someone told you today "I don't like that I am going to die one day", would you be able to iterate your way via customer development to stop death? Would the fact that people don't want to die be incorporated in the understanding of the process that led to any solution that gets created? No. Some problems are either so obvious or so non-obvious that the spark of insight that leads to solutions is barely influenced by the articulation of the problem or lack thereof by those who are suffering under it, often completely oblivious of it.
The key in my opinion is to be able to identify market opportunities and problems to solve, then be REALLY flexible about how you go about solving them b/c your first product solution ideas are very unlikely to work.
Another way to see it is it's a form of de-risking an already risky endeavor. Anytime you reduce risk by adopting a methodology or formula that has known benefits, you're basically trading options (in the financial markets sense.) In this case, you're (probably) reducing downside risk, while increasing the odds of success, but at a cost of reducing the overall maximum upside, since your odds of creating a once-in-a-generation disruption goes down if you're using a hill climbing methodology to create wealth.
This has its own pitfalls, but he was fundamentally "talking to users", he had just narrowed his focus group to a user of one. He used his own products as far as we know as well.
The thing is that most tech doesn't dogfood quite so easily. Most founders can't replicate that as they are generally selling outside of the consumer market that they can just adopt themselves.
So, maybe cast another way, every product must have a target demographic and you need to interact with that demographic to get feedback on what you're building. Jobs just happened to have a shortcut that isn't generally applicable.
I read Ken Kocienda's book about working on the original iPhone keyboard [1]. He says development revolved around demos: first to each other, then to senior leadership, then to Jobs.
2000s Apple tried to hire people with good taste, then constantly tested against that taste. It was all internal, but it was testing.
I don't believe that study has any relevance to the world of creative work -- like building products or becoming a Jedi ;) -- that is driven by intrinsic motivation to iterate and improve. Success rarely requires the same level of self-evaluation and reflection in creative contexts.
Two things come to mind:
1) Wants are not needs. To pun Ford a bit, "I want a bigger, faster more reliable horse to get to work. Something safer for my family" is what the clearly stated want is. Remove horse and you're at root need. The Need. Too often, The Need is never distilled from the want.
2) In Covey's classic book "7 Habits of Highly Effective People" he says something along the lines of: Seek first to understand, then be understood.
Yes, listen with intent. But before you write a single line of code, draw - yes, free-hand - a prototype of (what you think) you heard and share it with the other person - the user, the customer, even a colleague.
Never assume. Never assume a want is The Need. Never assume you understand until you've vetted that understanding.
Community channels are key as you grow. Frankly this is an area where we've underinvested as we've been heads down building/shipping product and is going to be a huge area of focus for us in 2021. The job of the founder is to really to give people who love your product a venue to share their ideas, questions with other users. Customer advisory boards work well too.
1 - https://www.savio.io/customer-feedback/ask-for-user-feedback...
Design thinking talks about being empathetic to your users/customers. Which also sounds like the essence of this post.
Thanks for writing it.
Ideas matter, if your ideas do not include a reflection of current ways people are solving problems you have bad ideas and would be better off thinking and researching before talking to anyone.
I think ideas that matter most for is for founders are the ones around a strong conviction on the market opportunity and then be flexible about the solution that is built to capture that opportunity.
The latter is where I think process is more important than the product solution idea having seen my product solution ideas fail hundreds of times without changing my market vision ideas.
