Michael outlines the process here that I've seen work well for dozens of user studies that we've done together. Hopefully it'll work for you too.
I would question the value of this advice for startups, particularly very early-stage startups. If you are idea-stage/pre-launch/pre-seed, I think this process is actively harmful, because it encourages the bias toward over-planning and under-doing that most of us already have (knowingly or not). Defining criteria, writing screeners, and scheduling testers all feels very productive, but in many cases, it is actually a form of procrastination -- avoiding that scary moment when you must expose your project to a true outsider's judgement.
Margolis writes: "Startups often point to recruiting users as one of the biggest reasons they’re not regularly talking to their users." I would submit that there is NEVER a good excuse to not talk to users, including "we don't have any users yet" and "we're not quite sure yet who our target user actually is."
The solution to this difficulty is not a 3-step process of screening, recruiting and scheduling testers. It's much simpler, though perhaps more difficult: swallow your shyness and go talk to people as many people as you can. Actually talk, face-to-face, not through a survey. Give your elevator pitch to everyone and anyone, every day, even if it's just to a stranger on the bus or random people at your nearest university campus. You will learn a lot more a lot faster than if you dither around with a more "systematic" process.
When I first started working on one particular startup idea for an entrepreneurship class, my partners and I spent a few days early on designing a survey that we were very excited about; it was going to help us understand people's habits, behavior, and preferences with respect to the service we were going to provide. The instructor for that class was a serial entrepreneur; I showed our survey to him for critique during one of my first office-hours meetings with him. He absolutely tore me apart (rightfully so, in retrospect). He told us we had no business making surveys at all, this early in the process, and he refused to meet us again until we'd talked to at least 50 strangers about our idea. So I did just that, and it was one of the most formative and eye-opening experiences I'd had up to that point.
I'm not saying that the process described in the article is always a form of procrastination, and certainly Google is beyond the point where it's efficient to test by interrupting random strangers on the engineering quad. I'm just saying that most startups are, almost by definition, not at a stage where "user studies" is a productive concept.
It’s true that for many products, some feedback from random people on the bus or street can be better than no feedback. However, user studies accelerate the learning cycle for startups. As I’ve seen over and over with many startups, companies can more quickly learn much more by talking to the right people rather than just button-holing 50 random strangers. And when building products to solve problems for specific users (such as sys admins, bicyclists, travelers, kids, English-language learners, people with email overload, etc.), it seems a little risky not to talk to those users.
