Stop listening to your customers

Nate Bolt is co-founder and chief executive of user research firm Bolt Peters. With Tony Tulathimutte, he is the author of the book Remote Research, which is out today.
A common assumption among startup entrepreneurs is that listening to potential customers is the best way to find out whether your product or idea will succeed in the market. Honestly — don’t bother. In our nine years of research consulting for startups and big companies alike, one thing we’ve seen time and again is that it’s user behavior, not opinions, that tells you whether people want to use your product.
The main problem with opinions is self-reporting bias: Opinions are often inconsistent with behaviors or other attitudes, especially when discussing hypotheticals. Remember Clippy, the little character that appeared in Microsoft Word years ago? That little bastard arose, in part, from Microsoft asking users if they wanted help working on their documents — everyone said, “Sure, sounds great.” But once people started actually using it in the real world, they hated it — it might be one of the most hated features in the history of computing. But Microsoft employs hundreds of researchers. So where did they go wrong, and how can you avoid making the same mistake?
It’s simple. Never ask people what they think of your product or idea.
Back in 2004, we worked with a web startup that allowed users to search online for products sold locally. The users we spoke to were unanimously in love with the idea, the look-and-feel received high praise, and most said they would recommend the site to their friends.
Their behavior, however, demonstrated major problems with the main product search, which had critical parsing errors and arcane syntactic rules: For example, searching for “x box” yielded a mess of results containing the letter “x”. It was awful to use. The founders took the research to be just one interesting data point, instead of as a warning sign that their business concept was built on a false premise: that they could eventually tweak the results to work at some later point. That never happened. Now they’re gone.
Here are three things you can do to get a sense of whether people would use your service.
1. Test ideas early by watching behavior. It’s fine if you don’t have a 100 percent functional interface — having eight people interact with a prototype or even wireframes or design mockups can be incredibly useful. Even recruiting strangers from the street to use your prototype is better than nothing. You can still ask all your needy questions about what they think after the session — just don’t take those too seriously. Recently there’s been lots of tools that let you quickly create and test prototypes, mockups, and comps online: Chalkmark, Pidoco, Balsamiq, Loop11, OpenHallway, Usabilla.
Even during design conceptualization, paying attention to existing user experiences can be invaluable. The award-winning finance site Mint (one of our clients) was born out of frustration with Quicken, which required users to manually log every transaction they made, across multiple accounts. It’s worth noting that Mint doesn’t do much usability research — the emphasis isn’t on whether the service is easy to use, but what needs people really have.
2. Get all stakeholders to watch the research. When more than three people are involved in a tech project, they cease to be normal people and instead become “stakeholders”. Unfortunately, the stake they’re holding is the one that goes right through the heart of user experience — technical and business constraints obscure the basic question of whether the interface is any good.

One way to get everyone’s priorities in order is to force everyone — engineers, designers, executives, sales people, anyone else who’s involved — to watch the user research sessions together. Force them. It helps everyone agree on what needs to be addressed in the design.
3. Use unorthodox methods. Companies like Apple and 37signals make a big deal about never conducting user research. They lie — they just don’t call it user research. Releasing products in generations, like Apple does, provides them with mountains of reviews, task-specific complaints, crash reports, customer support issues, and Genius Bar feedback. It’s audacious, large-scale behavioral research. The same goes for services with long beta testing periods (i.e. Gmail): Their logs and analytics constitute real-world behavioral data. Did you know Flickr started out as an image management tool inside an online game? Once they realized how good it was, they scrapped the game and focused on the good part. That is hardcore user experience insight matched with entrepreneurial insight.
Do whatever you need to do to understand how people use your product. If it’s a device meant to be used in cars, watch people use it cars; if it’s a video game, avoid sterile lab environments. Just don’t ask perfunctory, cookie-cutter survey questions to your potential customers, and expect that to ensure your product’s usefulness.