Use open-ended questions - or miss hearing your customers

This post was written by Dmitry Grenader, Luminoso's Vice President of Product Management. He challenges the status-quo in market-research and voice-of-the-customer surveys where multiple-choice questions dominate the scene and dwarf open-ended questions.

Why do we love multiple-choice questions so much? There are a few reasons.

  • They are easy to create!  Let's say I am surveying folks on their favorite ice-cream flavors - it soooo easy for me to just fill "strawberry, vanilla, chocolate, green tea, ...".
  • They are also easy to take - they are a perfect "don't make me think" task - as a respondent I just pick the one I like and move on.
  • They are easy to analyze - you get your bar-graphs and pie-charts quickly, and the statistically inclined might even look at the mean and (ahem) standard deviation.
  • If distributed to enough people, the results are statistically significant.

What about open-ended questions? Why do we shun them like the Wizarding world of Harry Potter shuns squibs? The main reason is that the analysis is just too damn hard.

Imagine some poor schmuck - sorry, I mean a Product Manager (like me) - trying to analyze results and having to look through 10,000 responses. Impossible! I mean c'mon people - I only get a two-day extension from CEO to research and validate the product viability before we commit like two million dollars to it, and I had to promise my first-born for even that reprieve - I surely don't have time to read what thousands of people said. So what do I do? Well, I read through a few of them, and pick the few juicy comments that support my view of the world plus a couple of negative ones to appear balanced, and copy-n-paste into my powerpoint deck - and call it done. Ok, I don’t really do that, I am sure you don’t either - but you know, those other people do.

Bottomline - multiple-choice questions are the main dish, whilst the open-ended ones are a side-dish at best, an after-thought forever destined to be a lowly “catch-all”. And everyone is ok with it.  Well, I am not!

There are huge problems with this tyranny of the multiple-choice:

  • Options are leading the witness! Let’s face it - when you ask about a favorite ice-cream flavor and you give a choice of five, and the user selects one you have no idea that the user actually would have selected a different one. They have a real favorite - but it was not listed.
  • By pre-creating the choices you are quite literally biasing the user to respond within the framework you have created - so ultimately you are only getting answers within the space of answers you yourself kind of pre-cooked (there is something from quantum theory in this - something about the eye of the beholder and how what you find is determined by how you look for it)
  • They are not allowing your customers to fully express their voice. It is not hard to imagine this internal dialog that happens in your customer’s head when they are asked to choose between strawberry and vanilla ice-cream flavors on the survey: “Well, I kind of like strawberry but I wish it were more aromatic and not as sweet. Vanilla flavor is ok, but I have recently tried this french vanilla by their competitor and it had a hint of pistachios, which I absolutely adored.  I wonder if they can tweak their vanilla to be bolder.” - Wow! Mic drop. No seriously - this is incredible feedback, a voice of the customer at its purest, glorious, unrefined, primordial and incredibly useful state. The kind of feedback that if falls on hearing ears can make the product and company great and create lifelong brand-champions. But NOOOOO, instead this comment gets lost and ends up on the cutting floor by default - ‘cause nobody had time to read it. Or worse nobody dared to even ask the question in the first place.

I hope you are sharing in my righteous indignation, and pounding your fist on the table in outrage and screaming “IS THERE A BETTER WAY?!” The answer is yes.

There is a way to understand the feedback without having to yourself read through the 10,000 responses. You can do that by relying on the text-analytics of a modern voice-of-the-customer platform. The technology has recently come of age where you can rely on machine-learning based approaches to do the feedback reading for you and provide you with insights and the tools to understand what your customers are saying and how they are feeling. Distilling feedback into concepts and topics, finding the connections between them and surfacing the emotional context enable you to quickly and effectively analyze responses. Think of it as “augmented reality” where you can wear a lens with which you can read thousands of responses and understand what is contained in them. And you know what else? This technology can turn qualitative into quantitative - i.e. you can get actual number that are statistically significant on how people feel about various aspects of your business.

This breakthrough allows you to rethink how you construct your surveys and hear more of your customer’s voice. Delve into nuances! Be provocative!  Ask really Open Ended Questions like

What can we do to serve you better, Ms./Mr. Customer?

Do not shy away from the uncertainty - embrace it, you now can with voice-of-customer technology in your corner.  You will be amazed at the results you get.

P.S. As for the multiple-choice questions, keep those too of course - but remember they are not panacea.