Why do we love multiple-choice questions so much? There are a few reasons.
- They’re easy to create. Let’s say I’m surveying folks on their favorite ice-cream flavors – it’s soooo easy for me to just fill in answers: strawberry, vanilla, chocolate, green tea, etc.
- They’re easy to answer. Multiple choice questions are a perfect “don’t make me think” task – as a respondent, just pick the answer you like, and move on.
- They’re easy to analyze. You get your bar graphs and pie charts quickly, and the statistically inclined might even look at the mean and (ahem) standard deviation. And if distributed to enough people, the results are statistically significant.
What about open-ended questions? Why do we shun them like the Wizarding World of Harry Potter shuns squibs? The main reason is that the analysis is just too damn hard.
Imagine some poor schmuck – sorry, I mean a Product Manager (like me) – trying to analyze results and having to look through 10,000 responses. Impossible! I mean c’mon people – I only get a two-day extension from my boss to research and validate the product viability before we commit like two million dollars to it, and I had to promise my first-born for even that reprieve – I surely don’t have time to read what thousands of people said. So what do I do? Well, I read through a few of them, and pick the few juicy comments that support my view of the world plus a couple of negative ones to appear balanced, and copy-n-paste into my powerpoint deck – and call it done.
Ok, I don’t really do that, I am sure you don’t either – but you know, those other people do.
The bottom line: multiple choice questions are the main dish, whilst the open-ended ones are a side dish at best, an afterthought forever destined to be a lowly “catch-all”. And everyone is ok with it.
Well, we at Luminoso are not.
There are huge problems with this tyranny of the multiple-choice:
- Options are leading the witness. Let’s face it: when you ask about a favorite ice cream flavor and you give a choice of five, and the respondent selects one, you have no idea if they actually would have selected a different one. They have a real favorite – but there’s no guarantee it was one of the ones listed.
- Bias. By pre-creating choices you are quite literally biasing respondents to answer within the framework you have created. Ultimately, you’re only getting answers within the space of answers you yourself kind of pre-cooked. (There is something from quantum theory in this … something about the eye of the beholder and how what you find is determined by how you look for it.)
- Limited Voice of the Customer. Multiple choice questions don’t allow respondents to fully express their voice. It is not hard to imagine this internal dialogue that happens in someone’s head when asked to choose between strawberry and vanilla ice cream flavors:
Well, I kind of like strawberry but I wish it were more aromatic and not as sweet. Vanilla flavor is ok, but I have recently tried this french vanilla by their competitor and it had a hint of pistachios, which I absolutely adored. I wonder if they can tweak their vanilla to be bolder.
Wow! Mic drop. Seriously – this is incredible feedback, a voice of the customer at its purest, glorious, unrefined, primordial and incredibly useful state. The kind of feedback that if falls on hearing ears can make the product and company great and create lifelong brand-champions.
But with multiple choice questions, this comment gets lost and ends up on the cutting floor by default – ‘cause nobody had time to read it. Or worse, nobody dared to even ask the question in the first place.
I hope you’re sharing in my righteous indignation, and pounding your fist on the table in outrage and screaming “IS THERE A BETTER WAY?!” The answer is yes.
There is a way to understand the feedback without having to yourself read through the 10,000 responses. You can do that by relying on the text analytics of a modern VoC platform. The technology has recently come of age where you can rely on machine learning-based approaches to do the feedback reading for you and provide you with insights and the tools to understand what your customers are saying and how they are feeling. Distilling feedback into concepts and topics, finding the connections between them and surfacing emotional context enable you to quickly and effectively analyze responses. Think of it as “augmented reality” where you can wear a lens with which you can read thousands of responses and understand what is contained in them. And you know what else? This technology can turn qualitative into quantitative: i.e. you can get actual number that are statistically significant on how people feel about various aspects of your business.
This breakthrough allows you to rethink how you construct your surveys and hear more of your customer’s voice. Delve into nuances! Be provocative! Ask really open-ended questions like:
What can we do to serve you better?
Don’t shy away from the uncertainty – embrace it, and use it to your advantage. You will be amazed at the results you get.
P.S.: As for the multiple choice questions, keep those too, of course – there’s a time and a place for them. But remember, they are not panacea.
