Where should you start your user research? How will you uncover meaning from the data? In this post, I’ll illustrate how you can begin with a short open-ended survey and a research synthesis method called thematic analysis. It’s quick to start this process, but it is iterative, meaning you work through the data in several passes. That’s why it’s time-consuming and potentially inconsistent. Even with those caveats, it can be a great jumping-off point for improving research outcomes in the future.
Thematic analysis can help shape future research
When you want to survey users, you have to be quick. Ideally, you will prompt for a response as close to the experience as possible. Surveys also have to be short. A simple survey could be a quick free-trial exit survey, or it could be a question to capture reactions to a preview of a new feature. Should you provide a few options? Or keep it open-ended?
You’d be rightfully hesitant about using a multiple choice question before you even know what the options should be. Thematic analysis of a set of data from open-ended questions can help your design follow up surveys. Open-ended questions allow users to reply in their own language, without limiting responses with inadequate options. Later, you can create multiple-choice questions with options based on your findings, and speed up responses.
Thematic analysis is a qualitative method for data analysis that helps you discover patterns emerging in these unstructured responses.
Through this process, you apply a code to categorise similar responses together.
The codes are usually grouped in themes so you can connect the patterns.
A report summarises the findings under each theme and suggests further research design.
The main caveat with this method is that coding responses is time-consuming. It takes a while to uncover what you should be capturing and it takes multiple passes. You will start to see patterns emerge that you might not have expected, and codes need to be updated, added, or split. That’s why it would be quite a lot of work to hand-code open-ended questions on a long-running survey.
However, it’s a great way to get started exploring user research on a targeted sample. From the findings, you can make recommendations for structured surveys, user interviews, product experiments, competitor analysis, and sales or marketing.
A note about writing good survey questions
Avoid leading questions which assume a positive or negative sentiment, or assume irrelevant experience or context.
Briefly remind respondents about the purpose of the question and define any terms.
Consider allowing people to opt-out of answering with an explicit option such as “I prefer to not respond” so you can also capture non-respondents and avoid forcing people to supply an answer, and therefore irrelevant data.
Test test test - Probably the most important step in survey design is to pilot the survey with a small sample of respondents and ask them for feedback! You can also tell from the pilot responses if you’re getting useful data.
How does thematic analysis work?
Thematic analysis involves reviewing data and findings themes in the responses. This can be an
For example, when you analyse responses to a free-trial exit survey, you can count the number of times a product feature or certain incidents are mentioned. However, word occurrence alone isn’t usually much of an indicator. Someone may mention “set up” but you don’t know what’s that referring to without context. You will have to infer from the context what the respondent is talking about.
You may also notice that customers use different terminology than you. For example, they may talk about their onboarding experience but say, “It was hard to know where to start,” or “I found the initial experience confusing.” You can code both as “onboarding.”
This process is iterative. You won’t get through the analysis in one pass, iterating through is part of how you discover the emerging patterns. Soon you’ll find where your codes aren’t working. Maybe there are just so many related responses that you have to split one code out into two. This means you have to iterate through and update your analysis as you make improvements.
shows how you might even do this on paper, marking up phrases, applying shorthand codes, or even using sticky notes.
However, I find it just as easy to use a spreadsheet, with one column per code to count occurrences. This makes it easy to calculate as you go.
I set up the spreadsheet like this:
The first tab in the table has all of the codes, their definitions, and themes defined.
Each tab (illustrated in the table below) would include all responses to one question, so if you have multiple open-ended questions to analyse, put them each on their own tab.
Each row is a single response.
The column headings are labelled with the codes.
I add a 1 to that cell per response under each code when a response relates to that topic. That means I can sum up the columns quickly.
Every response will likely have multiple codes.
Example tab per question
The package was confusing to set up. I didn’t hear back from the support team.
I found installation easy, but then I couldn’t make use of it due to a bug.
There are no rows in this table
Report on your findings
When you’re finished with the analysis you can write a summary of findings and recommendations. The narrative of your findings should answer your original research question.
You should start by explaining briefly how you conducted the analysis. Explain who makes up the sample of respondents, how the survey was run, how they accessed the survey, and for how long the survey ran. Indicate how many incomplete or non-responses too.
Next, summarise each theme and the patterns that emerged, with evidence of their significance.
Indicate the number of respondents who mentioned X or Y. Keep in mind though that frequency of terms isn’t the only measure. You may want to weigh each response to capture positive/negative sentiment.
Use direct quotes, but make sure to anonymise the response. Direct phrases and narratives from customers can have a great impact, particularly on sales and marketing.
You should be able to say “X is important because of these reasons.” The main goal is to demonstrate to the reader that you didn’t just make stuff up and you have evidence to prove it!
Conclusions and caveats
One concern about this research method is that you could end up with inconsistent results or missed opportunities based on how you code the responses. That’s why you will corroborate your findings with follow-up interviews, user observations, analytics data, competitor research, etc.
That’s the most important part of your conclusion in this thematic analysis. You should indicate what research should be done next.
You can even use the customer’s responses to help you with phrasing questions or which multiple-choice options you should provide for later surveys. Or you can even use responses or phrasing to conduct a content analysis in customer support channels or in SEO research.
In summary, thematic analysis is good for…
When you need the flexibility to initiate a research project.
When you want a springboard for further research, such as designing surveys or writing interview scripts.
But the drawbacks are that it’s…
Time-consuming if you have large data sets.
Biased by limited or incomplete data.
Known for inconsistent findings. However, these can be corroborated by other data sources.
As I mentioned before, you can purchase tools to make this easier, but nothing should stop you from using a simple spreadsheet or even paper to get started.