Explore

Pilot your next study with thematic analysis

An easy method to kickstart user research.

⁠

Where should you start your user research? How will you uncover meaning from the data? In this post, I’ll illustrate how you can begin with a short open-ended survey and a research synthesis method called thematic analysis. It’s quick to start this process, but it is iterative, meaning you work through the data in several passes. That’s why it’s time-consuming and potentially inconsistent. Even with those caveats, it can be a great jumping-off point for improving research outcomes in the future.

⁠

Thematic analysis can help shape future research

When you want to survey users, you have to be quick. Ideally, you will prompt for a response as close to the experience as possible. Surveys also have to be short. A simple survey could be a quick free-trial exit survey, or it could be a question to capture reactions to a preview of a new feature. Should you provide a few options? Or keep it open-ended?

You’d be rightfully hesitant about using a multiple choice question before you even know what the options should be. Thematic analysis of a set of data from open-ended questions can help your design follow up surveys. Open-ended questions allow users to reply in their own language, without limiting responses with inadequate options. Later, you can create multiple-choice questions with options based on your findings, and speed up responses.

Thematic analysis is a qualitative method for data analysis that helps you discover patterns emerging in these unstructured responses.

Through this process, you apply a code to categorise similar responses together.

The codes are usually grouped in themes so you can connect the patterns.

A report summarises the findings under each theme and suggests further research design.

The main caveat with this method is that coding responses is time-consuming. It takes a while to uncover what you should be capturing and it takes multiple passes. You will start to see patterns emerge that you might not have expected, and codes need to be updated, added, or split. That’s why it would be quite a lot of work to hand-code open-ended questions on a long-running survey.

However, it’s a great way to get started exploring user research on a targeted sample. From the findings, you can make recommendations for structured surveys, user interviews, product experiments, competitor analysis, and sales or marketing.

A note about writing good survey questions

Avoid leading questions which assume a positive or negative sentiment, or assume irrelevant experience or context.

Briefly remind respondents about the purpose of the question and define any terms.

Consider allowing people to opt-out of answering with an explicit option such as “I prefer to not respond” so you can also capture non-respondents and avoid forcing people to supply an answer, and therefore irrelevant data.

Test test test - Probably the most important step in survey design is to pilot the survey with a small sample of respondents and ask them for feedback! You can also tell from the pilot responses if you’re getting useful data.

How does thematic analysis work?

Thematic analysis involves reviewing data and findings themes in the responses. This can be an

inductive or deductive approach⁠

where you either let the data decide what the codes should be, or you can seek information based on your original research objectives.

First, you’ll review all of the data and identify the initial codes and themes.

A code is a shorthand label that represents a way to connect and categorise responses.

These codes are linked together in themes that you discover.

Next, you begin to mark up responses with codes in the first pass. This involves identifying a phrase or sentence in response and marking it with a code and a theme.

Here’s

an example, from an academic paper⁠

showing how responses are coded and calculated in this kind of analysis.

Example showing how raw data is coded

Example showing how occurrences are calculated

⁠

From

Example of Inductive Content and Thematic Analysis

⁠

There are no rows in this table

⁠

For example, when you analyse responses to a free-trial exit survey, you can count the number of times a product feature or certain incidents are mentioned. However, word occurrence alone isn’t usually much of an indicator. Someone may mention “set up” but you don’t know what’s that referring to without context. You will have to infer from the context what the respondent is talking about.

You may also notice that customers use different terminology than you. For example, they may talk about their onboarding experience but say, “It was hard to know where to start,” or “I found the initial experience confusing.” You can code both as “onboarding.”

This process is iterative. You won’t get through the analysis in one pass, iterating through is part of how you discover the emerging patterns. Soon you’ll find where your codes aren’t working. Maybe there are just so many related responses that you have to split one code out into two. This means you have to iterate through and update your analysis as you make improvements.

Tools for thematic analysis

There are

Digital Ethnography Tools⁠

such as

Delve⁠

- see

thematic analysis with Delve⁠

, though it might be out of budget. This article

How to Do a Thematic Analysis of User Interviews⁠

shows how you might even do this on paper, marking up phrases, applying shorthand codes, or even using sticky notes.

However, I find it just as easy to use a spreadsheet, with one column per code to count occurrences. This makes it easy to calculate as you go.

I set up the spreadsheet like this:

The first tab in the table has all of the codes, their definitions, and themes defined.

Each tab (illustrated in the table below) would include all responses to one question, so if you have multiple open-ended questions to analyse, put them each on their own tab.

Each row is a single response.

The column headings are labelled with the codes.

I add a 1 to that cell per response under each code when a response relates to that topic. That means I can sum up the columns quickly.

Every response will likely have multiple codes.

Example tab per question

Example tab per question

Reponse

Installation

Support

Bug

The package was confusing to set up. I didn’t hear back from the support team.

I found installation easy, but then I couldn’t make use of it due to a bug.

There are no rows in this table

⁠

Report on your findings

When you’re finished with the analysis you can write a summary of findings and recommendations. The narrative of your findings should answer your original research question.

You should start by explaining briefly how you conducted the analysis. Explain who makes up the sample of respondents, how the survey was run, how they accessed the survey, and for how long the survey ran. Indicate how many incomplete or non-responses too.

Next, summarise each theme and the patterns that emerged, with evidence of their significance.

Indicate the number of respondents who mentioned X or Y. Keep in mind though that frequency of terms isn’t the only measure. You may want to weigh each response to capture positive/negative sentiment.

Use direct quotes, but make sure to anonymise the response. Direct phrases and narratives from customers can have a great impact, particularly on sales and marketing.

You should be able to say “X is important because of these reasons.” The main goal is to demonstrate to the reader that you didn’t just make stuff up and you have evidence to prove it!

Conclusions and caveats

One concern about this research method is that you could end up with inconsistent results or missed opportunities based on how you code the responses. That’s why you will corroborate your findings with follow-up interviews, user observations, analytics data, competitor research, etc.

That’s the most important part of your conclusion in this thematic analysis. You should indicate what research should be done next.

You can even use the customer’s responses to help you with phrasing questions or which multiple-choice options you should provide for later surveys. Or you can even use responses or phrasing to conduct a content analysis in customer support channels or in SEO research.

In summary, thematic analysis is good for…

When you need the flexibility to initiate a research project.

When you want a springboard for further research, such as designing surveys or writing interview scripts.

But the drawbacks are that it’s…

Time-consuming if you have large data sets.

Biased by limited or incomplete data.

Known for inconsistent findings. However, these can be corroborated by other data sources.

As I mentioned before, you can purchase tools to make this easier, but nothing should stop you from using a simple spreadsheet or even paper to get started.

References

⁠

Thematic Analysis: Striving to Meet the Trustworthiness Criteria⁠

⁠

Heather McNamee

B2B Product Manager facilitating collaboration, communication, and knowledge sharing

⁠

https://www.linkedin.com/in/heatherjames/⁠

⁠

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.