The Inclusive Innovation Playbook

Explore

How To Guides

Measure & Improve

"You can't manage what you don't measure."

This statement is becoming increasingly true as organisations strive to become more data-driven. Measuring the impact of your inclusive product development efforts is no different. It is essential to ensure your efforts to drive diversity, foster inclusion and create equity are making a meaningful difference. Metrics provide a way to track progress, identify areas for improvement, and demonstrate the impact of your inclusive product development initiatives.

As a product innovator you may be wondering what a quantitative framework for measuring product inclusion might look like. What metrics are meaningful and how can these metrics be used to drive continuous improvement and increase the impact of your initiatives?

We aim to provide a framework that is simple to implement and can be extended over time as your practice matures.

Metrics are about measuring how well you are doing. So it is important to understand what you are aiming for. When it comes to inclusive digital products and services, essentially you are trying to create a great experience for all users. It also requires creating a diverse and inclusive innovation teams where everyone feels like they belong and can thrive. This means paying specific attention to the experience of marginalised people and making sure everyone feels welcome, safe and empowered.

Once you have a baseline understanding, you can build systems to systematically measure and improve.

Continuous improvement

Inclusive innovation is an ongoing process that requires continuous assessment and adaptation to ensure that products and services are inclusive and equitable for all users. Without continued focus, an inclusive product today, can cause exclusion tomorrow. Establish a system to continuously assess and improve your products and services. Commit to discovering and resolving inevitable inequities. Measure and evaluate the effectiveness of your innovation diversity and inclusion efforts.

Key questions

It's important to use metrics that capture progress and impact across the entire process, from user research to product launch and beyond. Your metrics should aim to help you answer key questions such as:

How diverse is your customer base?

How diverse are the people involved in the product development process, either internal teams or user research groups?

Is your product reaching all of the people you hope it will?

Are features being used as you intended?

Are you providing equal access and opportunity to all users?

How does your product or service experience vary across different aspects of diversity?

Is your product negatively impacting certain groups, more than others?

How will you evaluate impact? How will you ensure this is reflective of what success looks like from the perspective of your users, especially those who have been historically underserved or marginalised?

Diversity, inclusion and equity metrics

In order to determine the most relevant metrics for your circumstances it is important to understand the difference between diversity, inclusion and equity metrics. All are important and you’ll need to consider ease of data capture and value to determine what to track.

Diversity metrics refer to the measurement of the representation of different groups. This could be in your organisation, product team, user research group or customer base. This includes demographic data such as race, gender, age, and disabilities.

Inclusion metrics focus on the experiences and attitudes of diverse individuals within these groups. This includes measures of engagement, satisfaction, and sense of belonging.

Equity metrics go beyond diversity and inclusion, evaluating whether diverse individuals have equal opportunities, treatment, and outcomes. This includes measures of access, impact, and outcomes for underrepresented groups. These metrics help to identify and address systemic barriers and inequalities, ensuring that the benefits of digital products and services are distributed equitably.

Establishing your metrics

You should monitor inclusion in the normal course of product or service development, by setting up key performance indicators and measurable benchmarks. Once these are in place you can communicate your aspirations and set targets to aim for.

Representation metrics

This involves measuring the diversity of different groups of people and enables you to measure equity of access and opportunity. It also helps you measure penetration into new customer segments e.g.

How diverse our are customers? For example, by gender, ethnicity, age, disability, sexuality or other diversity dimensions?

How diverse are our user research panels? For example, by gender, ethnicity, age, disability, sexuality or other diversity dimensions.

How diverse are our staff? For example, by gender, ethnicity, age, disability, sexuality or other diversity dimensions.

Product metrics (by diversity dimensions)

Most companies have product metrics in place. However, in order to track the real world impact of your product or service across your user base it is critical to understand how these product metrics vary by different aspects of diversity such as gender, ethnicity, age and more. This enables you to measure the equity of impact. e.g.

Growth metrics: What overall results is the product or service delivering in terms of revenue, user base, transaction volume or market share and how is this broken down by different diversity groups?

Usage metrics: How are users using the product and how does this vary by different aspects of diversity such as gender and ethnicity?

Adoption and referral metrics: Is the product or service being used as much as we’d hope and in the ways that we’d like by all users?

Health metrics: Is the product available & performing in the manner that users would reasonably expect, even those with network, bandwidth or device limitations?

Experience metrics (by diversity dimensions)

It is important to compliment your product metrics with subjective measures based on explicit feedback on people’s experiences, sense of belonging and overall satisfaction. This provides a measure on the equity of experience across your key stakeholders such as staff, customer base, research panels or wider community. It enables you to quantify a user’s subjective experience and set targets against it. e.g.

What is our customers’ overall sentiment towards the product or service and how does this vary by different diversity groups?

Would our customers recommend our product or service to other people ‘like them’?

What is the feeling of belonging and inclusion within our product teams?

Were people on the user research group able to share their true and authentic experiences and feedback?

What impact, positive or negative, has this experience had on our direct users or others impacted?

Pick a few key metrics and start to break these measures down by different aspects of diversity. For example you might notices that certain features are used more by some groups than others or that onboarding conversion is lower for people using screen readers or that satisfaction is lower for black users. Use your metrics to signal where to focus your inclusion efforts. You can then start to identify steps you can take to create a more inclusive experience for everyone.

If you do not view your product metrics through a diversity lens, you may have a misguided view on the true experience of your product or service.

Balance

Creating inclusive products is a mix of science and art. A balanced approach is needed to steer just the right path for your organisation.

Avoiding exclusion requires you to look at edge cases. Use data and metrics to guide you, but if you only focus on the headline numbers you will miss the nuances of your true user experience.

For example if you rely solely on data driven qualitative insights, you may never understand the true reality of your users from minority groups who you are underserving. Similarly if you focus too much on certain groups of users, you run the risk of excluding others.

Collecting data from people

In order to measure the impact of your diversity and inclusion efforts you’ll need to capture data from people. The

Inclusive Data Collection⁠

section provides a number of considerations to help you do this in an inclusive way.

⁠

Feedback please⁠

: Would you be willing to share the appoach you take to measuring your inclusive innovation efforts? We are looking to create case studies of best practice.

Get in touch⁠

if you’d like to know more.