Open-source product playbook

Explore

Measure & iterate

Measuring feature impact

How PMs ensure success from the moment of launch and beyond

Sani Djaya

Ilya Tregubov

Jordan Milner

There is something powerful about having a simple graph that everyone sees every day

Is it up and to the right?

Flat?

In a steep decline?

Or, even worse, you don’t have a graph at all.

I’ve been there.

In some organizations, the success of a feature is unknown to the team, even to the PMs or leadership, because the tracking or tooling wasn’t part of the requirements.

For an early startup, that is OK. You tend to understand your users by simply talking with them, but once you get passed 10 users, you quickly realize you don’t know if users are using your product or simply saying they are using it. Once you 100x your user base to 1000 users, it is impossible to talk to all of them.

Dashboards rally the team

⁠

Screen Shot 2022-12-15 at 12.40.33 PM.png

⁠

Having a dashboard displayed publicly for your team — and the whole company — can be quite a motivating factor. A line graph up and to the right is a great morale boost for the team. You can celebrate and reward everyone for a job well done. A flat or declining graph calls for the team to focus and think of ways to improve it.

This begs the question, what metric do you track? Do you track multiple?

⁠

⭐ Insight 1: There is no universal framework for feature metrics.

Maximizing the time a user spends on your product may make sense for social media platforms, but it may also indicate that it takes a lot of time for your user to get value from your B2B SaaS product. A metric framework that worked for

Google⁠

to measure UX quality may not work for you as a data warehousing product. The popular

Pirate Metrics⁠

(AARRR) may not make sense for an Associate Product Manager in a large organization that is focusing on a narrow piece of the product.

There is no framework that works well in all contexts. You have to decide for yourself what metrics are important in your context.

⭐ Insight 2: Start from the overall business impact.

The key to deciding what to measure is to start with the overall business metric you are trying to impact. This can be a pre-defined

north star metric⁠

, an overall metric that is important to your product area, or simply the metric that will make your manager’s job successful.

Once you’ve decided on the business impact, you can start to think through how the feature you have decided to be built can be measured to see if it moves the needle.

Let’s go through an example.

⁠

Solution

You are working on a B2B SaaS product that provides users with data. You’ve heard from customers that they would like to get this data in their email inboxes on a daily or weekly basis. Let’s assume that you’ve determined that this is a project worth undertaking after

Prioritizing features for the product roadmap⁠

🎉 Awesome! Next you:

🎨 Work with your designer to think of a snazzy solution.

⚙️ Collaborate with your engineering lead along the way to consider technical trade-offs and edge cases.

📄 Keep the marketing and customer service team up-to-date on the progress of the feature.

But what about the success metrics? You should have at least a draft of your metrics at the beginning of the process. As part of creating the Product Requirements Document (PRD)/

PR/FAQ⁠

process.

Keeping in mind

⭐ Insight 2⁠

: you know that you should start with the potential business impact of this feature. Let’s assume that the most important metric for the product team is increasing Weekly Active Users (WAUs).

Based on this, you decide to use a common framework for feature adoption:

TARS⁠

- a framework popularized by

Reforge.⁠

It’s light-weight and comprehensive enough for your needs.

TARS is the acronym that stands for:

Target: all active users interested in using the new feature (it can be 80%, 50%, or even 30% of your all active user base)

Adopted: how many users from the target used the feature, measured by Adopted Users / Target Users

Retained: the share of adopted users who are still coming back to the feature in daily, weekly or monthly routing - depending on your product usage frequency

Satisfied: the share of retained users who are satisfied with the new feature

TARS is a funnel where you can identify some drop-offs at every step of the process and reprise them as long as you reach the level you are satisfied with.

With this framework in mind, you decide to make the following targets.

⁠

@Adopted⁠

: We target that 20% of our users will use this feature at least once within 90 days of launch.

⁠

@Retained⁠

: We target 40% of our users who started using the feature will still be using it 90 days later.

⁠

@Satisfied⁠

: After 90 days of the launch of our feature, our target satisfaction rate is 80%

Get the template

TARS metrics template Connect your data sources to your collaborative doc.⁠

Gallery

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.