Journalism Product: Tools and Resources
Share
Explore
Journalism Product: Tools and Resources
Supplemental Resources

icon picker
Experiment-Driven Product Development - Experiment Design Cheatsheet

Experiment-Driven Product Development

Paul Rissen (@r4isstatic), Springer Nature - SRC:CON Product 2020

Generating the raw material for experiments

Assumptions
Assumptions can range from accepted industry wisdom to assumptions about the demographics, behavior, and beliefs of our audience.
Ideas
Rather than seeking to ‘validate’ your proposed design solutions via experimentation, take a step back and test the premises that underlie them.
Claims
What claims do we, or others within and outside the organisation, make about our product (both positive and negative)? How can we find out whether those claims have any weight?
Knowledge Gaps
What are the known unknowns - the identifiable gaps in our knowledge? Where do we have simply no idea what the answer to a question might be? What impact might discovering an answer have?
⁠

Two flavours of experiment-ready question


Belief-led questions where you have a prior belief, premise, or assumption that you want to test against reality. These tend to either test a statement about the state of the world, or a statement about a proposed intervention.
Exploratory questions which seek to gather knowledge where you have no existing beliefs, premises, or assumptions. They often have a lurking belief behind them - that there is something to be discovered. Despite these kinds of questions not relying on a hypothesis, it’s still important to carefully design an experiment around them.
⁠

Formulating Hypotheses

A hypothesis should be the presumed outcome of your experiment. It often describes a difference or change between two or more conditions. It should be falsifiable, testable and measurable - though this last criteria doesn’t mean it has to be quantifiable.
At an absolute minimum, a hypothesis statement should contain three things:
We believe the answer is <our belief>,
The evidence for which would be reflected in <our measures>,
Under the circumstances of <our conditions>.
Remember: An experiment seeks to discover an answer. When trying to answer belief-led questions, the answer we discover will determine whether there is enough evidence to abandon the null hypothesis and instead accept our prior belief.
⁠

Selecting Measures

How would you expect your answer to manifest in the world, in a way that is possible to detect?
Track only the things you need to track in order to answer the question, and remove any coded tracking once the experiment is over. Think about your health metrics - what should you be keeping an eye on, to ensure your experiment hasn’t lead to lasting harm, either to your product or to your audience?
⁠

Choosing Conditions

Conditions represent the motivating factor behind the answer to the question - the why.
Determine conditions based on your hypothesised reasons for your prior belief, if you have one. For exploratory questions, think about the characteristics or categories that would define whether or not something qualifies as an object of study for your experiment.
⁠

Determining Scale

Scale - how much evidence you need in order to find a useful, meaningful, applicable answer - depends on two things - how certain you need to be (how much you care about errors creeping in), and how precise (how granular you need the answer to be).
The minimum length of an experiment is determined by the amount of evidence you need to collect, which in turn is determined by your chosen level of certainty & precision.
Statistical power is the probability that if evidence to support your hypothesis exists, you’ll find it. The higher the power, the lower the chances are that you’ll mistakenly stick with the null hypothesis.
Significance is the probability that what you detect is down to chance. That is to say, you see an effect where in reality, it doesn’t actually exist. The lower the significance level, the less chance of detecting misleading evidence, i.e. the more stringent your standard of certainty.
Remember: NO PEEKING at the results until the minimum amount of evidence has been collected!
⁠

Deciding a Method

Choose the method once you’ve determined all the above. Don’t fixate on a ‘Minimal Viable Product’ to validate your proposed solution. Instead, ask yourself - what’s the simplest, useful thing we could do in order to answer the question?

Gallery
Share
 
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.