would correlate well with the North Star metric, for which changes be detected more quickly?
Trade-offs of that metric vs other metrics (avoid gaming)
Why might the North Star metric you picked be a bad one? Or proxy metrics? Layer trade-offs and critical thought throughout
"Trade-off" questions (eg: "How would you decide between doing X or Y?")
What is the trade-off (plug in infinite to both sides)
If you're weighing between two things, imagine you only did one of the forever and went all in on it. What positive things would happen short term, but what might not go well long-term?
Find the common thread between them both
eg: Facebook: ad versus organic engagement module both long-term optimize for money
eg: Lyft: Getting one more Driver hour versus getting a Rider to book one more ride both long-term optimize for most aggregate rides
Objective function (goal to meet through experimentation)
The right point in the trade-off is what will maximize the common long-term goal that both sides you're considering have
You can run experiments with different levels of both sides and whichever mix is best for your long-term goal is the right mix!
Further optimize (network effect, personalization, etc.)
The above is a starting point. You can always better optimize over time by having different mixes for different circumstances
eg: Facebook: for highly-engaged users who aren't ad-sensitive, can show more ads
eg: Lyft: in a region where there are too many Drivers, might have less need for one incremental hour from a Driver
"Root cause analysis" questions (eg: "XY metric is down. What happened?")
The goal here is to systematically explore all reasonable explanations that could have contributed to the metric changing
that cover areas like specific technical attributes, specific funnel step, region, time frame, product/feature change, and demographic
There's no right answer. There's no root cause that you have to get to pass. The goal is breadth so don't worry at all if you never find the "correct answer"
In fact, when I ask candidates this question, if they ever guess the "correct" root cause I had in mind, I just change it to keep them going 😈