RCA Questions
Question 1 (Easy):
Your product’s daily active users (DAUs) have dropped by 10% in the last week. How would you diagnose the issue?
Clarifying Questions:
Is this drop consistent across all user segments, or is it specific to a particular demographic, geography, or platform? Were there any recent changes to the product (e.g., new features, bug fixes, UI updates)? Did any external events occur that might have impacted user behavior (e.g., holidays, competitor launches)? Are there any specific times or days where the drop was more pronounced? Are there other metrics (e.g., session length, retention, churn) that also show unusual trends? Possible Causes:
A recent product update introduced bugs or negatively impacted the user experience. A/B tests or experiments caused unintended consequences. A critical feature or functionality is broken. Push notifications or marketing campaigns were paused or reduced. A competitor launched a similar product or aggressive promotion. Seasonal trends or holidays affecting user behavior. Negative press or public sentiment about the product. Broader industry or economic factors (e.g., a global event reducing app usage). Follow-Up Question:
"Can we segment the DAU drop by user cohort (e.g., new vs. existing users, geography, or platform) to identify if the issue is isolated to a specific group?" Question 2 (Medium):
The conversion rate for your e-commerce platform (add-to-cart → purchase) has dropped by 15% in the last two weeks. What could be causing this?
Clarifying Questions:
Is the drop consistent across all product categories or specific to certain ones? Were there any changes to pricing, discounts, or promotions recently? Are there any technical issues in the checkout process (e.g., payment gateway errors, slow page load times)? Did traffic sources change recently (e.g., more users from ads versus organic traffic)? Are there any customer complaints or feedback related to the checkout experience? Possible Causes:
A recent UI/UX change made it harder for users to complete the checkout process. Payment gateway issues or bugs in the checkout flow. A/B testing of pricing or promotions backfired. Product inventory issues (e.g., items showing as out of stock during checkout). Competitors offering better deals or discounts. Economic factors reducing users’ willingness to spend. Seasonal trends (e.g., post-holiday shopping slump). Negative reviews or word-of-mouth affecting trust in the platform. Follow-Up Question:
"Can we analyze user behavior in the checkout funnel to identify where the biggest drop-off is occurring (e.g., payment step, shipping selection)?" Question 3 (Hard):
The retention rate for your app (users returning after 7 days) has dropped by 20% over the past month. How would you approach solving this?
Clarifying Questions:
Is the retention drop specific to a certain user segment (e.g., new users vs. existing users, geography, or device type)? Were there any changes to onboarding or first-use experiences recently? Has user engagement with key features changed (e.g., are users using fewer features or spending less time in-app)? Have there been any external factors (e.g., competitor launches, market trends) that could explain this drop? Are there any patterns in user feedback or complaints that could provide clues? Possible Causes:
Poor onboarding experience for new users. A major feature or functionality is underperforming or broken. Users are not finding value in the product (e.g., unclear value proposition). Push notifications or re-engagement campaigns have been paused. Competitors released a similar product that is drawing users away. Broader market trends (e.g., reduced demand for this type of app). Seasonal or cultural shifts (e.g., users spending more time outdoors during summer). Negative reviews or social media sentiment. Follow-Up Question:
"Can we analyze retention by cohort (e.g., users who signed up in the last 30 days versus older users) to determine if the problem is isolated to new users or affects all users?" Question 4 (Very Hard):
Your product’s Net Promoter Score (NPS) has dropped by 25% over the last quarter. What would you investigate?
Clarifying Questions:
Is the NPS drop uniform across all user segments, or is it concentrated in specific groups (e.g., premium users, free users, specific geographies)? Were there any recent changes to the product, pricing, or customer support policies? What themes or trends are emerging from qualitative feedback (e.g., open-ended NPS responses)? Are there any external factors (e.g., competitor activity, market trends) that could be influencing user sentiment? Have there been changes in how NPS surveys are distributed (e.g., timing, frequency, audience)? Possible Causes:
Product changes that negatively impacted user experience or satisfaction. Poor customer support or unresolved issues leading to frustration. Pricing changes that alienated users. Bugs or performance issues affecting key features. Competitors offering better value or features. Negative press or social media sentiment about the product. Broader economic or market trends reducing user satisfaction. Shifts in user expectations or needs. Follow-Up Question:
"Can we segment NPS responses by user type and analyze qualitative feedback to identify specific pain points driving dissatisfaction?" Summary of Framework:
Start with clarifying questions to gather context and narrow down the scope of the problem. List potential internal and external factors to consider all possible causes. Ask a follow-up question to identify patterns, isolate variables, and prioritize areas for deeper investigation. This framework ensures students develop a structured, hypothesis-driven approach to diagnosing product issues.
Guesstimates
Q1: How many cups of coffee are consumed in New York City every day?
Step 1: Define the problem
Estimate the total number of cups of coffee consumed daily in New York City.
Step 2: Break it down
Population of NYC: ~8 million people. Assume 70% of people drink coffee (5.6 million people). Assume coffee drinkers consume an average of 2 cups of coffee per day. Step 3: Calculate
5.6 million people × 2 cups/day = 11.2 million cups of coffee/day. Step 4: Sense-check
Does this seem reasonable? Coffee is extremely popular in NYC, so this estimate aligns with expectations.
Q2: How many gas stations are there in the U.S.?
Step 1: Define the problem
We want to estimate the total number of gas stations in the U.S., considering factors like population, car ownership, fuel consumption, and gas station capacity.
Step 2: Break it down further
Population and car ownership: U.S. population: ~330 million. Assume there is 1 car for every 2 people → 165 million cars. Fuel consumption per car: Average car drives ~12,000 miles per year. Average fuel efficiency: ~25 miles per gallon. Annual fuel consumption per car: 12,000 ÷ 25 = 480 gallons/year. Total fuel demand: 165 million cars × 480 gallons/year = ~79 billion gallons/year. Assume a typical gas station sells ~2 million gallons of fuel per year. Number of gas stations needed: 79 billion ÷ 2 million = ~39,500 gas stations. Step 3: Sense-check
The U.S. is a large country, and the estimate seems reasonable given the scale of the market. Real-world data suggests ~40,000–50,000 gas stations, so this estimate aligns well.
