Explore

Experiment reads

v1 launch RCA

Metric type

Metric

Test - W/O FTUX

Significance

Platform

Gross orders / view

+1.04%

Platform

Gross views / visitor

-1.64%

Platform

Orders / clicks

+0.70%

Platform

Gross orders / unique visitors

-0.61%

Platform

GMV / Vi

-0.62%

Platform

Net orders / Vi

+2.11%

Platform

NMV / Vi

+2.32%

Platform

Net AOV

-0.009%

Platform

Ad revenue per user

-2.65%

Platform

Plat T/Vi

-0.56%

Gold

Gross orders / view

-9.9%

Gold

Gross views / visitor

+14.64%

Gold

Gross orders / Vi

+3.27%

Gold

GMV / Vi

+1.84%

Gold

NMV / Vi

-8.3%

Gold

Gold T/Vi

+2.5%

Gold

Gold VC

+18.25%

Gold

Gold OC

+4.07%

Gold

Gold T/Vi (Plat)

+2.5%

Gold

Gold T/Vi (Reco)

+7.30%

Gold

Mall

Gross orders / view

-4.45%

Mall

Gross views / visitor

+7.32%

Mall

Gross orders / Vi

+2.55%

Mall

GMV / Vi

+3.45%

Mall

NMV / Vi

+10.86%

PDP reco metrics

Gross orders / view

+1.44%

PDP reco metrics

Gross views / visitor

-4.81%

PDP reco metrics

Gross orders / Vi

-3.43%

PDP reco metrics

AOV

+0.25%

PDP reco metrics

GMV / Vi

-3.19%

PDP reco metrics

NMV / Vi

-4.67%

Usage metrics

User level CTR - % test users who’ve opened parallel feed once post visiting reco section

Usage metrics

Event / Net CTR for parallel feed - (All parallel feed opens) / (Total PDP reco scrolls) for test

3.4%

Usage metrics

Event / Net CTR for Gold parallel feed - (Gold parallel feed opens) / (Total PDP reco scrolls) for test

Usage metrics

Event / Net CTR for Mall parallel feed - (Mall parallel feed opens) / (Total PDP reco scrolls) for test

1.4%

Usage metrics

Distribution of clicks vs swipes for parallel feed opens in test

Clicks - 67%

Swipes - 33%

Usage metrics

Number of times FTUX seen

⁠

Summary of Experiment Data for Real Estate Placements

The table provides data comparing the Control and Test groups across different Real Estate (RE) placements within an app. Metrics such as viewers, views, and delta values help understand the differences in performance between these groups.

Column Descriptions:

RE:

Represents different real estate sections of the app where the product or feature is displayed. The sections analyzed are:

PDP_Reco: Product Detail Page Recommendations.

CLP_collections: Collection List Page.

Search: Search results page.

FY_feed: Personalized "For You" feed.

Control_viewers:

The number of unique viewers interacting with the control version for each real estate section.

Control_views:

The total number of times the real estate was viewed in the control version.

Test_viewers:

The number of unique viewers interacting with the test version for each real estate section.

Test_views:

The total number of views in the test version.

delta:

Represents the percentage change between the control and test versions for views, highlighting the differences in engagement or interaction for each real estate section.

Summary by Real Estate:

PDP_Reco (Product Detail Page Recommendations):

Control Viewers: 6,508,782

Test Viewers: 6,357,763

Delta: -8.69%: There is a significant decline in the number of views, indicating a drop in engagement or interest in the recommendation section.

CLP_collections (Collection List Page):

Control Viewers: 5,929,225

Test Viewers: 5,935,594

Delta: 0.51%: A small positive increase in views, indicating a marginal improvement in engagement for the collection list page.

Search:

Control Viewers: 5,429,040

Test Viewers: 5,430,258

Delta: 0.56%: Shows a slight improvement in engagement with the search feature, with more views in the test group.

FY_feed (For You Feed):

Control Viewers: 4,483,664

Test Viewers: 4,490,197

Delta: 0.86%: The "For You" feed experienced the highest positive delta among the sections, indicating improved engagement and interaction in the test group.

General Insights:

PDP_Reco experienced a notable decline with a delta of -8.69%, suggesting that the changes made to this feature have led to reduced engagement, and further investigation may be needed to understand why.

CLP_collections, Search, and FY_feed have all shown positive deltas, with the highest improvement seen in the FY_feed (0.86%). This suggests that users responded positively to changes in these areas, though the improvements are relatively small.

Conclusion:

The table highlights the impact of changes in different app sections by comparing control and test metrics. The Product Detail Page Recommendations needs attention due to a significant decline in views, while other areas like the For You Feed show moderate improvements. This data is useful for focusing on optimizing specific sections to improve overall user engagement.

Summary of Browsing Behavior Data

The data tables compare the browsing behavior across three conditions: PF_browsed, PF_not_browsed, and Control. The metrics recorded are V/Vi (likely Views per Visitor or a similar engagement metric) and scroll_depth (indicating how far users scroll within the page).

Table Breakdown:

PF_browsed:

This condition represents a group where a certain feature (perhaps "PF") is actively browsed by users.

Columns:

date: Represents the timeline from 2024-08-30 to 2024-09-08.

V/Vi: Shows fluctuations in engagement metrics for each date.

scroll_depth: Represents how far users are scrolling within the page on average.

Key Trends:

V/Vi values peak on 2024-09-01 at 436.29 and then gradually decrease to 354.15 by 2024-09-08.

scroll_depth starts at 45.04 on 2024-08-30, drops to 43.12 on 2024-09-08, with minor fluctuations during the period.

PF_not_browsed:

This condition represents a group where users do not browse the "PF" feature.

Columns:

date: Timeline from 2024-08-30 to 2024-09-08.

V/Vi: Engagement metrics for users who did not browse the "PF" feature.

scroll_depth_2: Indicates average scroll depth for non-browsers.

Key Trends:

V/Vi shows a gradual decline from 188.93 on 2024-08-30 to 160.14 on 2024-09-08.

scroll_depth_2 starts at 39.21 on 2024-08-30 and ends at 37.38 on 2024-09-08.

Control:

This table represents the Control condition where there were no modifications related to the "PF" feature.

Columns:

date: Timeline from 2024-08-30 to 2024-09-08.

V/Vi: Engagement metric in the control condition.

scroll_depth: Average scroll depth for the control group.

Key Trends:

V/Vi generally fluctuates between 250 to 280 across the timeline, with a peak at 281.45 on 2024-09-01.

scroll_depth shows some variation, starting at 42.53 on 2024-08-30 and ending at 41.65 on 2024-09-08.

General Observations:

Engagement (V/Vi):

PF_browsed has higher engagement levels compared to PF_not_browsed and Control throughout the period.

Peak engagement for PF_browsed is on 2024-09-01 at 436.29, whereas PF_not_browsed has significantly lower values, ending at 160.14 on 2024-09-08.

The Control group’s engagement levels are in between, peaking at 281.45 on 2024-09-01.

Scroll Depth:

Users in the PF_browsed condition tend to scroll deeper compared to the PF_not_browsed group, indicating higher engagement with the content.

Control scroll depth is generally in the middle, starting at 42.53 and ending at 41.65.

PF_not_browsed users have the shallowest scroll depth, indicating lower engagement, with values declining from 39.21 to 37.38.

Conclusion:

PF_browsed group exhibits the highest engagement levels (V/Vi) and scroll depth, suggesting that the "PF" feature is effective in capturing user interest.

PF_not_browsed consistently shows the lowest engagement and scroll depth, indicating a lack of interest in the content or inability to find what they need.

Control sits between the two experimental groups in terms of engagement and scroll depth, providing a useful baseline for comparison.

This data highlights the positive impact of users browsing the "PF" feature, with higher engagement metrics and deeper scroll behavior compared to those who did not interact with the feature or were in the control condition.

Summary of DOD Viewer Delta of View Buckets with Respect to Control

The graph represents the Day-over-Day (DOD) viewer delta for different view buckets compared to a control group. The line chart tracks the delta changes from 2024-08-26 to 2024-09-05.

Graph Elements:

X-axis (Date):

The timeline ranges from 2024-08-26 to 2024-09-05, depicting how the viewer delta for each view bucket changes over time.

Y-axis (Delta Percentage):

The Y-axis represents the percentage change in viewers relative to the control group.

Values range from -7.5% to +5%, with positive values indicating an increase compared to the control and negative values indicating a decline.

View Buckets Represented by Lines:

a. <5 (Blue Line):

Represents the delta for viewers who viewed less than 5 items.

Starts around 0% on 2024-08-26 and increases consistently, reaching about 3% by 2024-09-05.

b. 40+ (Red Line):

Represents viewers who viewed more than 40 items.

Starts near 0% on 2024-08-26 and gradually declines, dropping to around -7.5% by 2024-09-05.

b. 5-20 (Yellow Line):

Represents viewers who viewed 5 to 20 items.

Starts slightly above 0% and gradually increases, reaching about 2.5% by 2024-09-05.

c. 20-40 (Green Line):

Represents viewers who viewed 20 to 40 items.

The line remains relatively flat around 0%, indicating minimal change throughout the observation period.

Key Observations:

Positive Delta for Low and Moderate View Buckets:

<5 (Blue Line) and 5-20 (Yellow Line) segments exhibit a positive delta trend over the given timeline.

<5 views reach approximately 3% positive delta by 2024-09-05, indicating increased engagement among users who typically view a small number of items.

5-20 also shows an upward trend, with a 2.5% increase in viewer engagement.

Negative Delta for High Viewers (40+ View Bucket):

The 40+ (Red Line) bucket consistently declines, ending at around -7.5%.

This indicates a notable drop in engagement among users who previously viewed more than 40 items, suggesting they might be disengaging or experiencing a negative effect due to changes.

Stable Behavior for Moderate View Bucket (20-40):

The 20-40 (Green Line) bucket remains mostly unchanged, with a delta around 0%.

This indicates stable engagement for users who typically view 20 to 40 items, with neither a significant increase nor decrease in their behavior.

General Conclusion:

The <5 and 5-20 view segments show positive increases in viewer delta, suggesting improved engagement among users who typically have moderate to lower levels of interaction.

The 40+ view segment experiences a sharp decline in engagement, which could indicate issues affecting frequent viewers, such as changes that make browsing more cumbersome or less rewarding for users who typically engage deeply.

The 20-40 segment remains stable, showing no significant change in user behavior over the period.

This data is useful for identifying which user segments are positively or negatively affected by changes in the platform. The decline in the 40+ segment might need further analysis to identify reasons for the drop, while the growth in the <5 and 5-20 segments could represent a positive effect of recent changes that make initial interactions more engaging.

Summary of VbyVi Data for Different Variants

The table presents VbyVi values for three different variants: Custom, Other, and Stock. The metrics are compared between Control and Test conditions.

Column Descriptions:

variant:

Represents different versions or segments under analysis:

Custom: Represents a specific customization.

Other: Represents other non-specified segments.

Stock: Represents the standard, non-customized version.

Control:

Metrics for the control version of each variant.

Test:

Metrics for the test version of each variant.

Key Highlights:

Custom Variant:

Control: 466.93.

Test: 439.55.

Observation: There is a decline of approximately 27.38 in the Test condition compared to Control, indicating reduced engagement or performance in the Custom variant.

Other Variant:

Control: 490.38.

Test: 464.90.

Observation: The Other variant also shows a decline of about 25.48 from Control to Test, suggesting a drop in performance.

Stock Variant:

Control: 472.05.

Test: 436.88.

Want to print your doc?
This is not the way.

Try clicking the ··· in the right corner or using a keyboard shortcut (

CtrlP

) instead.