Skip to content
Platform filter usage has reached prior baseline at ~5% with scale up of legacy HVF models on 02/08. Model had specifically failed for CLP and collections starting Jul’24
Relaunch update - Post launch deep-dives for parallel feeds indicate sharp drop in reco V/Vi leading O/Vi and T/Vi drop on platform. Team had found these issues to be pre-dominantly affecting the generic reco tabs where scrolling problems led to ~70% of the net drop.The fix for these was rolled out for 100% users on 27/08. The performance post app update has significantly improved, but we still see intermittent scrolling issues when testing across devicesRelaunch - We’d be relaunching the build starting 29/08 for external users. This is basis TAT for ABacus to showcase the updated app version in selecting user cohortsThe launch of the parallel feed feature resulted in a ~3% increase in Orders per View (O/V) but a ~3.3% decrease in Views per Visitor (V/Vi), leading to -0.3% drop in Orders per Visitor (O/Vi) on the platform. The V/Vi drop was primarily concentrated on PDP reco with test underperforming control in terms of views by ~13%.
Scrolling problems on the parallel feed, causing users who previously contributed >40 catalog views to shift to fewer views, impacting V/Vi by ~45%.Users who contributed >40+ catalog views on an average on PDP reco in test cohort shifted to buckets contributing <20 views with a significant switch to <=5 views after the launch of experiment (26th July)
Fast upward scrolling, which skipped the recommendation section, also reduced duplicate view counts, contributing ~20% to the V/Vi drop
Backend deployment and Android release for the feature went out on 27/08The build is working fine except the following caveats - CTA is visible on reco with parallel feeds, This was missed in execution and will be fixed iterativelyFTUX will only be visible on wishlist screen for the MVP build. Extending that across REs would require significant effort which Android team would be picking up nextExperiment details - Cohorts - Test 1 - All REs on variant 1Test 2 - All REs on variant 2Control - No REExperiment would be launched on Multi RE / Future use 8 plane where we would free out ~650 audience buckets contributing to ~80 Mn users. We want to keep the cohorts minimised for higher audience exposure and reach significance in the next 2 weeks
Restoring hero PID model would lead to hero products getting updated for ~15% OC on the platformModel restoration is complete, with following plan for experimentation - Updating hero products for 3 SPs - Men fashion / Home decor and furnishing and Women kurtis and Kurta setsThe experiment would lead to ~3% platform OC getting updated hero products, ~4L catalogsEstablishing experiment success - Comparing funnel metrics like CTR or C/V, O/C, O/V within SP for catalogs where hero pid changed vs not changedComparing SP performance for above funnel metrics w.r.t. a similar SPExperiment to start 29/08
Model has been built for CLP, Model for search and collections to be closed by 29/08Other pre-requisites - Image QC for all filter values was started 27/08, Closure TAT - 02/04Needs tech support to check - If tech handles / caps visibility to X IF rowsDo the remaining IFs show up if user applies a filterDo IFs go in the selected state at top of page similar to HVFs and users have the option to remove those? What happens post, are the applied IF row again visible?Instrumentation for IFs - Do we instrument impressions on IFs like HVF?Aligning on table structure for model data export to backendExperiment launch - Experiment is slated to launch post code freeze. Team would setup a small experiment starting 3rd Sep to understand initial performance of the model / any updates required in the core feature to iterate and keep the build experiment ready post code freeze
HVF v2 was launched on Aug 7th across REsReads signify drop in O/Vi across all cohorts driven by sharp drop in O/V for the new model, with the old restored variant emerging as the winnerRecommendation - We’d be scaling the old models for remaining users across the platform and also enable dynamic hero product capabilityDeep dives on the reasons for O/V drop and L2 analysis to be picked during code freeze. Platform trends -
Post deployment, devs found an issue with new DS label values not being properly tagged to catalogs due to increase in config size and heavy compute. Dev is on-going here and In parallel, we’ve initiated QC of the new labels from Biz. Update - While dev would continue, deployment would be planned post MBS sale due to ramifications of the flow increasing latency of core services
(P0) Tech - Ranking for filtered feeds(P0) Prod / Design - Revamp of static filter screens(P0) DS - Category order for adjacent category ingress on parallel feed v2(P1) Analytics - Using search NER, identify most relevant and highly converting PSTs for a given feed. To be ingested using product card SDUI
28/08/24
Metrics
Key discussion points (Pre code freeze) -
Parallel feed relaunch
Parallel feed relaunch
Deep dives indicate that users who visited only the first recommendation tab had significantly lower V/Vi compared to those navigating parallel feeds, which had ~41% higher V/Vi than the control group
Cohort
VbyVi
ObyVi
TbyVi
User Distribution
scroll_depth
PDP view share
Test - Users only browsing Similar
265.18
0.69
0.08
85.10%
32.82
72.3%
Test - Users browsing parallel Feed
580.41
1.23
0.12
14.90%
39.87
27.7%
Control - PDP recos
411.26
2.38
0.09
100%
31.17
There are no rows in this table
About 85% of users primarily consumed recommendations from the first tab, contributing to the V/Vi drop. Issues identified included -
Portfolio level cuts suggested that V/Vi drops were consistent across categories, indicating relevance or selection was not the main driver. A list of SSCATs to potentially skip in future relaunches is being considered, based on further analysis of user behaviour
Instrumentation alignment - We would be recording parallel_feed as part of primary real estate in base views and clicks events. Decision to be aligned cross org to align ranking stakeholders on funnel attribution changes. TAT - 28/08
More like this
Hero product model restoration
Key discussion points (During / Post code freeze) -
Interstitial filters
HVF
Cohort
O/C
C/V
O/V
V/Vi
O/Vi
Gmv/Vi
T/Vi
Net_orders/Vi
Nmv/Vi
Control_2 (w/o HVF)
-0.09%
-0.03%
-0.12%
0.03%
-0.09%
-0.13%
-0.03%
0.03%
0.05%
Test_all_4_dh
-0.01%
-0.17%
-0.17%
-0.02%
-0.19%
-0.21%
-0.03%
-0.07%
-0.10%
Test_all_4
-0.03%
-0.12%
-0.15%
-0.04%
-0.20%
-0.30%
-0.06%
-0.14%
-0.32%
Test_all_5_dh
-0.08%
-0.09%
-0.17%
-0.01%
-0.18%
-0.30%
-0.10%
-0.20%
-0.24%
Test_clp_4_dh
0.02%
-0.07%
-0.05%
-0.01%
-0.06%
-0.21%
-0.04%
0.08%
-0.11%
Test_search_4_dh
-0.16%
-0.01%
-0.17%
-0.15%
-0.31%
-0.30%
-0.08%
-0.28%
-0.36%
Test_coll_4_dh
0.03%
-0.05%
-0.02%
-0.17%
-0.19%
-0.33%
-0.10%
-0.09%
-0.18%
There are no rows in this table
CLP metrics -
Cohort
Type
V/Vi
C/Vi
O/Vi
Gmv/Vi
O/C
T/Vi
Test_all_4_dh
meritocratic
-0.11%
-0.44%
0.25%
-0.73%
0.59%
-0.21%
Test_all_4
meritocratic
-0.28%
-0.48%
0.23%
-1.37%
0.43%
0.00%
Test_all_5_dh
label wise
-0.12%
-0.54%
0.16%
-0.87%
0.58%
-0.49%
Test_clp_4_dh
meritocratic
0.04%
-0.33%
-0.55%
-1.69%
-0.18%
-0.58%
There are no rows in this table
Search metrics -
Variant
Type
V/Vi
C/Vi
O/Vi
Gmv/Vi
O/C
T/Vi
Test_all_4_dh
meritocratic
0.01%
-0.08%
-0.41%
-0.49%
-0.32%
-0.28%
Test_all_4
meritocratic
0.05%
-0.17%
-0.66%
-0.86%
-0.44%
-0.43%
Test_all_5_dh
label wise
0.03%
0.14%
-0.52%
-0.64%
-0.62%
-0.40%
Test_search_4_dh
meritocratic
-0.07%
-0.14%
-0.61%
-0.72%
-0.54%
-0.50%
There are no rows in this table
Collections metrics -
Variant
Type
V/Vi
C/Vi
O/Vi
Gmv/Vi
O/C
T/Vi
Test_all_4_dh
meritocratic
-0.04%
-0.11%
-0.13%
0.30%
-0.05%
0.42%
Test_all_4
meritocratic
-0.13%
-0.04%
-0.70%
-0.81%
-0.79%
-0.28%
Test_all_5_dh
label wise
0.06%
-0.13%
-0.09%
-0.04%
0.10%
0.02%
Test_collections_4_dh
meritocratic
-0.23%
-0.37%
-0.35%
-0.37%
-0.21%
-0.36%
There are no rows in this table
DS filter label rationalisation
Roadmap / Backlog (Solutioning to be picked during / after code freeze)
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.