‍ Short-Term Signals, Long-Term Truth

This is a small town - Pahadi. All around it, the ethereal Himalayas rise up, cradling this lovely place and right at the top sits a majestic monastery, perched like the crown of a king. All visitors - new and locals visit the monastery. But Pahadi is remote, and new visitors here are still just a handful. Your task is to watch those who set out on the trek, and to evaluate the success metrics of everyone who begins the climb.

You might have observed that every metric you chose as your winner was indeed solving something useful. But yet it seemed more short-term than anything else.

Because this trail is remotely located, getting new people to visit just once isn't enough to keep the small businesses alive. So, while summit successes are crucial, trekker retention among locals is the more important goal.

But how do you measure a delayed metric such as retention in a short evaluation? Cannot observe forever as the execution costs are high. And without evaluation of short term metrics, taking a bet on long term metrics is also risky.

The solution is Surrogate Metrics: statistical stand-ins that bridge short-term user behavior to long-term impact.

Often, in online product experimentation, isolated short-term metrics rarely capture the effect of the long term metric, which is what the business focuses on.

What it is in short:

For instance, imagine you are at YouTube testing a new feature designed to increase consumer retention (your long-term metric Y). For this case, let’s assume its a two week experiment that you execute.

In this approach, you collect short-term signals over the course of 2 weeks- watch hours, active days, subscriptions.

Now look at your archive of old experiments far enough to have the actual retention data. You have both: the two-week signals back then, and what actually happened later. Learn the pattern between them once, and you've built a translator early signals in, long-term retention out.

Run your new test for two weeks and it tells you the likely retention impact. You get a decision now instead of a 3/6/12 months from now.

For people not interested in Math, this is where we take different paths!! The rest of the article is for data scientists.

Let’s continue on the Youtube example

Short term metrics: Day-1 watch time, channel subscriptions, search queries executed, or "Niche Depth" (watching >= 3 videos in one topic)

Assumption- The treatment affects the long-term outcome ONLY through the observed short-term surrogate metrics.

A good thumb-rule: Always include multiple short-term behaviors to capture both engagement and friction signals.

The Surrogate Index Calculation

  • Take completed historical experiments which have actual data available on Y. Given Y here is 12m retention, all experiments conducted at least 12 months before are eligible, since we can evaluate the true Y on customers. 

  • Measure actual observed short-term metrics (S) at the end of experiments and the actual observed Y that is available post 12 months from the experiment.

Regress Y on both X and surrogate vector S.
X: Pre-experiment user covariates (e.g., country, account age, historical activity).

The Causal Forest:

  • Impute each user’s estimated Y from the surrogate index of the previous step. This is done at the end of experiment, predicting Y using available data on short term metrics

  • Train a Generalized Random Forest with that imputed Y, treatment W and covariates X

Post-Experiment Decision & Conclusion Framework

At Day 14 of your live experiment, aggregate individual Causal Forest predictions across all treatment and control users to compute the overall Average Treatment Effect (ATE):

τ̂_ATE = (1 / N) * Σ  τ̂ᵢ(Sᵢ, Xᵢ)

Rollout Decision Rules:

SHIP FEATURE

• τ̂_ATE is statistically significant (p < 0.05)
• 95% Confidence Interval is strictly > 0
• Predicted lift exceeds Minimum Detectable Effect (MDE) to offset engineering costs
• Guardrail metrics remain healthy

ABANDON FEATURE

• τ̂_ATE < 0 with high statistical confidence
• Feature damages predicted long-term retention, even if short-term metrics appear positive

INCONCLUSIVE

• 95% Confidence Interval overlaps zero
• Action: Extend test window for higher power or iterate on feature design before re-testing

Key Resources & Recommended Reading

  • A Methodology for Experimentation Included in Short-Term Measurements (Athey, Chetty, Imbens, & Kang, 2019) 

  • Focusing on the Long Term: It’s an Innovation Problem (Google Research, 2015) 

  • How Uber Predicts Long-Term User Value in Experiments (Uber Engineering Blog)