Measurement for Shared Compute

Whenever we deal with causal questions, it is worth thinking about the ideal experiment. Always ask yourself, if you could, what would be the perfect experiment you would run to uncover this causal effect? This tends to shed some light on how we can discover the causal effect even without the ideal measurement environment.

1. Name the decision. Is the decision to ship broadly, target a segment, change a routing policy, reserve more capacity, or instrument before acting?
2. Choose the comparison. The counterfactual might be the old policy, the untreated users, a different cluster, a prior model checkpoint, or a threshold-adjacent group.
3. Pick the estimand. Average Treatment Effect (ATE) is for broad rollout, Average Treatment Effect of the Treated (ATT) is for the users who actually received treatment, Conditional Average Treatment Effect (CATE) is for targeting, and policy value is for routing or allocation decisions.
4. Unit economics as a guardrail. A policy can improve task success while burning too much compute. Pair retention or task completion with latency, errors, GPU-hours, and cost per successful request.
5. End with a rule. Ship, ramp, target, hold out, pause, roll back, or instrument more. The point is to state the recommendation and the tradeoff that could change it.

Estimand Cheat Sheet

In shared compute, the "treatment" is often a policy rather than a single static action. A router, rate limit, cache policy, or capacity allocation rule is dynamic, and chooses an action based on the request-level workload and the live system state.

This changes what has to be measured. A credible evaluation requires the inputs the policy saw, the intended action, the executed action, the fallback path, and the reason for any divergence. Otherwise the analysis can confuse the policy effect with the state that caused the policy to act.

In the router example, premium-fleet assignment is not simply "treated." The same request may be routed to premium capacity when the queue is shallow and sent to a cheaper fleet when the system is congested. The estimand must therefore attach to the allocation rule, not merely to the serving path that happened to execute.

Minimum Logging for a Policy Readout

The control question should come from the assignment story, not from feature importance. Adjustment variables should exist before the policy decision and plausibly cause both assignment and the outcome. The analysis should not control away the mechanism by which the treatment worked.

System State Is Both a Confounder and an Outcome

Shared inference systems make the timing of state measurement central. At time t, system state can be a pre-treatment confounder for the current routing, admission, or fallback decision. At time t + 1, that same class of state variables can be downstream of earlier policy decisions.

Queue depth before a routing decision may be an adjustment variable. Queue depth after the policy has run may be part of the policy's effect. In the router example, congestion can determine whether a high-value request reaches the premium fleet, while the routing decision itself can also change later congestion for untreated requests.

1. Control for common causes. If queue depth, customer tier, workload length, or provider health affected both routing and success, it belongs in the adjustment story.
2. Do not control for mediators in a total-effect question. Latency, errors, cache hit, and completion may be how the routing change affects retention or task success.
3. Do not select only completed requests. Completion can be caused by the treatment. Dropping failures makes a bad serving policy look cleaner than it is.
4. Name the spillover path. If the treatment changes a shared queue, cache, fleet, or regional capacity pool, user-level treatment and control groups may no longer be independent.

Probably Bad Controls

For AI infrastructure and inference products, a plain difference-in-means readout can be fragile when the metric is tokens, cost, or GPU-hours per user. Those outcomes can be dominated by a small number of very large users, long-context workflows, or high-concurrency windows.

Tokens per request are bounded by the context window, but the variance can still be large as use cases get more heterogeneous. True resource cost is messier because it depends on live system state: cache pressure, batch concurrency, prefill/decode balance, provider health, and the current capacity boundary. The sample mean can move because the policy worked, or because one arm happened to receive more unusually large workloads.

Two tail risks should be kept separate. Cross-sectional tail risk means a small number of users, accounts, or workflows dominate cost. Temporal and system-state tail risk means peak-concurrency periods push the system toward capacity boundaries and alter batching, cache pressure, fallback behavior, and untreated-user experience.

Robust Cost Readouts [needs review]

For a heavy-tailed cost metric, the raw mean should remain visible because it is the business cost, but it should not be the only readout. Pair it with a winsorized or trimmed estimate, a quantile readout if tail risk matters, and CUPED or regression adjustment using pre-period usage. If the confidence interval depends heavily on normal approximations, use randomization inference tied to the actual assignment mechanism.

When the policy changes a shared system, uncertainty may need to be computed at the time-window, cluster, fleet, or account level. In the router example, a policy can look efficient on ordinary traffic while creating unacceptable fallback rates during peak-concurrency blocks. That is a capacity-risk question, not only a central-tendency question.

Observational work should not oversell the estimator. The real question is whether the historical comparison has the right pre-treatment variables and enough overlap to support the recommendation.

These are strongest when the business already created quasi-random variation: a phased rollout, a threshold, a beta invite, or one region/fleet/provider changing before the rest. Pick the design based on the assignment story, then state the assumption that could fail.

Heterogeneity should change the allocation rule, not merely decorate the analysis. If the result does not change who gets capacity, which workload is routed differently, or which customer segment is worth serving differently, it is mostly decoration.

1. Start with the average effect. That shows whether there is a real lever worth segmenting.
2. Pre-specify operational segments. Good candidates are workload length, customer tier, model family, region, latency sensitivity, and peak versus off-peak demand.
3. Use CATE only if targeting is the decision. The segment has to be stable, interpretable, and simple enough to operate.
4. Convert lift into net value. A high treatment effect can still be a bad allocation if it uses expensive capacity for low-margin traffic.
5. Keep some exploration alive. If the router only exploits the current best segment, future evaluation loses support for alternatives.

Offline policy evaluation is useful for routers, schedulers, ranking policies, and allocation rules, but only if the logs contain enough examples of the actions the new policy wants to take. Check support before estimating anything.

Logged propensities can support short-horizon contextual routing evaluation, especially when the action affects the current request and the relevant state was logged. They are less sufficient for full dynamic-policy evaluation when actions alter future queue and cache state. In that setting, IPS or doubly robust estimates can help screen policies, but live validation is still needed before broad deployment.

1. Check support. Did the old policy try the same actions for similar state often enough?
2. Check propensities. If action probabilities were logged, IPS and doubly robust methods become more credible.
3. Inspect weights. If a few records drive the answer, the offline estimate is too fragile for a broad rollout.
4. Validate live. Even a good OPE result should usually become a limited ramp or controlled exploration bucket before full deployment.

Short-run metrics are useful, but they should not be treated as business outcomes by default. First state what part of the mechanism they measure, then explain why that mechanism should translate into retention, expansion, or LTV.

Surrogate Ladder

1. System metric: TTFT, latency, cache hit rate, error rate, retry rate, or cost per successful request.
2. Session outcome: task completion, abandonment, re-run rate, or successful workflow completion.
3. User behavior: return rate, deeper usage, broader adoption, or more high-value workflows.
4. Business outcome: retention, expansion, renewal, LTV, or margin.

The ladder can break. A faster request is not automatically higher retention, and more usage can still hurt margin if the workload is expensive or low value. A metric that predicts retention is not necessarily a valid surrogate for retention. A valid surrogate claim requires evidence that movement in the short-run metric captures the causal path by which the policy changes the long-run outcome.

If the long-run outcome takes too long to observe, the recommendation should be conditional: ramp only if the mechanism metric improves, the cost and reliability guardrails hold, and early behavior moves in the direction past experiments suggest. For policies expected to affect retention or expansion, keep the smallest durable holdout that can answer the long-run question. If no holdout is possible, label the surrogate risk explicitly rather than pretending the short-run metric proves the whole case.

The check is not there to show diligence for its own sake. It should change the recommendation: ship, ramp, pause, roll back, narrow the target population, or instrument more.

Common Decision Patterns

Methodological references: potential outcomes and bias, randomized experiments, graphical causal models, Causal Inference for the Brave and True chapters on propensity scores, doubly robust estimation, DiD, synthetic control, RDD, IV, and heterogeneous treatment effects, plus Why fat tailed costs emerge at scale.