Learn home · TRACE theory · Explaining a shift · Research landscape · References · Research board v4

Research board v4: why TRACE falls short where it does, and what to change

This page explains, in plain language, the v4 research board. The full technical version, with the maths, is BOARD_V4.md; every paper is listed with a checked link in REFS_V4.md. Labels: proven = a theorem or identity; measured = a number from this repo's frozen results; guess = our expectation, which the stated test can prove wrong.

1. The short version

The frozen results list eight weaknesses. They trace back to five causes:

Cause	In one sentence	Weaknesses it explains
Wrong projection	The fitting loss pays most attention to where the data are dense, so it quietly shrinks the tail-shape numbers we then report.	mean accuracy, band coverage, part of stability
Average of curves ≠ curve of the mixture	TRACE averages each person's quantile curve; the real overall distribution is a mixture, and mixing people with different centres changes the shape on its own.	shape shifts missed, the 3% model-vs-data gap
Error bars on a squared quantity, by refitting	Squaring a noisy difference always adds a positive bias, and refitting the model to get error bars repeats the original bias and costs minutes.	band coverage, null test size, speed of bands
Reweighting matches averages, not histograms	The step that asks "what if period B had period A's mix of people?" uses a model that only matches averages, while TRACE needs each feature's histogram matched.	weak reweighting
The per-feature split is not pinned down	When two features carry the same signal, the booster splits the attribution between them by accident, and a resample flips it.	per-feature stability

2. The key idea, with a picture in words

Take two groups whose outcomes are both bell-shaped with the same width, one centred at −1 and one at +1. TRACE's "barycenter" averages their quantile curves and gets a bell centred at 0 with the same width. But if you pool the people, the pooled distribution is wider (and, with lopsided noise, more skewed). Now suppose period B simply has more people far from the centre (a composition change). The pooled distribution changes shape; the averaged curve does not. TRACE had no place to book that change, so in the tests where composition alone changed the shape, it found nothing (recovery 0 of 6 cells, measured), while the rival that averages distributions instead of curves (CFM) did well.

The fix (M1) keeps TRACE's game and adds one more kind of player: a mixing player. For small spreads there is a simple formula (proven, second-order expansion): the extra shape equals half the variance of the predicted centres times a fixed curve that depends only on the noise shape. The variance splits exactly across features as covariances, so each feature still gets its own piece.

3. Top five changes

#	Change	What it fixes	How we'd know it worked (pre-registered bar)	Cost at 1M rows × 100 features
1	Mixing players (M1)	shape shifts from composition; model-vs-data gap	recovery ≥ 0.6 (n=2k) and ≥ 0.8 (n=20k) on the three composition cells; error not worse than CFM	~3 s
2	Fit the target we report (M2): plain least squares for the mean; for shape, regress on a "PIT" stand-in whose average is the shape number; smooth across channels; recalibrate on the probability scale	mean accuracy; shrunken shape numbers	mean error ratio vs EBM: upper confidence limit ≤ 1.00; fitted/true shape numbers within 5% for the first three; CRPS ≤ EBM+conformal in 2 of 3 cases	+25–40 s
3	Error bars without refitting (M3): estimate the squared quantity from two independent halves (removes the bias exactly), linearise, resample cheap weights instead of refitting; exact label-shuffle test	band coverage; false alarms; slow bands	coverage ≥ 0.88 pooled, ≥ 0.85 everywhere; false-alarm upper limit ≤ 0.09 at 200 runs; bands ≤ 2× one fit	~10–30 s (vs ~30 min now)
4	Match histograms, not averages (M4): reweight so each feature's bin counts match the other period (raking)	weak reweighting	per-feature composition error halved	~5 s
5	Stability layer (M5): smoothing penalty on bins, 8 bags averaged, a canonical centring, grouping of near-duplicate features, selection frequencies	per-feature stability	rank stability ≥ 0.77 (bootstrap) and ≥ 0.46 (split-half), the level of the best rival	+20–40 s wall

Why number 4 is so clean (proven): TRACE's per-feature pieces only depend on each feature's histogram over its bins. If the weights reproduce every histogram exactly, every composition piece is exact, for any fitted model. No propensity model is needed.

4. Ideas borrowed from other fields, and what they bought

Physics. Jordan–Kinderlehrer–Otto gradient flows say the natural "small step" for distributions is a penalised step; for one-dimensional outcomes that is just a ridge penalty on quantile curves, which is what M2 adds. Entropy-regularised transport (statistical mechanics) taught us the debiasing trick behind M3; we don't need its solver, because exact transport in one dimension is already fast. Multiscale ("renormalisation") smoothing helps thin bins in M5.
Fluids. Following a particle (Lagrangian) versus watching a fixed spot (Eulerian) is exactly the curve-average versus mixture gap of cause 2. Benamou–Brenier's moving-mass view gives a second proof that TRACE's split is the natural one. Helmholtz–Hodge splitting has nothing to add while the outcome is one number.
Computer vision. Linear-transport and cumulative-distribution-transform embeddings turn out to be quantile curves in one dimension, which confirms TRACE's basis. Sliced distances matter only for a future multi-output version.
Biology. Single-cell trajectory tools (Waddington-OT, moscot) handle growth and death with unbalanced transport. Their lesson for entry/exit: when you know who entered and who left, use the exact counts; the unbalanced penalty is only for when you don't.
Econometrics. Distribution regression computes the mixture counterfactual (used in M1). Riesz-representer "automatic debiasing" and balancing weights give exact, model-light reweighting (M3, M4).
Statistics. L-moments (the basis), CRPS and isotonic distributional regression (M2), studentized permutation tests (M3), stability selection and knockoffs (M5), conformal prediction (M2), kernel two-sample tests (a backup null).

5. Where each weakness comes from (one line each)

(a) Mean a bit worse than EBM, CRPS behind EBM+conformal: the loss is not squared error, and the raw bands are too wide.
(b) Per-feature rankings unstable: shared signal between correlated features is split by accident; tail pieces are noisy; one bag.
(c) Shape shifts missed: composition changes the mixture's shape, which the curve average cannot see.
(d) Bands cover 81%, tails worst: bands are centred on shrunken values (and, in composition cells, on the wrong target); squared quantities are biased up.
(e) 6.8% false alarms: deciding "shift" by "band excludes zero" is slightly loose; every rival using the same rule shows the same thing.
(f) Up to 3% model-vs-data gap: the mixing term, plus interactions the additive model leaves out.
(g) Weak reweighting: a linear-index odds model matches averages, not each feature's histogram.
(h) Bands are slow: each band needs ~100 refits.

6. Proposed v4 pipeline

Fit (8 bags): least-squares mean; PIT-based shape channels with smoothing; canonical centring; groups for near-duplicate features; recalibration.
Counterfactual: histogram-matching weights (exact for the curve average) and Riesz weights (for the mixture).
Game: the same symmetric switch game, run on the overall distributions, with mixing players and exact entry/exit.
Inference: half-sample products, linearised error bars, cheap resampling, exact shuffle test, selection frequencies.

Estimated total at 1M × 100 including bands: about 1.5–2.5 minutes (guess, from component timings), versus about 30 minutes today. Suggested order: M2, M4, M1, M3, M5, then entry/exit.