How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model

The 30-second take

• What: The authors pretrain a 34.6-million-parameter decoder on 1.4 billion coded events from 3.6 million North West London patients, then test colorectal-cancer prediction on the same 1,008 cases and 9,582 matched controls at 15 nested lookbacks from 2 to 30 years.
• Why it matters: EHR foundation models have been sold on token context. If time-context is a separate design axis, clinics may be keeping decades of records for a discrimination gain that mostly saturates in a decade, and rotary embeddings may alias beyond about 20 years.
• Who should care: Clinical-prediction teams, EHR-foundation-model builders, health-system data officers deciding how far back to store identifiable events, and anyone scoring medical AI as if longer context were automatically better.

What the paper actually did

The authors argue that long context in electronic-health-record foundation models has been measured in tokens, not in calendar time. They introduce time-context as the lookback window before the prediction point, and they say this is the first systematic evaluation of decade-scale time-context for an EHR foundation model.

They pretrained a decoder-only transformer with 34.6 million parameters on 1.4 billion coded events from 3.6 million primary-care patients in North West London, then fine-tuned it to predict incident colorectal cancer with a 12-month pre-diagnostic buffer. A frozen cohort of 1,008 cases and 9,582 matched controls, all registered for at least 30 years, was scored at 15 nested lookback windows from 2 to 30 years so every truncation is compared within the same patients.

Discrimination rose from an AUROC of 0.551 at 2 years to 0.590 at 18 years, then plateaued. About 80 percent of the gain was reached within 10 years. An XGBoost baseline converged to the same level, with a peak AUROC of 0.589. A simulation suggests rotary position embeddings over elapsed time alias beyond about 20 years at the default base frequency, which they say motivates a retuned theta. Time-context, they conclude, is a measurable design axis distinct from token-context and should be reported.

What makes this disruptive

If the core claim holds, more years of chart history is not a free lunch for cancer-risk models. The scarce capability under pressure is a clinical predictor that uses the right years, not every year a warehouse can store.

The distinctive move is holding the patient set fixed and truncating lookback from 2 to 30 years, then showing that a 34.6-million-parameter transformer and a gradient-boosted baseline meet at almost the same plateau. An 0.039 AUROC lift from 2 years to 18 years, with 80 percent of it in the first decade, is a small, specific number rather than a slogan about long context. The RoPE-aliasing warning is a second, independent design claim: the clock inside the model may not even be honest past about 20 years.

This week's editorial ranking placed the preprint first of twenty, with a score of 93. That is triage for the Disruptive Concepts brief, not a peer-review verdict. Treat the work as a measurement of how far back the signal lives, not as a screening tool ready for clinic.

Why it matters (outside the lab)

Abundance lens (today's luxuries toward tomorrow's defaults): expert judgment and long, identifiable medical histories are still scarce and expensive. If time-context is reported the way token-context is reported, health systems can stop treating infinite lookback as a default and start treating a decade of coded events as the part that actually moves discrimination.

Near-term, the preprint is a baseline for teams already training EHR foundation models and for privacy officers who currently keep 30-year charts because a model might want them. Medium-term, whether the 0.590 plateau is clinically useful, whether other cancers and other countries look the same, and whether a retuned rotary embedding actually fixes the 20-year alias decide if this is a method or a London-only curve.

No calendar date is implied. Disruptive Concepts, which publishes these explainers free for humans, reads this as a possible brick toward making long-horizon clinical prediction less of a data-hoarding luxury, not as a promise of a consumer cancer app.

Limitations & open questions

This is a preprint, not a screening product. The abstract reports one cancer, one health system, one frozen cohort of 1,008 cases, and AUROC numbers that remain modest even at 18 years. We have not re-trained the 34.6-million-parameter model or re-cut the nested lookbacks.

The abstract does not report calibration, positive predictive value, sensitivity at a fixed alert rate, fairness across practices, or how the 12-month buffer was chosen. Matching 9,582 controls is not a prospective deployment. An XGBoost peak of 0.589 is evidence that the transformer is not uniquely magical here; it is also evidence that the task may be intrinsically weak.

Abundance is not automatic. A measured time-context curve does not demonetize oncology or make 30-year identifiable records a default on a fixed date. Consent, coding quality, and whether 0.590 AUROC is worth acting on still sit between this study and a public default.