Robustness monitor · Backman, SAFE

Advice Drift Tracker

Every wave re-asks one question — what share of $50,000, stated in the local currency equivalent, a 40-year-old investing over five years should put in a broad stock index fund rather than government bonds — of a panel of models, in English: one English prompt per country, repeated 25 times per model (50 in the July wave). Language is held fixed, so the only things that move between waves are the models and the date. This is the paper's English arm, not its primary local-language cell. The paper's own July 2026 collection is the first wave, cut to the same slice.

By model, across waves

Where each model lands, and how far it moves across countries

One row per model. For each wave the model was in, the dot is its average recommended equity share over all its kept responses in that wave, and the bar behind it runs from its lowest to its highest country average — a short bar means the model gives nearly the same answer everywhere, which is the compression the paper documents. A model in two waves gets two dots joined by a line. For the seven anchor models, queried under the same slug in every wave, that line is change on the provider's side, a rotated snapshot or a serving change, not a change of model. For every other pair the difference mixes model vintage with the later query date. Models tagged partial were still collecting when the page was built.

Table view

By vintage

Does the answer change with model generation?

Every model placed by the day its slug went live on OpenRouter. The vertical axis is either the model's mean recommended equity share or its country spread, the distance from its lowest to its highest country average. One dot per model per wave, so an anchor appears twice at the same date, joined by a line. The shaded band is the paper's collection, 2–4 July 2026. No line is fitted: with a handful of models per vendor this is a picture, not an estimate.

Each model

Country profiles, one chart per model

Each chart plots the model's mean recommended equity share in every country, one dot per wave, in a fixed order: the USA first, then the other five English-prompt countries, then Europe, Asia and the Americas. The dashed line is the model's USA value in its latest wave, so dots below it are countries where that wave's advice sits below the model's own US level. All charts share one vertical scale, set by the models currently shown. Hover a column for the numbers.

By country and model

Every cell in one wave

Mean recommended equity share, in percentage points, for the wave chosen in the controls above. The USA row is outlined because it is the paper's baseline. Hover any cell for its standard deviation across repetitions and its observation count.

Table view

Reading this

What the tracker does and does not show

These are raw cell means. The tracker does not reproduce the paper's estimates: there are no country fixed effects, no wild cluster bootstrap over models, and no Holm correction for multiple comparisons. The country gaps shown are plain differences against each model's own USA cell, which is a descriptive summary and not an estimated treatment effect. The provider average in the collapsed view is an unweighted mean of that provider's models.

The July wave is the paper's main collection restricted to the tracker's slice, so its thirteen models are directly comparable with every later panel, with one difference in precision: July cells hold 50 repetitions, tracking cells 25. A difference between a July model and a September model still mixes two things — the model is newer and the query is later — and only the anchors, which are the same slug queried in both waves, separate them. Release dates are the day the slug went live on OpenRouter. In July the API returned dated snapshot names (shown as “served as” in the table view); in September it returned the bare slug, so the exact snapshot behind a September answer is not observable from the response.

Excluded from every figure, following the pre-registered rules: truncated responses, parse failures, and refusals. Also excluded, by the tracker's own rules rather than the pre-registration: zero-token provider errors, and rows from models dropped from a panel after collection had started, which the strip counts outside the total. “Slug drift” in the strip counts rows whose returned slug differs from the one every other row of that model got in the same wave. Temperature is fixed at 0.7 with no system prompt, and max_tokens is 4000, matching the main collection.