Canary Every CLI Upgrade: Catch Default and Compaction Drift on Unpinned Lanes

On Sep 11 the model behind kimi-for-coding changed and the ID did not. Kimi’s changelog was cheerful about it: “The Model ID is unchanged”, so clients and third-party tools needed no configuration changes, and K2.8 Preview’s default thinking effort is max. Every lane that inherited those defaults was running a different model that morning, at max unless it said otherwise, with a config file that diffed clean against the night before.

That is the hole in most AI CLI upgrade testing. We check that the new version installs, that the flags parse, maybe that the deny rules hold. We rarely check what the lane was served afterwards: which model, at what effort, under what turn cap and wall-clock limit, at what token count it compacted, and what it forgot when it did. The move here is an upgrade canary per lane: record those served values from logs, bump one lane, replay the same long session, diff, and fail the upgrade on any change the lane’s manifest did not declare.

A chatbot whose defaults drift gives you a different answer, and you notice. An unattended lane whose defaults drift runs longer, spends more, compacts at a new point and carries a different summary into its next hour of tool calls. The first person to notice is usually whoever reads the invoice.

Twelve harness changes in eighteen days, Sep 4–21

September moved lane behaviour three ways, and only one of them arrived with a date on anyone’s calendar.

Defaults. Codex CLI 0.153.4 (Sep 4) made GPT-6 Astra “the bundled default when no model is explicitly configured” (Codex 0.153.4 release notes). Kimi’s K2.8 Preview rolled out Sep 11 behind the unchanged kimi-for-coding ID, and Kimi CLI 0.43.0 (Sep 14) removed goal mode’s 24-hour cap (Kimi Code what’s new). Claude Code 2.1.277 (Sep 18) started reading AGENTS.md in projects with no CLAUDE.md. GitHub’s Copilot Auto tiers (Sep 14) say “Auto individually evaluates each prompt to select a best-suited model”, and usage is “charged based on the model auto selects, regardless of tier” (GitHub changelog).

Screenshot: Kimi Code Docs, “What’s New” (Sep 11, 2026 entry), captured Sep 21, 2026.

Compaction. Claude Code 2.1.273 (Sep 15) fixed auto-compact counting advisor-tool turns at roughly twice their real size, which had made it fire at about half the real window (Claude Code changelog). Codex 0.155.0 (Sep 17) moved to always-streamed remote compaction for supported providers and now preserves reasoning effort through compaction (Codex 0.155.0 release notes). Kimi CLI 0.43.0 added loop_control.compaction_max_attempts with a default of 5, and 2.0.2 (Sep 19) fixed compaction failing after a switch to a smaller-window model (Kimi Code CLI changelog); Kimi Code 2.0.0 (Sep 17) omits the oldest images and videos once a session’s media passes 20 MB. Pi 0.86.0 (Sep 19) added per-model compaction.modelOverrides, and Pi 0.87.0 shipped today with breaking changes for extension authors (Pi changelog).

Screenshot: GitHub, “Release 0.155.0 · openai/codex” (Sep 17, 2026), captured Sep 21, 2026.

The line that changes how a Codex lane compacts sits between a reviewer-extension refactor and a removed Windows slash command. Nobody reads every row, which is the argument for testing behaviour instead of reading prose.

Calendar. Antigravity’s May agent version shuts down Oct 5, GPT-5.5 leaves ChatGPT, ChatGPT Work and Codex (not the OpenAI API) on Oct 14, Copilot deprecates a batch of models on Oct 19, and Gemini 3.8 Flash’s price doubles on Jan 1, 2027. Dated swaps belong in the deprecation ledger that forced model routing continuity already runs. This piece covers the undated kinds, which land the moment a lane picks up a new binary.

AI CLI upgrade testing has to start from what was served

A Codex lane that never set a model got Astra. A lane pointed at kimi-for-coding got K2.8 at max. A Claude Code lane that used the advisor tool had been compacting at half its window and now runs to the full one: longer context, later summary, bigger bill per turn. In every case the config file is byte-identical before and after, so the config diff most change reviews look at reports nothing.

The config is the declaration; the wire and the session logs are the evidence. The canary compares the two on one session you replay every time and treats any gap the manifest did not predict as a failed upgrade. The milestone compaction habits in the context engineering playbook assume a human who would notice a summary arriving early. An unattended lane has nobody watching.

Step 1: Write a lane manifest that says what the lane may be served

One file per lane names every served value you care about and the band it may sit in. Pinned lanes fill it with what they pin; unpinned lanes fill it with what they inherited at the last baseline, which is exactly what an upgrade can change.

# lane-manifest.yaml (illustrative shape): one per lane, read by your runner, never by the agent
lane: nightly-deps
harness: claude-code
cli_version: 2.1.272            # the version the current baseline was recorded on
provider_path: anthropic-api    # auth and billing path; defaults differ by path
served:
  model: [opus-5]               # allowed set; exactly one entry when you pin
  effort: high
  max_turns: 60
  wall_clock_min: 45
instructions:
  agents_md: not-loaded         # repo carries AGENTS.md and no CLAUDE.md
compaction:
  first_fire_turn: [16, 20]     # band from two baseline replays
  first_fire_tokens: [460000, 505000]
  summary: readable             # readable | opaque
survivors:
  pinned_facts: [freeze-friday, no-vendor-edits, release-branch]
  note_to_self: n/a             # "verbatim" on lanes that write one
cost_band_usd: [3.10, 4.40]     # client-side estimate per replay
declared_changes: []            # filled in before a bump, from the changelog range

Two rules keep it honest. Every field is pinned by config or observed at baseline, never copied from a docs page. And provider_path is a field because one CLI version can serve different features on different paths: Claude Code skips feature-flag fetching on third-party providers and with telemetry disabled, and without the flags it loads CLAUDE.md files only and can’t read AGENTS.md as project instructions (Claude Code environment variables). A canary on a different path tests a different product.

Twelve changes between Sep 4 and Sep 21, 2026, by what they moved. Each maps to a manifest field, and most change served behaviour on upgrade with no config edit.

Step 2: Record the baseline from wire and session logs, not from config

Record before you bump, on today’s version, from sources downstream of the harness where the served value is visible.

Manifest field	Where to read it	Why the config can’t tell you
Model served	A logging proxy or gateway between lane and provider; Claude Code’s -p --output-format json per-model breakdown; the model Copilot’s CLI prints per response	An unset key inherits the bundled default; an unchanged ID can front a new model
Effort	Request parameters at the proxy	Defaults belong to the model and move with it
Turn cap, wall clock	The run’s end reason, plus start and end timestamps	A renamed key can stop working with only a startup warning
Compaction fired at	Input tokens on the last request before compaction, plus the turn index; Claude Code’s opt-in x-claude-code-compaction and x-claude-code-context-compacted headers; Pi’s tokensBefore	Trigger points come from windows, reserves and counters the harness owns
Summary	Claude Code PostCompact payload; Pi’s compaction entry summary; Codex marked opaque	Readability depends on the provider path
Cost	total_cost_usd from the JSON output, or proxy token counts times your rate file	Estimates move whenever the model does

The Claude Code headers arrived in 2.1.273 behind CLAUDE_CODE_GATEWAY_HINT_HEADERS=1 and let a gateway log tell a compaction request from a work request, the cleanest wire-level record of when a lane compacted. The headless JSON’s cost figures carry the docs’ own warning: “Both figures are client-side estimates and can differ from your actual bill” (Claude Code headless docs). Use them for the diff, never for chargeback. Pi compacts when context tokens exceed the window minus reserveTokens (16,384 by default) and stores tokensBefore on each compaction entry (Pi compaction docs).

Record the turn index beside the token count. When an upgrade swaps the default model, the same text can tokenize differently; a compaction that moved in tokens but not in turns is a tokenizer change, not a moved threshold.

Step 3: Build one canned long session that forces a compaction

The canary is only as good as the session you replay. Build one per harness family and treat edits to it like edits to a test suite.

1. Fixture repo at a pinned commit. Same SHA every replay, from a clean checkout, with an AGENTS.md and no CLAUDE.md so instruction-file drift shows up.
2. Scripted prompts in a fixed order. Twenty to forty turns shaped like the lane’s real job: read, edit, test, fix.
3. Pinned facts early. At turn two, state three facts the lane must honour after compaction: a freeze date, a directory it must never edit, the release branch. Give each an ID.
4. A context push. Tool calls that read large files, so the session crosses the compaction point at least once, ideally twice.
5. Recall probes at the end. One question per pinned fact, plus one task that breaks a fact if the lane forgot it, such as an edit inside the forbidden directory.
6. The production flag line. Same unattended flags, permission mode and environment as the lane. The headless agent trust tier piece already pins that flag line per tier; reuse it.

Replay the baseline twice before you bump anything. If the two runs disagree on the compaction turn by more than a few turns, widen the band now, or every upgrade will fail on noise and people will learn to wave failures through.

Step 4: Bump one lane, replay twice, and diff everything served

Upgrade the CLI on the canary lane only; every other lane keeps its pinned version until this one passes. Then replay the canned session twice.

The second run exists because of documented Claude Code behaviour: in the first session after an install, or after an upgrade that adds a feature, “a flag-gated feature can be missing”, and it appears in the next session once the flags are fetched (Claude Code environment variables). Reading AGENTS.md and claude.ai skill and plugin sync are both flag-gated, so run 1 and run 2 can load different instructions from the same repo. Diff each run against the baseline and against each other. The full clean-home versus used-home differential for instruction files, synced skills and memory is a separate test; the canary only asks whether served behaviour moved.

Here is a weekly Claude Code bump from 2.1.272 to 2.1.278 through the canary. Values are illustrative; the mechanisms are the two changelog entries above.

Field	Baseline (2.1.272)	Run 1 (2.1.278)	Run 2 (2.1.278)	Declared?	Verdict
Model served	opus-5	opus-5	opus-5	n/a	pass
Effort	high	high	high	n/a	pass
First compaction (turn)	18	31	31	yes, 2.1.273	pass
First compaction (tokens)	485K	950K	955K	yes, 2.1.273	pass
AGENTS.md loaded	no	no	yes	no	fail
Pinned facts recalled	3/3	3/3	3/3	n/a	pass
Summary readable at PostCompact	yes	yes	yes	n/a	pass
Cost estimate per replay	$3.60	$5.10	$5.20	band raised with 2.1.273	pass

The compaction move was declared because someone read 2.1.273 before the bump. The AGENTS.md load was not, and it appears only in run 2, so a single replay would have passed this upgrade.

Two replays before the bump, two after. The second replay after the bump catches features that switch on only once the new version has run once.

Step 5: Check what survived the compaction, and whether you can still read it

Served model and effort are the easy half. The expensive drift happens inside the summary.

• Pinned facts. Score each recall probe and the forbidden-directory task. A lane that forgets a pinned fact fails the upgrade even if every other field passed; on a real run the next compaction will forget something you did not pin.
• The note-to-self. On lanes that write a handoff note before compacting, check it came back byte for byte. The compaction contract piece covers how the note crosses the cut; the canary checks it still does. Pi 0.87.0 changed the extension API today, so an extension that intercepts compaction can load cleanly and still stop doing its job.
• Readability. On Claude Code, confirm PostCompact still receives compact_summary; the summary lint depends on that payload. On Codex lanes whose provider path returns an encrypted compaction item, confirm the manifest says opaque and your outside evidence still captures the transcript. On Pi, confirm the compaction entry still carries readable summary text.
• Compaction failures. Count compaction attempts per run. Kimi’s compaction_max_attempts allows five total attempts for a failing compaction by default, and the 2.0.2 fix shows the failure was real after a model switch. A run that compacted once at baseline and three times after the bump has drifted, even if it finished.
• Per-model overrides. Pi’s compaction.modelOverrides falls back to the ordinary settings when no override matches the served model. When an upgrade changes the default model, a tuned lane quietly loses its override and the compaction point moves with no setting touched.

Step 6: Fail the upgrade on anything the manifest did not declare

Every difference between the baseline and either post-bump run must match an entry in declared_changes, or the upgrade fails and the fleet stays pinned.

# declared_changes for the 2.1.272 -> 2.1.278 bump (illustrative)
declared_changes:
  - field: compaction.first_fire_turn
    expect: [28, 34]
    source: Claude Code 2.1.273 changelog entry on advisor-tool turn counting
    owner: platform-oncall
  - field: compaction.first_fire_tokens
    expect: [900000, 967000]
    source: same entry; the lane now compacts near the model default
    owner: platform-oncall
  - field: cost_band_usd
    expect: [4.60, 5.60]
    source: follows from the later compaction point
    owner: platform-oncall

1. Declare before you bump. Someone reads the changelog range, writes the expected value or band for each field it touches, cites the line, and signs it.
2. Fail on the undeclared. Any diff with no matching declaration fails the canary, however harmless it looks.
3. Flag the declared-but-absent. A declared change that did not show up means the lane is not on the path you think. Check provider_path first.
4. Pin or declare, then replay. Either set explicitly the value the lane used to inherit (model, effort, compaction window, cap) and replay until the diff is clean, or declare the new behaviour with a reason.
5. Promote, then rebaseline. A passing canary becomes the fleet’s pinned version, and its run 2 becomes the next baseline.

Silent config death is what this catches best. Kimi CLI 0.32.0 renamed two [loop_control] keys, including max_steps_per_run, and the changelog says the old keys “stop working with a rename warning at startup”. A lane whose step cap lived under the old key lost it on upgrade. The canary sees a run far longer than the manifest allows; nobody had to read stderr.

Run two sibling tests in the same gate: the deny-rule canary repo from restricted mode as fleet policy, which proves permissions still hold, and the guard contract suite, which proves a renamed tool did not slip past a hook matcher.

Step 7: For auto-routers you cannot pin, log the model per prompt and cap the minutes

Copilot Auto picks a model per prompt and bills at that model’s rate, so there is nothing to pin. The manifest declares an allowed set instead.

• Log the model served on every prompt. GitHub shows the routed model per response (printed in Copilot CLI, on hover in Chat) but documents no export, so capture it from the CLI output into the run log.
• Fail on any model outside the allowed set, including a newly added one. A router gaining a model is an upgrade you did not install.
• Replay three times and keep a distribution. Per-prompt routing turns the compaction point and cost into ranges; set the bands from the spread.
• Cap wall-clock minutes per run. You cannot cap the rate of a model you did not choose, but you can stop the run, with the same minutes cap the ledger piece applies to forced swaps.

Where AI CLI upgrade testing misleads you, and the signal for each

Noise read as drift. Nondeterministic tool output or routing moves the compaction turn. Signal: two baseline replays on the old version disagree. Fix: widen the band; never skip the baseline pair.

A tokenizer change read as a threshold change. Signal: compaction tokens moved while the compaction turn held. Fix: declare it as a model change.

First-session artifacts. Signal: run 1 matches the baseline and run 2 does not. Fix: judge the upgrade on run 2; keep run 1 as evidence.

The wrong provider path. The canary runs on an API key while production runs through a gateway or a subscription login. Signal: the canary passes and a production lane still diffs. Fix: a provider_path mismatch fails the canary before it runs.

A stale fixture. Signal: a baseline replay changes with no upgrade in between. Fix: hash the fixture and prompt script into the manifest and rebaseline when either hash changes.

Estimates read as bills. Signal: the canary’s cost diff and next month’s invoice disagree. Fix: diff tokens by model and price them from your own rate file.

Upgrades are change events, and the fleet layer owns them

Nothing in this runbook lives inside the harness. The manifest, the canned session, the two-plus-two replays and the fail rule belong to the layer that decides which lanes run which binary, the same layer that holds kill switches, approvals and evidence. In a multi-agent command center, that layer is where an overnight CLI update becomes a change record with a baseline, a diff and a name on the approval.

Vendors will keep shipping defaults that are better on average. The canary is for the lane that is not average, and for the night the release notes skipped the line that mattered.

FAQ

Does pinning the CLI version stop default drift?

Pinning stops the harness from changing under a lane, but not the provider. An unchanged model ID can front a new model, as kimi-for-coding did on Sep 11, and hosted routers such as Copilot Auto pick per prompt. Pin the version, set model and effort explicitly, and replay the canary on every bump anyway.

Why does Claude Code behave differently in the first session after an upgrade?

Some Claude Code features are flag-gated. Its docs say that in the first session after an install, or after an upgrade that adds a feature, the feature can be missing until the flags are fetched. Reading AGENTS.md and claude.ai sync are both gated, so replay twice after every bump and judge run 2.

Sources

• Kimi Code: What’s New — K2.8 Preview, unchanged ID, default max (Sep 11); 24-hour cap removed (Sep 14); 20 MB media rule (Sep 17)
• Kimi Code CLI changelog — 0.43.0 compaction attempts; 2.0.2 fix; 0.32.0 key renames
• Codex 0.153.4 release notes — Astra as bundled default (Sep 4)
• Codex 0.155.0 release notes — streamed remote compaction (Sep 17)
• Claude Code changelog — 2.1.273 (Sep 15), 2.1.277 (Sep 18)
• Claude Code environment variables — feature-flag fetching and the first-session rule
• Claude Code headless docs — JSON cost figures are client-side estimates
• GitHub changelog: Copilot auto model selection — per-prompt selection and billing (Sep 14)
• Pi changelog — 0.86.0 overrides (Sep 19); 0.87.0 breaking changes (Sep 21)
• Pi compaction docs — trigger, reserveTokens, tokensBefore