CI Failure to Fix: The AI Agent CI Loop That Doesn't Thrash

Suppose a red check lands at 11:52 p.m. on a PR nobody is watching. Under a cloud agent that subscribes to the PRs it creates, the loop can be on attempt two before midnight and can stack more attempted fixes before anyone opens a laptop. An AI agent CI loop is cheap to switch on and expensive to leave uncapped, because every turn costs a clone, a context load, a model run, and a CI run, and nothing inside the loop knows it is going in circles.

Five guards fix that: a retry cap, a rule that another attempt needs a new failure signature, a thrash detector on the diff hash, a cost ceiling per PR, and a human gate with a timeout. Below: the state machine, the assertions the CI job makes before the agent runs again, an illustrative GitHub Actions job, and a state table for the design review. The agentic CI/CD overview covers what a self-healing pipeline is; this piece is the governor on its retry.

The requirement changed the day the loop closed with no person in it. A review bot that comments “you might try X” leaves a human to decide. An agent that pushes X, waits for CI, and reads the new failure as its next instruction has made the pipeline its own prompt. Whatever CI says, the agent hears a task.

What Cursor shipped on Aug 19, and the loop it closes

Cursor’s Aug 19, 2026 changelog, “Cloud Agents and Cursor Harness Improvements,” says cloud agents “automatically subscribe to PRs they create,” “fixing CI and addressing bot comments” (Cursor changelog). Two entries in the same release make the loop long-lived: /goal lets you “give the agent a long-lived objective to work towards until it’s fully complete,” with the example “fix all flaky tests and make CI green,” and /loop schedules recurring check-ins. Steering changed too: “Follow-ups wait for the next tool call instead of cutting the agent off mid-action,” so your stop lands after the current push.

Screenshot: Cursor changelog, “Cloud Agents and Cursor Harness Improvements” (Aug 19, 2026), captured Sep 13, 2026.

On Sep 10 the Projects launch put a coordinator above that loop. The blog says it “can watch a Slack channel, run on a schedule, or follow all your PRs, fixing CI and acting when they open or merge,” and describes the intended path: “Early on, you review each PR closely. As the fixes hold up, you review less, and the coordinator keeps working through the migration on its own” (Introducing Projects). Reviewing less is the goal. The guards below are what make reviewing less safe rather than merely quiet.

Screenshot: Cursor changelog, “Cursor Projects” (Sep 10, 2026), captured Sep 13, 2026.

Two other facts frame the runbook. AWS’s Sep 8, 2026 pattern for evaluating agents in GitHub Actions is a vendor’s own CI recipe with explicit budgets: poll for traces every 30 seconds for up to 10 minutes, invoke only after the runtime reports READY, and count “4 evaluators × 5 prompts = 20 judge calls per PR” as a cost, under the line “Without automated evaluation, agent quality is subjective” (AWS Machine Learning Blog). And Novee’s Black Hat 2026 research (Aug 5–6) documented CVE-2026-54316 in Claude Code Action, a quote-stripping bug that became command injection inside a GitHub workflow (Novee; claude-code-action). An agent that “addresses bot comments” reads untrusted text inside your CI. The news stops here.

Three ways an AI agent CI loop goes in circles

From inside the loop, every thrash looks like progress. Three shapes:

1. The flaky test. The failure is random. Every “fix” is noise, and the third attempt passes for the same reason the first failed: chance. The history now holds three unrelated changes to code that was never broken.
2. Fix-the-fix. Attempt one breaks a second test; attempt two fixes that and re-breaks the first. The diff oscillates, A, B, A, B, and each run is a genuine red check the agent is genuinely answering.
3. The wrong layer. The agent edits the test, adds a retry to the CI config, or bumps a lockfile until the check goes green. CI is satisfied, the bug is not, and a reviewer who reviews less merges it.

None of the three is visible from a single run. They show only across attempts, which is why every guard below keys on state that survives the run: failure signature, diff hash, attempt count, spend.

Five guards for the AI agent CI loop

Only Classify decides whether the agent runs again. Everything it needs is in the PR state comment.

Guard 1: cap attempts at three per PR

Three is the number. One attempt is the honest fix; the second is a legitimate correction; the third is where every oscillation starts. Count attempts per PR, not per check run, and reset only when a human pushes a commit of their own. The cap is a constant in the workflow, not a prompt instruction; the agent cannot be trusted to count itself when the count is what stops it.

Guard 2: require a new failure signature before another attempt

A failure signature is a hash of what failed, stripped of what varies between runs:

sig = sha256(
  sorted(failing_test_ids)              # "tests/api/test_login.py::test_refresh"
  + error_class_and_message_normalized   # digits, paths, timestamps, hex -> "#"
  + failing_job_name                     # "unit" and "integration" differ
)

Same signature twice in a row means the last fix changed nothing that mattered. Stop and hand off, even if the count says two attempts remain. A new signature means the loop learned something, which is the only time another paid attempt is worth it. This guard separates spending on information from spending on counting; the chart below shows the difference in dollars.

Guard 3: a thrash detector on the diff hash

Hash the agent’s diff on every attempt (the normalized patch). Two patterns stop the loop cold: a diff hash that matches any earlier attempt on this PR, meaning the agent proposed the same patch again, and an A-B-A alternation across three attempts. Add a path rule: a diff that touches test files, CI workflows, or lockfiles is not a fix but a change of subject, and it goes to the human gate with a label saying why. The thrash detector is the CI cousin of a stall flag: the run is active, and it is not making progress.

Guard 4: a cost ceiling per PR

Model usage, sandbox minutes, and CI minutes are three meters, and OpenAI’s Agents API pricing spells out the first two as separate lines: “Model usage is billed at the selected model’s API rates” and “OpenAI-hosted sandboxes use standard container rates” (Introducing the Agents API). Sum all three per attempt, write the running total into the state, and stop at the ceiling. The illustrative number below is $5 per PR at $1.50 an attempt; yours comes from a week of measured attempts, not a pricing page.

Illustrative, not measured: at $1.50 an attempt, no cap reaches $12 by attempt eight; a cap of three stops at $4.50; signature gating spends $6 because four attempts each surfaced a new failure. Run the cap and the gate together.

Guard 5: a human gate with a timeout, not a queue

When any guard trips, the job posts the state to the PR, applies agent-fix:needs-human, and stops. A person can grant one more attempt with agent-fix:approve-one, the only way past the cap. No answer in eight hours parks the PR; parked PRs never re-enter the loop on their own. The gate has to be cheap to clear and impossible to skip, or it becomes an approval queue nobody reads and the cap is theater.

State	Enter when	Exit to	Meter touched
CI red	A required check fails on an agent/* branch	Classify	none
Classify	Every red	Fix (new signature, under cap and ceiling) or Human gate	none
Fix	Classify passes	Re-run	model, sandbox
Re-run	The agent’s commit lands	CI red or Green	CI minutes
Human gate	Repeat signature, thrash, cap, or ceiling	Fix (approve-one) or Parked (8 h)	none
Parked	Declined or timed out	Manual only	none
Green	All required checks pass	Review queue; the agent stops	none

What the CI job asserts before the agent runs again

The guards live in CI because every attempt already passes through it. Before the fix step starts, the job asserts, in order:

1. The failing check is a required check on a branch the agent owns (agent/*), on a PR from this repository, not a fork.
2. The attempt count is below the cap, or the PR carries a fresh approve-one label that the job consumes.
3. The failure signature is new for this PR.
4. The last diff hash is not a repeat and not the second half of an A-B-A.
5. Cumulative spend is under the ceiling, with the coming attempt’s budget added in.
6. The agent’s token cannot merge, edit workflows, or reach production secrets; branch protection with required reviews does the merge part (GitHub Docs).
7. Bot comments and failure logs reach the agent as files it reads, never interpolated into a shell command or a prompt string the job builds. That is the Novee lesson applied to the fix loop.

After the attempt, the job asserts once more: the diff touched allowed paths only, no test was deleted or skipped, no workflow or lockfile changed. Then it appends the attempt to the state and uploads the patch. A separate gated job may apply and push it with a write credential the agent never receives.

# .github/workflows/agent-fix.yml (illustrative shape, not a drop-in)
name: agent-fix
on:
  workflow_run:
    workflows: ['ci']
    types: [completed]
concurrency:
  group: agent-fix-${{ github.event.workflow_run.head_branch }}
  cancel-in-progress: false # never two attempts on one PR
permissions:
  contents: read
  pull-requests: write # labels and the state comment
jobs:
  guard-and-fix:
    if: >-
      github.event.workflow_run.conclusion == 'failure' &&
      github.event.workflow_run.head_repository.full_name == github.repository &&
      startsWith(github.event.workflow_run.head_branch, 'agent/')
    runs-on: ubuntu-latest
    timeout-minutes: 25
    env:
      RUN_ID: ${{ github.event.workflow_run.id }} # env, never inline
      ATTEMPT_CAP: '3'
      CEILING_USD: '5.00'
      ATTEMPT_BUDGET_USD: '1.50'
      ALLOWED_PATHS: 'src/ lib/'
    steps:
      - name: Check out trusted controller scripts
        uses: actions/checkout@v4
        with:
          ref: main
          path: controller
          persist-credentials: false
      - name: Check out the candidate branch without credentials
        uses: actions/checkout@v4
        with:
          ref: ${{ github.event.workflow_run.head_branch }}
          path: candidate
          persist-credentials: false
      - name: Failure signature and logs, as files
        working-directory: candidate
        run: ../controller/scripts/ci/failure-signature.sh "$RUN_ID" sig.txt ci-log.txt
      - name: Load loop state from the PR
        working-directory: candidate
        run: ../controller/scripts/ci/loop-state.sh load > state.json
        env: { GH_TOKEN: '${{ github.token }}' }
      - name: Gate (exit 78 = hand to a human)
        id: gate
        working-directory: candidate
        run: ../controller/scripts/ci/loop-gate.sh state.json sig.txt
      - name: One fix attempt
        if: steps.gate.outcome == 'success'
        working-directory: candidate
        run: >-
          codex exec --sandbox workspace-write
          "Read sig.txt and ci-log.txt. Fix the failing tests. Do not edit tests, workflows, or lockfiles."
      - name: Assert the diff is allowed
        working-directory: candidate
        run: git add -N . && ../controller/scripts/ci/diff-policy.sh "$ALLOWED_PATHS"
      - name: Record the attempt and stage the patch
        working-directory: candidate
        run: ../controller/scripts/ci/loop-state.sh append sig.txt && git diff --binary > ../attempt.patch
        env: { GH_TOKEN: '${{ github.token }}' }
      - uses: actions/upload-artifact@v4
        with: { name: agent-fix-attempt, path: attempt.patch }
      - name: Hand to a human
        if: failure() || steps.gate.outcome == 'failure'
        run: controller/scripts/ci/park.sh # label needs-human, post the state
        env: { GH_TOKEN: '${{ github.token }}' }

The fix step runs codex exec because it is non-interactive by design, and the documented --sandbox workspace-write flag scopes file changes to the checkout (Codex non-interactive mode). The same shape works with Claude Code’s documented non-interactive -p mode, with allow and deny rules in settings.json carrying the path policy (Claude Code permissions). The concurrency block matters more than it looks: without it, a coordinator that fans out can start two attempts on one PR, and the second diff hash is measured against a state the first is still writing.

The state is one JSON block in a PR comment, readable by people and by the next run:

<!-- agent-fix-state -->
{
  "attempts": 2,
  "cap": 3,
  "spend_usd": 3.1,
  "ceiling_usd": 5.0,
  "history": [
    { "attempt": 1, "sig": "9f2c...", "diff": "4b7e...", "usd": 1.55, "result": "red" },
    { "attempt": 2, "sig": "c31a...", "diff": "77d0...", "usd": 1.55, "result": "red" }
  ]
}

Two attempts, two signatures, two diffs, still red, so attempt three is allowed. If it comes back with sig: "c31a..." again, the gate stops it, and the human reading the comment sees in ten seconds that the agent fixed the login test, broke the session test, and is out of ideas.

What breaks, and the signal that tells you

Failure mode	The signal	First response
Flaky test loop	Signature alternates between two values across attempts	Quarantine the test; the agent never “fixes” a flake
Fix-the-fix oscillation	Diff hash A, B, A	Thrash detector parks it; a human picks the layer to fix
Test, workflow, or lockfile edit	Diff touches tests/, .github/, or a lockfile, or adds a skip marker	Block at the diff-policy step; label the reason
Comment injection	Bot comment text reaches a shell or a built prompt	Files only; the job never interpolates comment text
Ceiling counts one meter	Spend under ceiling while sandbox minutes climb	Add sandbox and CI minutes to the running total
Hand-edited state	State comment hash mismatch	Treat as tampered; park and ask a human

The first two rows arrive in week one. The last two arrive in month three, after someone helpful hand-edits the state comment to unstick a PR and the ceiling turns out to have counted tokens alone. Wire the state comment into the same feed as your cost anomaly alerts, so a PR that trips the ceiling three times in a week is a signal on its own.

The loop is operating-layer work, and CI is where it lives

None of the five guards is a better prompt. “Do not repeat yourself” in the system prompt is a wish; the diff hash in the state comment is a control. The counter, the signature store, the ceiling, and the gate live outside the agent, in the layer that decides whether a run starts and who signs before anything lands, which is the operating layer the fleet thesis keeps arriving at. The state comment is also the replay: every attempt, its signature, its diff, its cost, in order, the same discipline fleet replay asks for at fleet level. Once the loop has a governor, the next questions are the ones this piece skips: what an overnight PR must pass before merge, how to fail a PR when the agent itself regresses, and the policy for review agents commenting on the same PR. A loop that starts in Slack rather than CI has its own gates.

FAQ: AI agent CI loops

How many times should an AI agent retry a failing CI job?

Three attempts per PR, counted in the workflow rather than by the agent, and only while each attempt produces a new failure signature. A repeat signature or a repeated diff stops the loop early. After the cap, a human can grant one more attempt with a label; eight hours of silence parks the PR.

What is a failure signature in CI?

A hash of what failed with the run-to-run noise removed: the sorted failing test IDs, the error class and message with digits, paths, and timestamps normalized, and the job name. Two runs with the same signature failed the same way, so a fix between them changed nothing that mattered.

Should an AI agent be allowed to edit tests to make CI pass?

No. A diff that touches test files, skip markers, CI workflows, or lockfiles is a change of subject rather than a fix, and the job should block it at the diff-policy step and hand the PR to a person with the reason labeled. Humans can decide a test is wrong; the loop cannot.

Sources

• Cursor changelog: “Cloud Agents and Cursor Harness Improvements” (Aug 19, 2026)
• Cursor, “Introducing Projects” (Sep 10, 2026)
• AWS, “Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions” (Sep 8, 2026)
• Novee, coding-agent flaws presented at Black Hat 2026 (Aug 5–6)
• anthropics/claude-code-action
• OpenAI, “Introducing the Agents API” (Sep 10, 2026)
• OpenAI Codex, non-interactive mode
• Claude Code docs, permissions
• GitHub Docs (branch protection, Actions concurrency)