AI Agent Environment Setup Is Two-Thirds of the Failure: Fleet Readiness Before the Prompt

Picture an illustrative run: an agent spends eighteen minutes in pip install, never reaches the task, and narrates what it would have done with a working OpenCV. The task never started. The bill did.

Two benchmarks put numbers on that last year, and it remains the most under-managed failure mode in fleet operations. AI agent environment setup, meaning the image, the lockfile, the toolchain, and the shell state an agent inherits before it reads a line of your prompt, accounted for 65.04% of failures in GitTaskBench and held repository-setup success to 38.9–57.4% in SetupBench. Both papers tested strong agents. The ground under them was the problem.

The move here is a preflight gate: six checks that run before any prompt, produce a setup-readiness score, and return GO or NO-GO. A NO-GO routes to a fix-image job or a human. It never routes to the agent, because an agent handed a broken environment does what the papers say it does: installs things twice, invents constraints, changes a shell that will not persist, and reports success.

Two papers, one number: 65.04% of failures happened before the task began

GitTaskBench (arXiv 2508.18993, v1 Aug 26, 2025, revised Sep 2025) gave agents 54 real tasks across 7 domains and 18 repositories: colorize a photo with one repo, transcribe audio with another, extract PDF text with a third. The repositories average 204 files, roughly 1,274 functions, and 52.63k lines of code; human completion averaged 1.34 hours per task. The best pair in the paper, OpenHands with Claude 3.7, passed 48.15% of tasks with an execution-completion rate of 72.22%; the abstract notes a later record of 62.96% for RepoMaster with Claude 3.5. Then the authors sorted every failure into five bins. Environment setup, “dependency conflicts, missing binary wheels, or absent system-level libraries,” took 65.04% of them. Their own gloss: “env setup doesn’t improve results but causes most failures.”

Screenshot: arXiv, “GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging” (2508.18993), captured Sep 13, 2026.

SetupBench (arXiv 2507.09063, Jul 11, 2025) isolated the bootstrap skill on purpose: 93 instances across 7 language ecosystems and 5 database engines, each starting in a bare Linux container and ending with a deterministic success command. Repository setup succeeded 38.9–57.4% of the time depending on the model; local database configuration, 20.0–53.3%. The abstract reports that agents spent 38–89% of their actions on steps an optimal human would not have taken; the detailed ten-instance analysis in Table 4 reports 38.17–68.77%. The three failure patterns the authors name will look familiar from your own logs: incomplete development tooling installation (runtime deps installed, test tooling ignored), hallucinated task constraints (ports and flags the task never mentioned), and non-persistent environment modifications, where a tool installed in one shell is gone when the harness opens the next one.

Screenshot: arXiv, “SetupBench: Assessing Software Engineering Agents’ Ability to Bootstrap Development Environments” (2507.09063), captured Sep 13, 2026.

One more number from GitTaskBench, because it is the budgeting argument: raising the per-iteration timeout from 120 s to 1,800 s raised both completion and pass rates, at the cost of more tokens, which the authors read as evidence that “environment setup may be the primary time-consuming step.” That is the news. The rest is the gate.

Why a coordinator makes setup failures more expensive, not rarer

A single agent that fails setup wastes one environment’s worth of tokens. A coordinator that spins up subagents in fresh, isolated environments repeats the setup once per subagent, and every one of them rediscovers the missing libGL.so on its own. Setup theater scales with fan-out, and the fan-out metering runbook already shows what that does to a bill. The coordinator then receives N reports that say “installed dependencies and completed the task,” and nobody else reads them.

AI agent environment setup, gated: six checks before the first prompt

Each gate has a check, a fail signal, and an owner. The score at the end is a weighted count of gates passed, logged with the run and compared to a threshold you set per repository. Below the threshold, the agent never starts.

Six gates, one score, two exits. NO-GO routes to a fix-image job or a human, never to the agent.

The chart is the reason the gate runs before the prompt and not as advice inside it.

Share of all agent failures in GitTaskBench (arXiv 2508.18993): 65.04% environment setup and dependencies; 34.96% all other causes combined.

Gate 1: Start from a golden image pinned by digest, with system libraries baked in

“Absent system-level libraries” sits in GitTaskBench’s environment-setup bin next to dependency conflicts and missing wheels, and no amount of agent cleverness installs libGL faster than a base image that already has it. Build one golden image per stack, pin it by digest rather than by tag (the Docker docs cover digest references), bake in the compilers and system libraries the repo’s own Dockerfile or contributing guide names, and rebuild it on a schedule from a job, never from inside an agent run.

# illustrative golden image; the digest is the pin, the tag is a comment
FROM python:3.12-slim@sha256:<digest-from-your-registry>
RUN apt-get update && apt-get install -y --no-install-recommends \
      build-essential libgl1 libglib2.0-0 ffmpeg git \
    && rm -rf /var/lib/apt/lists/*
ENV PIP_DISABLE_PIP_VERSION_CHECK=1

On Windows desks the same idea is a WSL distro exported once and imported per run, so every agent session starts from the same disk image instead of the one last week’s agent modified (Microsoft Learn documents wsl --export and wsl --import). The cold-restart survival table already lists what survives a reboot inside WSL and Docker; a golden image makes the answer “nothing you did not bake in,” which is the answer you want.

Fail signal: any apt-get, brew, or winget call inside an agent transcript. That is the agent telling you the image is incomplete.

Gate 2: Honor the lockfile, and treat a lockfile diff as a dependency PR

Dependency conflicts are the other big item in that bin, and most are self-inflicted: the agent runs pip install <package> with no version, the resolver picks today’s release, and NumPy’s ABI no longer matches the OpenCV wheel that was fine on Friday. The fix is a policy, enforced by the only install command the harness is allowed to run.

Stack	Allowed install command	What it refuses to do
Python (uv)	uv sync --locked	update uv.lock; it fails when the lockfile is missing or out of date (Astral docs)
Python (pip)	pip install -r requirements.lock --no-deps	pull transitive deps the lockfile did not list
Node	npm ci	write package-lock.json; it fails on drift (npm docs)
Rust	cargo build --locked	update Cargo.lock
Go	go build -mod=readonly	edit go.mod or go.sum

If the agent’s branch changes the lockfile, that commit is a dependency change, and the review policy for agent PRs says a human reads it; a CODEOWNERS rule on lockfile paths enforces that in GitHub. The agent is welcome to propose a bump. It is not welcome to bump on the way to something else.

Fail signal: a lockfile in the agent’s diff, or an install log that resolved a package the lockfile pins.

Gate 3: Pin the toolchain and check it against a manifest, not the README

“Incomplete development tooling installation” is SetupBench’s first pattern: the agent installs what the app needs and skips what the tests need, because the README covers one and tox.ini covers the other. The manifest is a small file in the repo that names every runtime and tool the smoke test depends on, and Gate 3 is a script that diffs reality against it.

# illustrative: toolchain.yaml, checked before any prompt
python: '3.12.6'
node: '22.11.0'
tools: [pytest, ruff, tox, ffmpeg, git]
databases: [postgres@16]

# illustrative: fail fast on any mismatch; exit 3 means "gate could not pass", not "agent failed"
python --version | grep -q "3.12.6" || { echo "::error::python mismatch"; exit 3; }
for t in pytest ruff tox ffmpeg git; do
  command -v "$t" >/dev/null || { echo "::error::missing $t"; exit 3; }
done

Fail signal: command not found anywhere in the first twenty lines of an agent transcript. If the agent is discovering the toolchain, the gate did not run.

Gate 4: Run the smoke test in a fresh shell, because the agent’s shell lies

SetupBench’s third pattern is the one that costs the most human time: the agent installs a tool, exports a PATH entry, runs the tests, reports success, and the next shell (yours, or the harness’s) cannot find the tool. The paper’s example is pnpm, installed and then unavailable to the evaluation harness in a fresh shell. Whatever the agent did to the environment during setup is a suggestion until a new login shell agrees.

# illustrative: the smoke test runs in a fresh login shell, never the agent's
docker exec -i "$CTR" bash -lc 'cd /work && make build && pytest -x -q tests/smoke' \
  || { echo "::error::smoke test failed in a fresh shell"; exit 4; }

One build and one test is enough; the point is proof that the environment survives a process boundary. Where you do want persistence, the paper’s own recommendation is right: write changes to a profile file a login shell sources, source it, then summarize what changed.

Fail signal: the smoke test passes inside the agent’s session and fails in the fresh shell. Log both; the pair is the diagnosis.

Gate 5: Give setup its own budget in minutes and tokens, then enforce it

GitTaskBench found that longer timeouts help and that longer timeouts cost tokens, and both are true at once because setup is where the time goes. So budget it separately from the task: wall-clock minutes for the gate sequence, and tokens the agent may spend on anything setup-shaped before the task prompt is issued, both checked by the harness rather than by the agent. Illustrative starting points: 10 minutes and 30,000 tokens for a repo the golden image already covers; 25 minutes and 100,000 tokens for a repo new to the fleet.

SetupBench’s Table 2 is the reason the token cap is not optional. Among its repository-setup rows, the highest-scoring model averaged roughly 1,158k tokens and 42.9 steps per instance, against about 323k tokens for the lowest. The best setup agent is also the most expensive one, and a coordinator will pick it every time unless the budget says otherwise.

Fail signal: the setup budget is exhausted and the transcript is still installing. That run is NO-GO regardless of what the agent says next.

Gate 6: Compute the readiness score before the prompt, and log it with the run

The score is boring on purpose. Weight the gates, add up what passed, compare to a per-repo threshold, and write the result into the run’s metadata next to the model name and the prompt hash.

Gate	Weight	Pass condition
Golden image	25	digest matches the fleet’s pinned digest
Lockfile honored	20	locked install exits 0; no lockfile diff
Toolchain pinned	15	manifest diff is empty
Smoke test	25	build plus one test pass in a fresh shell
Setup budget	15	gates finished within both caps

Illustrative thresholds: 85 to GO on a repo the fleet has run before; 100 on a repo touching production data or one that arrived by untrusted intake. A NO-GO creates a ticket that names the failed gate and its owner, and the coordinator moves on to work that is READY. Nothing about this needs a model; it is the harness doing harness engineering instead of asking the agent to do it for itself.

Once the environment scores GO, the same discipline hands off to the agent evaluation gate, which waits for a runtime to report READY before it invokes a single prompt, the pattern AWS published on Sep 8, 2026. Readiness before the prompt, at both layers.

Where the gate itself fails, and the signal for each

The golden image rots. Signal: a rising count of apt-get lines in transcripts across repos that used to score 100. Fix: rebuild on a schedule from the fleet’s manifest and diff the package list against last week’s; a diff is a review.

The lockfile is honored and still wrong. Signal: the locked install exits 0 and the smoke test fails on an import error. Fix: the lockfile was generated on a different platform (macOS wheels, Linux run); regenerate it inside the golden image and never on a laptop.

Timeouts hide as agent failures. Signal: transcripts that end mid-install with no error and a “completed” summary. Fix: exit codes 2–4 from the gates mean “could not run,” reported separately from “failed,” and never counted as agent regressions.

Hybrid fleets drift. Signal: the same repo scores GO in the cloud sandbox and NO-GO on the laptop. Fix: one manifest, one digest, both hosts, and the hybrid-fleet runbook’s identity-per-host rule so you know which host produced which score.

The agent games the smoke test. Signal: a test file edited in the same run that first failed it. Fix: smoke tests live outside the agent’s writable path, and the fresh-shell rule applies to the test files too.

Memory carries stale setup. Signal: an agent “remembers” that a repo needs numpy<2 from a session three images ago. Fix: setup facts are scored per image digest, and the memory bakeoff rule applies: recall that harms the task is a measured failure.

Readiness is a property of the desk, and no prompt can supply it

You can tell an agent “make sure the environment is set up correctly,” and the papers above are the measured result of that instruction. Environment readiness is a property of the desk: the image the fleet pins, the install commands the harness permits, the budget the meter enforces, and the score the run carries with it. A chatbot’s wrong guess about a dependency costs one reply. An agent’s costs an environment, a fan-out’s worth of tokens, and a report that says done.

That is why this belongs in the agentic-ops layer beneath every harness on the machine, alongside stall flags and cost caps, and inside no single agent’s system prompt. The prompt gets better on its own. The ground under it only gets better if someone owns it.

FAQ: AI agent environment setup readiness

Why do AI coding agents fail at environment setup so often?

Because setup is systems administration under uncertainty: system libraries the README never mentions, resolver choices that change daily, test tooling separate from runtime tooling, and shell state that does not persist. GitTaskBench attributed 65.04% of failures to it; SetupBench measured repository-setup success at only 38.9–57.4% for strong agents.

What should a preflight check include before an AI agent runs?

Six gates: a golden image pinned by digest, a locked dependency install, a toolchain diff against a manifest, a build plus one test in a fresh shell, a setup budget in minutes and tokens, and a readiness score with a per-repo threshold. Anything below the threshold routes to a fix-image job, never the agent.

Should an AI agent be allowed to install dependencies itself?

Only through a locked install command that honors the lockfile, and only inside a budget. A version-free pip install invites the resolver drift that produces ABI mismatches, and a lockfile change is a dependency PR a human reviews. The agent can propose a bump; it should not bump in passing.

Sources

• arXiv 2508.18993, “GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging” (v1 Aug 26, 2025; v2 Sep 2025): https://arxiv.org/abs/2508.18993
• arXiv 2507.09063, “SetupBench: Assessing Software Engineering Agents’ Ability to Bootstrap Development Environments” (Jul 11, 2025): https://arxiv.org/abs/2507.09063
• AWS Machine Learning Blog, “Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions” (Sep 8, 2026): https://aws.amazon.com/blogs/machine-learning/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions/
• Docker documentation, image digests and pinning: https://docs.docker.com
• Microsoft Learn, WSL distribution export and import: https://learn.microsoft.com
• Astral, uv lockfile and locked sync: https://docs.astral.sh
• npm documentation, npm ci: https://docs.npmjs.com
• GitHub Docs, CODEOWNERS and required reviews for lockfile paths: https://docs.github.com