Together AI's Agent Evals Found the Models Were Fine: Docs, Skills and Error Text Were the Bottleneck
Four months of coding-agent evals found Together AI's own docs, skills and error text — not the models — were the bottleneck. Its fixes are a checklist for any SDK owner.
Together AI published a four-month retrospective on how it makes its platform usable by coding agents, and the numbers undercut a common assumption: the models were mostly fine. The platform’s own documentation, skill files and API error messages were the bottleneck. Fixing that surface — not swapping models — lifted measured task completion from 67% to 96% on the docs-connected setup, with the skills path reaching 97% by October (Together AI).

Why developer experience is now agent experience
Together started the program after noticing developers were pointing coding agents at its SDK and CLI to wire up production inference — work that would take a human days and an agent minutes, in theory. In practice, the team had no visibility into whether agents were succeeding, wandering, or burning tokens failing. Their framing is worth adopting: as developers delegate more work, developer experience and agent experience converge. They point to GitHub as the benchmark — agents opened roughly 40% of all PRs in 2026, which they attribute to an API, CLI and docs designed to be machine-usable.
How the evals work
The methodology is simple enough to copy:
- Build a task list of real developer jobs — reserve GPU capacity, run batch inference, manage autoscaling. Tasks deliberately hide where information lives.
- Run each task three ways: a baseline agent with no special configuration, an agent with docs access via MCP, and an agent with a Together skill package.
- Automate it. Agents run in sandboxes, GitHub Actions handles setup and teardown, and a proxy records every API call so failures trace to a specific response.
- Score with an LLM judge into pass, incomplete, or fail — then investigate passing runs too, because a lucky recovery reveals a token-efficiency problem.
Teams running their own agent pipelines will recognize the shape from agent evaluation CI gates: the value is not the score, it is the reproducible failure you can hand to an owner.
Two failures that were nobody’s fault but the platform’s
The GPU cluster task. In early summer, Sonnet 4.6 passed only 4 of 15 attempts at a pricing-and-reserve-command task, even with skills enabled. The post-mortem found two documentation defects: a CLI reference sample output that looked like live inventory, and a skill file that linked to the GPU clusters marketing page instead of the pricing table. The agent never saw a real price, so it guessed. Notably, Opus 4.8 passed 13 of 14 on the same config — it noticed the regions response had no prices and kept searching. That contrast is the post’s sharpest operational lesson: test with weaker models, because strong models silently paper over broken docs. The fix was two sentences in the skill pointing to the pricing table and the live regions endpoint. After that, the task passed 100% of the time for every model tested.
The lying error message. In a baseline run, Opus 5.5 called /v1/compute/instance-types and got a 400 telling it to use /v1/clusters/regions. The agent obeyed — and that route does not exist. It burned turns chasing a 404 the platform’s own error text had invited. Together fixed the message to point at /v1/compute/regions. Any developer following that error by hand would have been just as stuck, which is the point: error strings are documentation, and agents are the users who always read them.
| Snapshot | Baseline | Docs MCP | Skills |
|---|---|---|---|
| June 15 (Opus 4.8) | 75% | 67% | 75% |
| Sept 30 (three models) | 82% | 96% | 90% |
| Oct 8 (unified skill) | — | — | 97% |
Together is explicit that this is not a controlled comparison — models, tasks and skill versions changed between snapshots, and they credit only about 7 points of baseline movement to model improvements. The 20+ point gains on the guided setups came from the docs and skills fixes.
When the fix made things worse
The most instructive stretch is the unified-skill rebuild. Together collapsed 14 overlapping per-product skills into one, expecting a win. The first version flopped: 54 of 72 runs passed (75%), 18 went incomplete instead of 7, and cost per run more than doubled. Transcripts showed three recurring defects — a roughly 14,000-character router loaded on every task (57% more skill content than before), stale model names and a deprecated-feature guide agents wasted turns retrying, and small bugs in example scripts that derailed every agent that used them. Cutting the router to under 6,000 characters, moving samples to confirmed-live models, and fixing the scripts brought the final version to 70 of 72 runs (97.2%), with runs 20% faster at original cost. Consolidation without evals is a regression factory.
The finding echoes work on the other side of Together’s own house: in the Together Compiler write-up, a hand-curated knowledge hub made agent-driven kernel optimization 25-31% worse than letting agents work unguided. Guidance is load-bearing, and bad guidance is a liability.
What to do with this
For anyone whose product is consumed by coding agents — SDK owners, MCP server maintainers, internal platform teams — the post is effectively a checklist:
- Make everything a human can do in the GUI available through a CLI or SDK; agents work best with CLIs because help text documents itself.
- Treat docs as product surface:
llms.txt, a skill package, or an MCP server, plus CLI help and API error text. - Test across the capability spectrum; strong models hide your defects.
- Keep guidance lean, current, and verified against the live product.
The cost dimension matters too. Together’s companion piece on long-horizon agentic workloads shows why bloated skill content is doubly expensive: agent turns resend the whole conversation every step, so anything that changes early in the prompt — a fresh timestamp, a bigger tool list, a bumped reasoning-effort setting — re-pays the entire session unless prefix caching holds. The same discipline is documented in Claude Code prompt caching: keep the prefix stable and read the cache line to prove it.
The signal fits a pattern: Microsoft open-sourced Scope for testing whether agents can use your SDK, and Scope’s AX benchmark found only 16.6% of agent sessions ended with a working hosted database. Together’s four months of data is the strongest evidence yet that the gap is closable — and that the fixes are mostly unglamorous prose. The harness will be open-sourced next week.