← AUTOMATER NEWSROOM

Hugging Face's harness-hub Ships a LoRA That Lifts Open Qwen3.8-27B's pi Coding-Agent Score to 64.7%, at a Quarter of the Tokens

harness-hub's Qwen3.8-27B-pi-adapted LoRA specializes an open 27B model for the pi coding agent, matching max-effort thinking accuracy with far fewer tokens. The model card also corrects its own launch-day benchmark numbers.

Diagram showing Qwen3.8-27B base model plus a rank-32 LoRA adapter feeding into the pi coding agent, with stat boxes showing SWE-bench Pro rising from 39.2 percent to 64.7 percent, Terminal-Bench 2.1 rising from 39.5 percent to 55.8 percent, and roughly half the turns per task plus about 4.5x fewer tokens than thinking mode.Diagram showing Qwen3.8-27B base model plus a rank-32 LoRA adapter feeding into the pi coding agent, with stat boxes showing SWE-bench Pro rising from 39.2 percent to 64.7 percent, Terminal-Bench 2.1 rising from 39.5 percent to 55.8 percent, and roughly half the turns per task plus about 4.5x fewer tokens than thinking mode.
Original illustration summarizing Hugging Face's harness-hub release: a rank-32 LoRA adapter specializes open Qwen3.8-27B for the pi coding agent, based on the corrected figures from the harness-hub/Qwen3.8-27B-pi-adapted model card (under half the turns per task and about 4.5x fewer tokens than thinking).

Hugging Face’s harness-hub team has published Qwen3.8-27B-pi-adapted, a rank-32 LoRA adapter that turns the open Qwen3.8-27B model into a specialist for one specific tool: the open-source pi coding agent. Rather than training a bigger base model, the team ran reinforcement learning inside real pi sessions, and the result is a small file you drop on top of a model you already run.

For anyone serving open-weight models for local or self-hosted coding agents, this is a concrete, reproducible example of a cheaper lever than buying a bigger model: adapt the model you have to the exact harness you run it in.

Hugging Face model card page for harness-hub/Qwen3.8-27B-pi-adapted showing the LoRA adapter description and benchmark tables
The Hugging Face model card for harness-hub/Qwen3.8-27B-pi-adapted, the LoRA adapter the harness-hub team published for the pi coding agent, as captured from huggingface.co. · Original source

What actually changed, with the corrected numbers

The model card reports these pass@1 results under the pi harness (one attempt per task, 100-step budget, thinking off, same sampling settings for both runs):

Benchmark Qwen3.8-27B (base) Qwen3.8-27B-pi-adapted Change
SWE-bench Pro (hard 51) 39.2% 64.7% +25.5 points (15 newly solved, 2 lost, p=0.0023)
Terminal-Bench 2.1 (86 tasks) 39.5% 55.8% +16.3 points (16 newly solved, 2 lost, p=0.0013)

On the tasks both versions solved, the adapter needed a median of 23 turns on SWE-bench Pro against 59 for the base model, and 8 turns against 18 on Terminal-Bench 2.1 — under half the work for the same outcome.

Bar chart comparing median turns per task between the base Qwen3.8-27B model and the pi-adapted LoRA, showing 59 versus 23 turns on SWE-bench Pro and 18 versus 8 turns on Terminal-Bench 2.1.Bar chart comparing median turns per task between the base Qwen3.8-27B model and the pi-adapted LoRA, showing 59 versus 23 turns on SWE-bench Pro and 18 versus 8 turns on Terminal-Bench 2.1.
On tasks both versions solved, the pi-adapted LoRA needed roughly half as many turns as the base model, per the harness-hub model card on Hugging Face.

With thinking turned off, the adapter also matches the base model’s own highest-effort “thinking” accuracy (64.7% on SWE-bench Pro, 55.8% vs 54.2% on Terminal-Bench 2.1) while generating about 4.5x fewer tokens per task, because nearly all of the base model’s extra tokens under thinking go to reasoning the adapter has learned to skip.

Worth flagging for anyone citing this launch: the figures above are corrected numbers. Hugging Face co-founder Clement Delangue’s announcement post and the same-day MarkTechPost coverage both cite a 29.4% to 62.7% SWE-bench Pro jump. The model card itself now explains why that figure changed: nine of the base model’s evaluation rollouts failed silently because the public tunnel between the sandboxes and the model server went down mid-run. After the team added a check for that failure mode and reran every affected rollout on both sides, the base model’s own score actually rose, from 29.4% to 39.2%, since several of its failed rollouts had been scored as losses rather than retried. The adapter’s score also shifted slightly, from 62.7% to 64.7%, but because the base model’s corrected score rose more than the adapter’s did, the net advantage narrowed from +33.3 points to +25.5 points. The operator lesson is as newsworthy as the model itself: a benchmark claim is only as good as the rollout-integrity checks behind it, a launch-day tweet can go stale within hours of the model card it’s citing, and a narrower corrected margin is still a large, reproducible one.

How it was trained

The adapter was trained with AsyncGRPO, an experimental trainer in Hugging Face’s TRL library, using a rank-32 LoRA (alpha 64) applied to the q, k, v, o, gate, up and down projections. Rollouts ran inside Harbor tasks served through OpenEnv in Hugging Face sandboxes, with every tool call screened by safeagents to block reward hacking, such as an agent fetching benchmark answers directly from the Hub. Training mixed 750 “scale800” tasks with 300 “TermGrade” tasks, rewarding verifier correctness plus a small tool-efficiency bonus, on 2x NVIDIA B200 GPUs. The run was stopped at step 191 of a planned 1,000, and the published checkpoint is step 160, picked by tracking scores on held-out subsets during training. Hugging Face says it will publish the full recipe in a release blog post.

Running it

The model card documents four deployment paths. The reference setup is vLLM:

vllm serve Qwen/Qwen3.8-27B \
 --enable-lora --max-lora-rank 32 \
 --lora-modules qwen3.8-pi=harness-hub/Qwen3.8-27B-pi-adapted \
 --language-model-only --enable-prefix-caching --mamba-cache-mode align \
 --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
 --default-chat-template-kwargs '{"enable_thinking": false}' \
 --max-model-len 131072

--mamba-cache-mode align matters because Qwen3.8-27B is a hybrid architecture mixing Gated DeltaNet linear-attention blocks with Gated Attention blocks; without it, prefix caching breaks and every agent turn re-reads its whole history. Thinking must stay off, since turning it on adds nothing to accuracy and roughly quadruples token use. The other three paths — a GGUF adapter for llama.cpp, a build for macOS’s Llama.app, and a PEFT merge into a standalone ~54GB BF16 checkpoint for engines without LoRA support — all carry the same thinking-off requirement, and pi itself must be pointed at the OpenAI Completions API rather than the Responses API, since that’s what the adapter was trained and evaluated against.

The caveats that matter before you cite these numbers elsewhere

The model card is explicit that these results are not comparable to the numbers on Qwen3.8-27B’s own model card, which use different harnesses (Terminus, Claude Code), maximum-effort thinking, larger step budgets and the full SWE-bench Pro set rather than the 51-task hard subset used here. A sibling adapter trained the same way for the OpenCode harness scores 51.0% on the same SWE-bench Pro subset — a reminder that “harness adapter” results only transfer within the harness they were trained for, and the adapter only works on top of Qwen3.8-27B, not any other base model. Results are also self-reported on small evaluation sets by the team that built the adapter.

For operators weighing local or open-weight coding agents, the practical takeaway is less about this specific 25-point jump and more about the method: a narrow, cheap LoRA trained inside the exact harness and tool-calling pattern an agent actually uses can close a large chunk of the gap to a model’s own best-effort mode, at a fraction of the inference cost. Automater covered the general harness-adapter approach when Hugging Face introduced it; this release is the first fully documented, downloadable instance of it for the pi agent specifically. Anyone already running open-weight models behind a coding agent or serving local checkpoints with llama-server flags tuned for agent workloads now has a concrete adapter to test against their own pi sessions before deciding whether a bigger model is actually necessary.

Sources

  1. harness-hub/Qwen3.8-27B-pi-adapted · Hugging Face
  2. clem 🤗 on X: "Introducing Qwen3.8-27B-pi-adapted 🤗 We adapted Qwen's model specifically for the pi coding agent using reinforcement learning and a LoRA adapter. The results, measured as pass@1: 🚀 SWE-bench Pro (Hard 51): 29.4% → 62.7% 💻 Terminal-Bench 2.1: 39.5% → 55.8% ⚡ 55–62% fewer tool ca…
  3. New LoRA Adapter Lifts Qwen3.8 Coding Success From 39% to 65% - MarkTechPost
  4. GitHub - earendil-works/pi: AI agent toolkit: unified LLM API, agent loop, TUI, coding agent CLI · GitHub