EAGLE-4 Ships as a Single Adaptive Drafter for Speculative Decoding, Claiming Up to ~3x Faster Inference on vLLM and SGLang
EAGLE-4 ships a single speculative decoding drafter that sizes token trees from measured costs, adapting to batch size and hardware without manual tuning.
EAGLE-4, the latest generation of the speculative decoding framework for large language models, shipped its first public release on October 20, 2026. The headline: one drafter model now adapts its speculative strategy to whatever hardware, batch size and workload it finds itself running on — no hand-tuned tree shapes; SGLang needs no offline calibration, vLLM a single calibration run. The EAGLE-4 repository reports speedups up to ~2.99× on vLLM with NVIDIA B300 GPUs and ~2.07× on SGLang with AMD MI355X.
How It Works
Speculative decoding speeds up LLM inference by using a smaller, faster “drafter” model to propose several tokens ahead, which a larger target model then verifies in parallel. The key tension has always been choosing the right tree shape — how many tokens to draft and how wide to branch at each step. Too shallow, and you leave speed on the table. Too deep or wide, and you waste compute on rejected tokens.
EAGLE-4 solves this by measuring draft and verification costs and sizing the speculative tree accordingly. The architecture uses a sliding-window draft layer, a low-rank candidate head and an append-only draft cache. Dense and latent-MoE draft structures are both implemented. The measured target pair for this release is Qwen3.8-27B with its EAGLE-4 draft.
The drafter registers through hooks that each engine exposes and does not patch the engine at runtime. On SGLang, the server prices the tree from costs measured during served rounds; vLLM sizes its static tree once via offline calibration, as detailed in the Engine Support section below.
Engine Support
eagle-harness ships as a pip-installable package (eagle_harness) that plugs into patched builds of three engines:
- SGLang — NVIDIA B300 (CUDA 13.0) and AMD MI355X (ROCm 7.2; ROCm wheels still under verification, container image not yet published), using the V2 speculative worker. Tree pricing happens from costs measured during served rounds, so no offline calibration is needed.
- vLLM — NVIDIA B300 (CUDA 13.0), using Model Runner V2. Uses a static tree whose depth stops on calibrated costs, produced once with
eagle-harness calibratewhile GPUs are idle. - TokenSpeed — experimental, NVIDIA CUDA only, with no throughput claims yet.
The patched engine builds ship as pinned wheels and container images from ghcr.io/eagle-hq, not as source trees in the main repository.
Measured Results
The support matrix reports every measurement with its conditions — separately for loading, greedy correctness, request features and throughput. All figures use the SPEED-Bench client at temperature 0 and represent the mean of two server starts.
| Engine | GPUs | Concurrency | No-spec tok/s | EAGLE-4 tok/s | Speedup |
|---|---|---|---|---|---|
| vLLM | 4× B300, TP4 | 1 | 241.6 | 722.8 | 2.99× |
| SGLang | 1× MI355X, TP1 | 8 | 552.7 | 1145.9 | 2.07× |
| SGLang | 4× B300, TP4 | 8 | 1479.4 | 2820.6 | 1.91× |
| vLLM | 4× B300, TP4 | 8 | 1699.9 | 2883.9 | 1.70× |
| SGLang | 1× MI355X, TP1 | 32 | 1269.2 | 1980.8 | 1.56× |
| SGLang | 4× B300, TP4 | 64 | 3925.3 | 5281.3 | 1.35× |
The pattern is clear: speedup is highest at low concurrency (2.99× on vLLM with a single request) and narrows as batch saturation kicks in. This is exactly what you would expect from adaptive tree sizing — when the GPU is already saturated, there is less headroom to fill with speculative tokens, and the drafter correctly narrows its tree.
Every row also passes the greedy correctness check, meaning the speculative server produces the same output as the non-speculative baseline.
Licensing: Source-Available, Not Open Source
EAGLE-4 is released under a custom source-available license, not an OSI-approved open-source license. Commercial use is permitted within specific conditions, including a US $20 million aggregate revenue threshold measured over any consecutive 12-month period. Exceeding that threshold requires a separate commercial agreement. Commercial use of a competing offering requires a separate written license regardless of revenue.
This is a practical constraint for teams evaluating speculative decoding in production. The repository also notes that engine forks (SGLang, vLLM, FlashInfer) retain their original project notices.
Why It Matters
Speculative decoding is one of the most impactful levers for self-hosted LLM inference costs — especially for agent workloads that make many sequential calls. Previous approaches required operators to profile their specific hardware, batch patterns and model combinations, then hand-tune tree shapes for each configuration. EAGLE-4’s adaptive approach collapses that into a single drafter that measures and adjusts at runtime.
The multi-engine, multi-vendor GPU support (NVIDIA B300 and AMD MI355X) is also notable. The release includes ROCm wheels with AITER and Triton kernels targeting GFX950, and the repository acknowledgements credit contributors Aditya Nandakumar and Andy Luo for the AMD port and kernels. For teams already running SGLang or vLLM in production, the install path is a patched engine wheel plus pip install eagle-harness==0.1.0 from PyPI.
The experimental TokenSpeed integration and the source-available license are worth watching. TokenSpeed has no throughput claims yet, and the license terms mean commercial operators should review the Commercial License Agreement before deploying. But for research teams and smaller deployments evaluating speculative decoding, EAGLE-4 removes the profiling and tuning step that has kept the technique from being a drop-in optimization.
For more on the shifting inference landscape, see our coverage of Inception’s Mercury 3 diffusion LLM and AWS TorchNeuron making Trainium a native PyTorch device.
