NVIDIA's Agent Substrate Multiplexes Idle Agent Sessions Onto a Shared Kubernetes Worker Pool, Plus NVKE and NVKR in Public Preview

AI agents spend most of their time idle: waiting on a human prompt, a tool result or a model response. NVIDIA’s Nov. 10, 2026 post Agent Substrate: Advancing Kubernetes for Agents at Scale explains why that is awkward on Kubernetes. Creating a Pod for an agent means a trip through the API server, admission, etcd, the scheduler, the kubelet, image pulls and network setup. That suits a service running for weeks, not a sandbox that must run a 10 ms bash command and get out of the way hundreds of times per second. Teams choose between one warm Pod per agent, paying for idle compute, or Pods on demand, paying a cold start on every step and losing agent state.

Actors over a worker pool

Agent Substrate keeps Kubernetes for what it is good at, running a pool of ready Pods, and adds a separate control plane mapping many actors — agents and their environments — onto that smaller pool. An idle actor is snapshotted out of its sandbox, gVisor or a microVM, memory and filesystem included, freeing its worker; the next call restores it into whichever worker is free. Waking it skips the API server, scheduler and kubelet, while the workers stay ordinary Kubernetes Pods, so quotas and placement policies still apply.

NVIDIA calls it an open-source, security-hardened, zero-trust runtime “engineered to safely run millions of agents and tools with 10x higher density than standard containers.” Both numbers are vendor framing; your gain depends on how long your own sessions idle.

The three workloads it targets

• Coding assistants and agent serving. The agent alternates between model calls and work in its own environment: cloning a repository, editing files, running builds. It must persist between turns, and a platform serving thousands of sessions cannot afford a warm sandbox each.
• RL post-training. Demand arrives in bursts as rollout batches dispatch actions, a run can grow from a few hundred sandboxes to hundreds of thousands, and the slowest environment decides when the trainer gets its next batch — so cold starts leave GPUs idle.
• Research GPU sandboxes. An agent needs a GPU to compile CUDA code, benchmark a kernel or run a calculation, then delete the sandbox.

Upstream changes you can read now

Change	Reference	What it fixes
Request parking	substrate #221, then #1229	A turn arriving while every worker is busy waits instead of failing
Per-container limits	#859, microVM sandboxes	A helper container over its memory limit is stopped alone; the sandbox survives
Device pluggability on DRA	#752	Each container requests only the devices it needs; NVIDIA GPUs first

Inference signals driving sandbox lifecycle

The deeper integration is with NVIDIA Dynamo, whose router already sees where reusable KV cache lives, which workers are busy with prefill or decode, and how load spreads across the GPU fleet. Substrate and the agent harness add session identity, request priority, tool-call boundaries and lifecycle events; Dynamo returns projected time to first token, completion time, cache readiness and GPU-pool saturation. Substrate uses those to keep an actor resident, suspend it during a longer inference wait, or resume it as the response nears completion — and at fleet scale to tune warm-worker capacity, so you stop activating CPU sandboxes that would only queue behind a saturated GPU pool.

NVKE and NVKR, in public preview

A second NVIDIA post the same day, Manage Kubernetes Clusters for GPU Infrastructure with NVIDIA Kubernetes Engine, opens two Apache 2.0 projects. NVKR builds reproducible node images and packages from version-pinned sources; NVKE provisions and manages clusters through a REST API and controllers on a management cluster. nvkectl bootstrap turns a temporary kind cluster plus three servers from NVIDIA Infra Controller into that management cluster, which then creates customer clusters with their own control planes. Node images carry checksum-pinned Kubernetes 1.36 packages, and nodes must present an attested SPIFFE identity before receiving join credentials. Both projects are public preview, explicitly not recommended for production workloads, with APIs subject to change.

OpenShell goes to CNCF

NVIDIA’s KubeCon announcement contributes OpenShell, its runtime for securing and governing autonomous agents, to the CNCF as a Sandbox project. It also confirms what the technical post leaves implicit: Agent Substrate is a Google-originated CNCF Sandbox project that NVIDIA contributes to and wires up with Dynamo and KAI Scheduler, and Substrate is not a named DSX OS component. If you already run OpenShell, that governance move follows the enforcement-outside-the-agent story.

What to do, and what to wait on

• Classify the workload first. Multiplexing pays when sessions are bursty and their state must survive between turns; long, continuously busy steps may still be cheaper as one Pod per session.
• Try NVKE locally. The post points at a single-machine quickstart needing no GPU and no NVIDIA account.
• Assume Substrate’s interfaces move. APIs are changing, GPU support is being rebuilt on Dynamic Resource Allocation, and parts of the Dynamo integration are experimental. The Nov. 10 post still carries a literal placeholder where the coding agents guide link belongs.
• Plan for GPU restore constraints. A checkpoint usually restores only onto the same GPU model, sometimes only the same driver version; Substrate falls back to a clean cold start when nothing matches.
• Demand your own density numbers. Vendor framing is not a measurement; what matters is your idle time per session and what a warm sandbox costs you.

Across all three documents, the agent environment is becoming a scheduled, checkpointable object rather than a container that lives as long as the session. The near-term work is measuring how long your sessions idle — the way Modal measured before adopting NVIDIA’s Vera CPUs — and knowing where a cluster layer fits in the open-source agent stack.