NVIDIA's Agent Knowledge Foundation: An Open Toolkit for Grounding Agents in Company Data
At GTC Berlin, NVIDIA added the Agent Knowledge Foundation to its open Agent Toolkit: retriever models, an ontology layer and Kumo predictors that ground agents in company data.
At GTC Berlin, NVIDIA expanded its open-source NVIDIA Agent Toolkit with a new component it calls the NVIDIA Agent Knowledge Foundation — a bundled set of libraries and open models meant to let enterprise agents find, understand and predict from company data without that data leaving its existing systems. For teams currently stitching together embedding models, rerankers, document parsers and a governance wrapper from separate vendors, it is a concrete open-source alternative path, announced on the NVIDIA blog on October 21.
What actually shipped
NVIDIA frames agents as needing an open stack that spans models, harness, runtime, data and tools — the same framing we discussed in the open-source AI agent stack serious builders run. The Agent Toolkit now bundles:
- NVIDIA Agent Knowledge Foundation — the new layer, built on CUDA-X libraries
- NVIDIA open models — including Nemotron Retriever and the new Kumo family
- NVIDIA OpenShell — a secure agent runtime boundary
- NVIDIA NemoClaw — secure agent blueprint reference designs
- Tools and skills — such as NeMo Relay and NeMo Compass
The Knowledge Foundation is the news. It combines:
- CUDA-X libraries: cuDF, cuGraph, cuML, cuOpt and cuVS, plus the new Auto-Ontology library, which translates raw company data into business concepts — customers, products, orders — and maps how they relate.
- Nemotron Retriever models for embedding, reranking and document parsing.
- Kumo Structured open models — Kumo Tabular, Kumo Relational, Kumo Time Series and Kumo Forecast — for prediction over structured business data.
The pitch is data grounding without data movement: NVIDIA says companies can use the foundation without moving data, rebuilding workflows or changing governance controls. Those are vendor claims, but the underlying pieces are downloadable: per NVIDIA’s open-models announcement, Nemotron Retriever models are available on Hugging Face, partner platforms and as NIM microservices on build.nvidia.com, and Kumo open models are on Hugging Face and build.nvidia.com.
Why it matters for agent builders
Retrieval-grounded agents usually fail on the boring parts: parsing messy documents, mapping records to business entities, and reranking results fast enough to stay inside a usable latency budget — the failure modes the context-engineering playbook spends most of its time on. The most interesting evidence in the announcement is where partners measured those pieces:
- Amazon OpenSearch Service improved relevance with fewer search steps and virtually no added latency using the Nemotron Retriever embedding model.
- SAP’s fine-tuned reranker ran nearly 3x faster.
- ServiceNow improved reranking quality while cutting latency by nearly a quarter.
- Zoom is using Nemotron Retriever models for document understanding, search relevance and context retrieval in Zoom AI Services.
Palantir, Databricks and Snowflake are also named as early users — NVIDIA’s own supply chain organization runs the foundation through Palantir Foundry, Ontology and AIP.
The governance half of the story
The knowledge layer arrives alongside a security framing. In his GTC Berlin keynote, Jensen Huang described an agent as five elements — a model, a harness, a sandbox, knowledge and tools — and argued that privileges, policy enforcement and monitoring must adapt dynamically because “every single time an agent is launched, the privileges are different.” The keynote introduced the NVIDIA OpenShell Reference Architecture for Safe AI Agents: an open-source, policy-controlled runtime with kernel-level sandboxing that sits between the agent and the systems it can affect. OpenShell provides the toolkit’s runtime boundary; the newest NemoClaw blueprint adds a step-by-step starting point for improving agent intelligence over time.
An agent that both knows your data and is boxed to least privilege is a stronger default than either half alone.
How to evaluate it
If you run coding or business agents over a private corpus, the pragmatic test is narrow:
- Benchmark retrieval on your own documents. Swap your current embed/rerank pair for Nemotron Retriever on a fixed eval set and compare relevance at equal latency, not vendor benchmarks.
- Check the ontology mapping. Auto-Ontology only helps if your entities (customers, orders, tickets) map cleanly; run it against a sample warehouse schema before committing.
- Try structured prediction only where a baseline exists. Kumo Tabular and Kumo Relational replace a per-task ML pipeline — worth piloting where you’d otherwise not build one at all.
- Keep the sandbox honest. OpenShell’s kernel-level sandboxing is the control that makes a knowledge-connected agent deployable; test that policy enforcement actually fails closed. Whatever data connectors you wire in afterward, run them through the MCP server inventory ritual before an agent gets to query company systems.
The stack-diagram below shows how the pieces fit:
Caveats
Everything here is NVIDIA-published; the adoption list and performance figures are vendor-reported, with no independent benchmarks yet. “Open” also spans a range — models carry open weights on Hugging Face, but production deployment is steered toward NIM microservices on NVIDIA infrastructure. And the Knowledge Foundation competes with retrieval stacks teams already run on LangChain, LlamaIndex or a vector database plus a reranker API; switching costs are real.
Still, a single open-source bundle that pairs retriever models with an ontology layer, structured-data predictors and a sandboxed runtime reduces the assembly tax on grounded agents. That is a practical consequence for anyone building agents that must answer from company data rather than training-set memory — and it is available today.
