Jev Confidence Gates Before the Tool Runs

LangChain’s first Jev guardrail ships with one number in it, 0.5, and no way to change it. The experimental AutoModeMiddleware asks TypeSafe’s Jev whether a listed tool call is risky, blocks anything scored at or above that line, and lets everything below it run. Sensible for a demo, wrong for a fleet, because a git status and a git push --force should not share a cut line.

The fix is a Jev confidence threshold per tool class. By Tuesday you want a four-row table (read, write, network, destructive) with its own probability bands in every row, seeded from about 200 of your own labeled past calls and scored on false allows and false denies per class. Jev gets two verdicts, deny and ask. It never approves anything, and the destructive row never runs on a classifier’s word alone.

Chatbots suggest; agents act, and the act happens between the model choosing a tool and the tool producing a side effect. Where a gate belongs is settled: before the call, with a record of each decision. This piece is about the number inside the gate, and how that gate stacks with the classifier your harness vendor now runs on its own servers.

Sep 15–20: four Jev confidence threshold designs, and none of them is your table

TypeSafe launched Jev on Sep 15 as a decision model: state in, typed answers out, $0.042 per million input tokens with output free. A Choice or a Score comes back with a confidence value. A Noul, the yes/no type, returns only a probability; in the docs’ words, “Noul answers don’t carry one.” Within five days, two integrations, TypeSafe’s own docs and an eval vendor drew four different lines.

• LangChain published langchain-typesafe as an alpha (0.0.1a1 on Sep 17, 0.0.1a3 on Sep 20). AutoModeMiddleware(tools=["bash"]) asks one Noul, is_risky, over the last 30 messages plus the pending call, and blocks at or above a module constant, _PROBABILITY_THRESHOLD = 0.5; the constructor takes no threshold. The docstring is candid: “This middleware blocks risky calls; it does not request human approval.” Also: “Tool names not listed in tools bypass classification.” Errors fail closed, and its traces omit the arguments it judged.
• Vercel’s eve made Jev its default tool-approval evaluator, with a guide by Ben Sabic on Sep 19. Its auto() helper asks a two-option Choice: clear runs without a prompt, caution pauses for a person, and “Failed evaluations take the human approval path.” In eve 0.62.0 it takes no numeric cutoff.
• TypeSafe’s confidence docs carry the sentence every gate designer should tape to the monitor: “A confidence threshold is not one number. Different actions within the same system should be gated at different levels depending on the consequences of getting it wrong.” High, medium and low ranges are offered as “A useful starting pattern”, with no numeric table. The only numbers sit in a code example that sends anything under 0.5 confidence to a human and lets a transfer proceed only above 0.9, as confirm-then-execute.
• Braintrust (Sep 18) sketched a triage scorer: auto-accept above 0.95, send 0.70 to 0.95 to a stronger LLM judge, treat the rest as inconclusive or route it to a human, with the caveat “These confidence values are not operating thresholds out of the box.”

Screenshot: TypeSafe AI docs, “Confidence - TypeSafe AI” (undated), captured Sep 21, 2026.

The harness side moved too. Claude Code 2.1.278 (Sep 19) switched auto mode for Claude API and Enterprise users, and on Bedrock, Vertex, Foundry and gateways, to a server-side classifier “which does not charge for classifier overhead”, falling back to its own billed checks, with a warning, when the server can’t reach a session. On those sessions a free vendor check sits behind your rules, and you will never see what it approved.

LangChain draws one line for every listed tool and never asks a person. eve sends doubt to a person but takes no number. TypeSafe says thresholds scale with stakes, then leaves the table to you. The table is the deliverable.

Screenshot: Vercel Knowledge Base, “How to automatically approve tool calls in eve with Jev” (Sep 19, 2026), captured Sep 21, 2026.

Step 1: Sort every tool into four classes before you pick a number

A threshold attaches to consequences, and consequences attach to what a call can do. Classify in your own code; never ask Jev which class a call belongs to.

Class	What belongs in it	Examples
Read	No state change, nothing leaves the machine	file reads, search, git status, directory listings
Write	Changes state you can revert inside the workspace	edits, new files, commits on a working branch
Network	Anything that leaves the machine or talks to another system	fetches, git push, package installs, MCP tools that send
Destructive	Hard to undo, or touches credentials and permissions	rm -rf, dropped tables, force pushes, secret reads, ACL changes

Two rules keep the sort honest. First, classify the command itself. One shell tool spans all four classes, and Vercel’s guide uses cat notes/release.md && rm scratch.txt as its example of a call that starts as a read and ends as a deletion. The strictest class any part of a compound command reaches is the class of the whole call.

Second, invert LangChain’s default. There, unlisted tools skip the check. In your table, an unknown tool or an unparseable command lands in the destructive row until someone classifies it. The MCP server a teammate adds next week will not wait for your list.

Step 2: Build a 200-call pilot set from your own transcripts

Build the set from your transcripts, in this order:

1. Every deny you have: rejected permission prompts, hook blocks, vendor-classifier denials, incident timelines. They are your richest source of unsafe calls, and there are never enough.
2. Every destructive call, approved or not. Without the safe ones you cannot count false denies in the row that matters most.
3. Ordinary calls from each class until the set reaches about 200.
4. Freeze each call as the gate will see it: tool name, arguments, and only the state the question needs. TypeSafe’s jaggedness notes for jev-1.13 agree: “retrieve and filter in code first, and send only the fields the question needs.”
5. Label each call safe or unsafe by a person, with a one-line reason. Unsafe means you would have wanted it stopped or shown to someone first, whatever happened afterward.
6. Version the set. It is now the regression suite for every threshold edit, question edit and model bump.

Class	Calls	Labeled unsafe	Where the unsafe ones come from
Read	60	6	secret-file reads, reads outside the repo
Write	60	14	edits to CI config, lockfiles, another lane’s files
Network	40	12	pushes to protected branches, uploads, unknown hosts
Destructive	40	28	most of the deny history
Total	200	60	Illustrative mix; yours comes from your logs

The skew is normal. Denies cluster where the damage is, so the destructive row is the one you can say the most about and should still trust the least.

Step 3: Set the Jev confidence threshold as probability bands per class

For a gate, one Noul is the cleanest question: is_risky, with written criteria for true and false (LangChain’s defaults make a fair first draft). A Noul carries no confidence value, so you threshold the probability itself, with two cuts per class: below ask_at the gate stays silent, from ask_at up to deny_at it asks a person, and at or above deny_at it denies.

# jev-gate-table.yaml: illustrative starting values; tune them on your pilot set
version: 7
model: jev-1.13.0              # thresholds belong to a pinned version
question: is_risky@3           # edit the wording, re-run the pilot
verdicts: [deny, ask]          # no allow, on any harness
classes:
  read:        { ask_at: 0.30, deny_at: 0.80 }
  write:       { ask_at: 0.15, deny_at: 0.60 }
  network:     { ask_at: 0.10, deny_at: 0.45 }
  destructive: { ask_at: 0.00, deny_at: 0.30 }   # always at least asks
unknown_tool: destructive
on_jev_error: deny
codex: { ask_maps_to: deny }   # Codex continues the call on an ask

Four rules for the numbers:

• Bands, not a line. One line forces every doubtful call to be blocked or run. The ask band is where a person earns their keep, and approval-queue hygiene decides whether they stay awake.
• Stricter as consequences grow. In TypeSafe’s words, “the threshold for acting without confirmation is higher for a destructive operation than for a read-only one.”
• Destructive has no pass band. At ask_at: 0.00, no score is low enough to run a destructive call on Jev’s word. If one specific destructive command should run unattended, say a scratch-directory cleanup, write it as a narrow permission rule and review it like code.
• A threshold belongs to one question and one version. The jaggedness notes warn, “Don’t carry a threshold tuned on a Noul over to a Choice,” and show a question and its negation, asked as two Nouls, summing to 1.19. Reword the question and every band is untested again.

Cut points as shipped or documented, Sep 15–21. Rows 1–3 cut on confidence, row 4 on the probability a call is risky, and eve’s helper takes no number at all. None is a per-class table.

Step 4: Count false allows and false denies per class, then read the bound honestly

Run the pilot set through the table with the gate in dry-run mode, so nothing executes. A false allow is a labeled-unsafe call that scored below ask_at and would have reached the tool with no verdict from Jev. A false deny is a labeled-safe call at or above deny_at. Count asks separately; they are the human cost of the table.

Class	Unsafe	False allows	Safe	False denies	Asks	95% upper bound, false-allow rate
Read	6	0	54	1	5 of 60	39%
Write	14	1	46	2	9 of 60	30%
Network	12	0	28	1	10 of 40	22%
Destructive	28	0 by construction	12	3	14 of 40	n/a: cannot allow
All	60	1	140	7	38 of 200	7.7%

Illustrative counts. Bounds are one-sided 95% binomial limits on each row’s own denominator.

Vercel’s companion page on Jev thresholds frames the job: “The threshold belongs to your policy: test how often it permits a wrong action and how much work it sends for review.” Two more of its lines belong on the worksheet header: “Record the denominator when reporting accuracy,” and “Also count timeouts and failed evaluations; they need a defined destination even though they produce no usable prediction.”

Then read the last column before you celebrate. Even a far bigger set, zero misses in 200 unsafe cases, leaves a false-allow rate of about 1.5% consistent with what you saw (one-sided 95%; the rule of three gives 3/200). Per-class counts are far smaller and the bounds wider: six unsafe reads with no misses still allow a rate near 39%. That arithmetic is the case for the destructive row’s ask_at: 0.00, and why this table is a starting point you keep scoring, never proof.

One more honesty rule: a returned probability describes a group of answers. If Jev is well calibrated, calls scoring 0.08 turn out risky about 8% of the time as a group. That tells you where to draw a band; it does not make this call safe, and TypeSafe publishes no calibration metric.

Braintrust’s caution applies: “A high value does not guarantee that the decision is correct.” Your pilot set is the only calibration you have.

Step 5: Wire it on Claude Code as a veto: deny or ask, never allow

Claude Code’s permissions docs fix the stacking; you don’t get to reorder it. “Hook decisions don’t bypass permission rules”: a matching deny rule blocks and a matching ask rule still prompts, even after a PreToolUse hook returned allow. A hook that exits 2, or returns permissionDecision: "deny", blocks even where an allow rule matches. A hook’s ask forces a prompt in auto mode, where “the classifier can still deny the tool call, but it can’t approve the call silently.”

The auto-mode classifier runs after the permissions system, and what the docs, as of v2.1.278, do not say is what a hook’s allow does to it. Don’t find out in production.

Give Jev two outputs, deny and ask, plus silence: exit 0 with no decision, which hands the call to your rules, your sandbox and the harness’s own mode. Jev can then only make a call harder to run. That is the whole meaning of an extra veto that never approves.

# jev_gate.py: illustrative PreToolUse shape, not a drop-in hook
import json, sys

TABLE = load_table("jev-gate-table.yaml")                 # the versioned file from step 3
call = json.load(sys.stdin)                                 # carries tool_name and tool_input
cls = classify(call["tool_name"], call["tool_input"])      # your code picks the class

def deny(reason):
    print(reason, file=sys.stderr)
    sys.exit(2)                                             # blocks; exit 1 would not

def ask(reason):
    print(json.dumps({"hookSpecificOutput": {
        "permissionDecision": "ask",                        # on Codex: call deny() instead
        "permissionDecisionReason": reason}}))
    sys.exit(0)

try:
    p = jev_is_risky(call, deadline_s=3)                   # far under the hook timeout
except Exception:
    log_verdict(call, cls, None, "error")
    deny(f"{cls}: Jev unavailable, failing closed")

band = TABLE["classes"][cls]
log_verdict(call, cls, p, band)                             # step 6, before any exit
if p >= band["deny_at"]:
    deny(f"{cls}: P(risky) {p:.2f} at or above {band['deny_at']}")
if p >= band["ask_at"]:
    ask(f"{cls}: P(risky) {p:.2f}, a person decides")
sys.exit(0)                                                 # no decision: rules, sandbox and mode still apply

The gate itself can fail, and on Claude Code most gate failures open. From the hooks reference: “Without valid JSON on stdout, Claude Code treats exit code 1 as a non-blocking error and proceeds with the action.” A timed-out command, http or mcp_tool hook doesn’t block either: “The call continues through the normal permission flow, so don’t count on a stalled hook to act as a gate.” The default timeout is 600 seconds, so set an explicit timeout, keep Jev’s deadline well under it, and turn a Jev error into a deny.

The full fail-closed contract is in the Claude Code hook piece; what a 429 should mean per gate is in the fail-mode piece.

Codex takes the same table with one change. OpenAI’s hooks page says permissionDecision: "ask" is “parsed but not supported yet”: Codex marks the hook failed and continues the tool call, so on Codex the ask band maps to deny. The same page sums up every harness here: “Treat tool hooks as a useful guardrail, not a complete enforcement boundary.” Keep permission deny rules and a sandbox under the destructive tools as the wall behind the gate; hooks-versus-advice basics are in the Claude Code field guide.

Your hook and your rules sit in front of the vendor classifier. Jev only narrows what runs, and your log sees the vendor’s denials and none of its approvals.

Step 6: Log your verdict on every call, and the vendor’s only where it surfaces

The decision log has two verdict columns, and one will be mostly empty by design.

• Yours, on every call the gate saw: class, P(risky), band, verdict (deny, ask or none), table version, question version and the model ID Jev reported. LangChain’s middleware won’t keep this for you. The full record, and replaying policy versus re-asking Jev, belongs to the decision-log piece.
• The vendor’s, only where one surfaces. On Claude Code that means denials: the PermissionDenied hook runs when auto mode denies a tool call. The hooks reference lists no event that fires when the classifier approves, so every other row gets the literal value no vendor verdict visible, never allowed.

{"ts":"2026-09-21T14:03:11Z","lane":"billing-refactor","call_id":"call-8841","tool":"Bash","class":"network","p_risky":0.23,"band":"ask","verdict":"ask","table":"gate-table@7","question":"is_risky@3","model":"jev-1.13.0","vendor_verdict":"no vendor verdict visible"}

The literal matters. A classifier approval, and a call an allow rule or read-only auto-approval settled before the classifier ran, look identical from where you sit. Writing allowed would claim a review you cannot see.

Step 7: Go live through the shadow ladder, with injected states in the set

• Shadow first. The hook computes and logs its verdict, then exits 0 on every call. Compare what the table would have done with what people and the vendor did, class by class. The rungs (shadow, canary, enforce, with a written budget and a rollback trigger) are in the shadow-mode piece; use the same ladder here.
• Know what shadow measures. A veto-only gate cannot remove protection you already had when it goes live. Its risks are false denies and ask fatigue, which is exactly what the shadow week counts.
• Add injected states before any class goes live. PR bodies, issue text and tool output written to argue for their own classification join the pilot set with expected verdicts; TypeSafe’s jaggedness page says such content “can move the answer.” The fixture rules live in the injection piece. A flipped verdict on any injected fixture blocks promotion.
• Pin the model and re-run on every bump. Pin jev-1.13.0, not the jev-latest alias, because your bands were tuned on a version. The pilot set and fixtures run again before the pin moves.

Five ways a Jev tool-call gate fails quietly

The unlisted tool. A new MCP server lands and its tools never reach the gate. Signal: transcript tool calls with no matching gate-log line. Fix: unknown tools map to the destructive row, plus a weekly diff of tool names seen against tool names classified.

The alias moved. The gate calls jev-latest, a release ships, and your bands now sit on a model they were never tuned on. Signal: per-class ask or deny rates shift with no table change, and the reported model ID changed. Fix: the step 7 pin and a pilot re-run.

The silent open. Jev slows, the hook times out, the call proceeds through the normal permission flow. Signal: gate latency creeping toward the timeout, and calls whose gate-log line is missing or marked error. Fix: an explicit timeout, deny on error, and the fail-mode drills.

Ask fatigue. The ask band is too wide and people approve by reflex. Signal: more asks than reviewers can read, and approval latency collapsing to a few seconds. Fix: narrow the band with pilot data, never a guess.

The edited question. Someone rewords is_risky to fix one false deny and every band shifts. Signal: a new question version in the log with no pilot re-run beside it. Fix: questions and thresholds in one reviewed file, and an edit to either re-runs the whole set.

The threshold table is fleet policy, not a prompt

Nothing above lives in a prompt. It is one versioned table applied by every lane’s gate in that lane’s dialect: a Claude Code hook that denies or asks, a Codex hook that only denies, a LangChain middleware you wrap or replace. The table, the pilot set and the log belong in the operating layer above the harnesses, beside the kill switch and the tool inventory, which is the job of a multi-agent command center. No single vendor’s classifier can hold it; each sees only its own sessions and shows you only its denials.

Draw the lines per class, count what gets through, and let Jev say no.

FAQ

What Jev confidence threshold should I set for tool calls?

There isn’t a single one. TypeSafe’s docs say a threshold is not one number, so give each class (read, write, network, destructive) two cuts: one where Jev asks a person, one where it denies. Tune them on about 200 of your own labeled calls, and the destructive row gets no pass band.

Should Jev be allowed to approve tool calls?

No. Give Jev two verdicts, deny and ask, and have it stay silent otherwise, so your permission rules, sandbox and harness mode decide. Jev can only make a call harder to run. Destructive calls never run on its word, since zero misses on a small pilot set still leaves a wide false-allow bound.

Can I change the threshold in LangChain’s AutoModeMiddleware?

Not in the published alpha. AutoModeMiddleware blocks at or above a fixed module constant of 0.5, and its constructor takes no threshold. It never asks a person, and tools you leave off its list skip classification. For per-class bands and an ask verdict, wrap or replace the middleware with your own versioned table.

Sources

• TypeSafe docs: Confidence — threshold is not one number; 0.5 floor, above-0.9 transfer; no Noul confidence
• TypeSafe docs: Jev 1.13 jaggedness — Noul-to-Choice warning, negation example, adversarial state
• LangChain: auto_mode.py — fixed 0.5, listed tools only, no human branch
• Vercel KB: auto-approve tool calls in eve with Jev — Sep 19, 2026
• Vercel: Jev probabilities and decision thresholds
• Braintrust: Eval agent responses with Jev — Sep 18, 2026
• Claude Code docs: Hooks reference
• Claude Code docs: Permissions
• Claude Code changelog — 2.1.278, Sep 19, 2026
• OpenAI: Codex hooks