Sentry Conversation Export: Build Regression Cases From Recorded Answers

A support agent gave the wrong refund advice, but rerunning the question now produces a correct answer. That rerun cannot tell you whether your evaluator would have caught the original mistake. Sentry conversation export lets you preserve the recorded answer, attach independently chosen expected truth, and score that historical output without calling the model again.

This tutorial is for a developer maintaining an instrumented Next.js agent with authorized access to nonsensitive conversations and an existing Braintrust project. You will export a bounded set of conversations, inspect their completeness, map approved records into a disposable dataset, and score the answers already recorded. The finished artifact includes a mapper, an explicit admission record and a deterministic scorer with a deliberately wrong control. All example data and expected observations below are illustrative; these scripts have not been executed for this article.

Sentry’s October 28, 2026 article, Five things you can do with Sentry’s Agent Tracing, provides the background: exported conversations can become evaluation records. The durable lesson is how to keep an observed answer separate from expected truth and refuse records whose apparent final reply is incomplete.

Establish the recording boundary

1. Use an existing Next.js application with tracing enabled and a working AI route.
2. Confirm compatibility with the documented recipe: Sentry JavaScript SDK 11 or later, Vercel AI SDK 7, Node.js 22 or later, and Sentry CLI 0.44 or later.
3. Work only with conversations authorized for export to your evaluation platform.
4. Keep the tutorial dataset separate from an established regression dataset.

These requirements come from Sentry’s conversation-to-evaluation recipe. An already instrumented application is the starting point, so this guide does not replace your route, choose a model or install another tracing pipeline. You need TypeScript tooling capable of running scripts, such as the tsx runner mentioned in that recipe, and the Braintrust SDK installed in your project. Braintrust’s installation documentation documents npm install braintrust --save. Use your existing authorized SDK environment and project access.

A conversation ID must follow the chat across requests. The Next.js Agent Tracing reference documents Sentry.setConversationId() before AI calls. Place this inside your existing request handler after obtaining the application’s session ID:

import * as Sentry from '@sentry/nextjs';

// Inside the existing handler, before its AI calls:
Sentry.setConversationId(conversationId);

Use the session identifier already threaded through your application. Do not give every request a new ID or reuse one ID for unrelated customers. Sentry’s Conversations documentation recommends short opaque identifiers with alphanumeric characters, dashes or underscores. The case label task:refund-policy-01 below belongs to our evaluation metadata; it is not the Sentry conversation ID.

Recording deserves a deliberate boundary. The current Next.js documentation says generative AI inputs and outputs are recorded by default. It also documents global controls that disable them. Do not enable collection on a sensitive route just to fill an evaluation dataset. Missing payloads should remain missing, and the importer should reject them. For a broader discussion of tracing boundaries, see finding and removing unwanted prompt exports.

If you temporarily change tracing settings for a nonsensitive development exercise, record their previous values first. The recipe’s tracesSampleRate: 1.0 is a tutorial configuration, not a universal production recommendation. Restore your previous sampling and collection policy after collecting the approved cases. This exercise needs recorded text, but it does not justify collecting unrelated traffic.

Before exporting, decide whether the first question and last recorded answer form a meaningful test pair. This example uses a single policy question whose answer can be judged without earlier private context. A longer chat might depend on an order number supplied halfway through the exchange or a policy exception established by a tool. A first-message summary would lose that dependency. Leave such conversations outside this narrow importer until you have designed an input representation that preserves their necessary context.

Export one project and inspect the transcript

1. Replace my-org/my-project with your existing authorized project selector.
2. Export the documented fields into conversations.json.
3. Inspect each proposed admission using its conversation ID.
4. Approve only a complete exchange whose expected answer you can establish independently.

The CLI command reference supports an explicit organization/project selector, --period, --limit, --json and --fields. The recipe documents the exported field names and a top-level data array. This bounded command selects one project and up to 50 conversations from the last seven days:

# Expected: JSON with a data array; fewer than 50 rows is possible.
sentry agent-conversation list my-org/my-project \
  --period 7d \
  --limit 50 \
  --json \
  --fields conversationId,title,firstInput,lastOutput,flow,llmCalls,toolCalls,toolErrors,toolNames,totalTokens,totalCost,webUrl \
  > conversations.json

# Expected: the selected conversation transcript; replace the illustrative ID.
sentry agent-conversation view my-org/conv-refund-01 --json

The export should be machine-readable JSON containing conversation records. The view command should return the transcript for the selected ID. These are expected observations, not captured results. The identifier in the second command is illustrative: substitute an ID actually present in your export. A missing ID is a selection problem to resolve before mapping.

Do not equate lastOutput with a successfully finished answer. Sentry explicitly warns that it is the last assistant message recorded; an interrupted conversation can end during an intermediate step. Read the turns and tool calls before marking a record complete. A sentence such as “I will check the policy” is present text, but it is not a finished refund decision. The mapper cannot infer completion from a nonempty string.

The selected project and time range also bound the exported totals. The CLI reference says a matching span selects its conversation, while totals include all spans in the selected projects and time range. A conversation extending beyond that range deserves special attention. Inspect its transcript and widen the selection if necessary; do not quietly treat a clipped extraction as the entire lifetime of the chat.

Source screenshot requested from Sentry’s Agent Tracing article. It provides publisher context for the workflow, not evidence that this tutorial’s scripts ran.

Preserve CLI totals rather than summing raw spans. The cookbook warns that agent spans can aggregate their children, making naive sums double-count tokens or cost. The importer below validates totals but never recomputes them from trace spans. If the figures look implausible, revisit selection and instrumentation rather than repairing a number until it appears reasonable.

A limit is a bound on the selection, not proof that every eligible conversation was exported. For this exercise, inspect the returned selection and choose the three illustrative roles from it: a complete correct answer, a complete wrong answer and an interrupted answer. Do not call this a comprehensive production sample. Keep the export unchanged while checking repeatability so that a moving time window does not introduce new records between comparisons.

Define truth before admitting an answer

The illustrative policy is deliberately narrow: an unused item is refundable within 30 days of purchase. Our question concerns an unused item purchased 14 days ago. The expected answer is exactly Eligible for a refund within 30 days. This is an invented teaching policy, not a claim about any retailer.

1. Keep the observed production answer unchanged.
2. Write expected truth from the policy, independently of that answer.
3. Record transcript completion separately from answer correctness.
4. Admit a complete wrong answer too: it is a useful negative case.

Save this fully populated admission document as admissions.json. The URLs and conversation identities used later come from the real export; the keys here are illustrative conversation IDs. For real data, replace these entries only after inspecting those conversations.

{
  "conv-refund-01": {
    "caseLabel": "task:refund-policy-01",
    "complete": true,
    "expected": "Eligible for a refund within 30 days.",
    "policyVersion": "illustrative-refunds-v1"
  },
  "conv-refund-wrong-01": {
    "caseLabel": "task:refund-policy-01-wrong",
    "complete": true,
    "expected": "Eligible for a refund within 30 days.",
    "policyVersion": "illustrative-refunds-v1"
  },
  "conv-refund-incomplete-01": {
    "caseLabel": "task:refund-policy-01-incomplete",
    "complete": false,
    "expected": "Eligible for a refund within 30 days.",
    "policyVersion": "illustrative-refunds-v1"
  }
}

Completion and correctness answer different questions. A complete reply denying the refund is scoreable and wrong. An interrupted reply promising to look up the policy is not ready for this dataset, even if its eventual answer might have been correct. An absent payload supplies neither a completed answer nor enough text to score.

Input	Action	Expected result
task:refund-policy-01; unused item, purchased 14 days ago; recorded answer: Eligible for a refund within 30 days.	Inspect conv-refund-01, approve completion, preserve its answer in production_output and insert with the conversation ID.	Illustrative: policy_exact scores 1; a second identical import leaves one dataset item for this ID.
task:refund-policy-01-wrong; same question; recorded answer: Refunds are never available.	Admit the completed conversation with the same independently established expected policy answer.	Illustrative: policy_exact scores 0, demonstrating that the historical mistake remains visible.
task:refund-policy-01-incomplete; same question; recorded answer: I will check the policy.	Set complete to false after inspecting the interrupted transcript.	Illustrative: importer reports not-approved-complete and performs no insert for this conversation.

This small admission file is an editorial control implemented by our example code, not a Sentry export field or built-in completion detector. Keeping it explicit avoids inventing a completion status the evidence does not document. For complementary synthetic coverage, the existing guide on simulated companies for agent testing covers a different source of fixtures.

Treat the policy version as part of the interpretation of the score. If the business later changes its refund window, retain the historical policy when asking whether the recorded answer was correct at that time. A separate experiment can ask whether the same question meets today’s policy. Editing expected truth in place without recording that change would make an apparent improvement or regression ambiguous: the evaluator and the standard could both have changed.

Illustrative decision matrix: completeness controls admission, while independent policy truth controls the score. Based on the Sentry recipe and Braintrust evaluation contract, read November 9, 2026; no measured results.

Map approved conversations without duplicating them

1. Save the following complete script as import-conversations.ts.
2. Set PROJECT to your existing Braintrust project name.
3. Keep the dataset name disposable and use it only for this exercise.
4. Run the script with your existing TypeScript runner, then inspect that dataset.

Sentry’s cookbook documents dataset.insert({ id }) as an upsert keyed by conversation ID. Braintrust’s TypeScript API reference documents initDataset, insert, metadata, expected values and flush. The additional validation and admission decisions below are original illustrative application logic.

import { readFileSync } from 'node:fs';
import { initDataset } from 'braintrust';

const PROJECT = 'my-project';
const DATASET = 'tutorial-refund-recordings-v1';
type Admission = {
  caseLabel: string; complete: boolean;
  expected: string; policyVersion: string;
};
type Conversation = {
  conversationId: string; title: string | null;
  firstInput: string | null; lastOutput: string | null;
  flow: string[]; llmCalls: number; toolCalls: number;
  toolErrors: number; toolNames: string[];
  totalTokens: number; totalCost: number; webUrl: string;
};
const nonempty = (v: unknown): v is string =>
  typeof v === 'string' && v.trim().length > 0;
const count = (v: unknown): v is number =>
  typeof v === 'number' && Number.isSafeInteger(v) && v >= 0;
const strings = (v: unknown): v is string[] =>
  Array.isArray(v) && v.every(x => typeof x === 'string');

async function main() {
  const envelope = JSON.parse(readFileSync('conversations.json', 'utf8'));
  if (!Array.isArray(envelope.data)) throw new Error('Missing data array');
  const admissions = JSON.parse(
    readFileSync('admissions.json', 'utf8')
  ) as Record<string, Admission>;
  const seen = new Set<string>();
  const accepted: { c: Conversation; a: Admission }[] = [];
  const rejected: { id: string; reason: string }[] = [];

  for (const raw of envelope.data) {
    if (!raw || typeof raw !== 'object' ||
        !nonempty(raw.conversationId)) throw new Error('Invalid identity');
    const id = raw.conversationId;
    if (seen.has(id)) throw new Error(`Duplicate export identity: ${id}`);
    seen.add(id);
    const reject = (reason: string) => rejected.push({ id, reason });
    if (id.startsWith('test-')) { reject('test-traffic'); continue; }
    if (!nonempty(raw.firstInput) || !nonempty(raw.lastOutput)) {
      reject('missing-payload'); continue;
    }
    const a = Object.hasOwn(admissions, id) ? admissions[id] : undefined;
    if (!a || a.complete !== true) {
      reject('not-approved-complete'); continue;
    }
    if (!nonempty(a.caseLabel) || !nonempty(a.expected) ||
        !nonempty(a.policyVersion)) throw new Error(`Invalid admission: ${id}`);
    if (!count(raw.llmCalls) || !count(raw.toolCalls) ||
        !count(raw.toolErrors) || !count(raw.totalTokens) ||
        raw.toolErrors > raw.toolCalls ||
        typeof raw.totalCost !== 'number' ||
        !Number.isFinite(raw.totalCost) || raw.totalCost < 0 ||
        !strings(raw.flow) || !strings(raw.toolNames) ||
        !(raw.title === null || typeof raw.title === 'string') ||
        !nonempty(raw.webUrl)) {
      reject('invalid-metadata'); continue;
    }
    const url = new URL(raw.webUrl);
    if (url.protocol !== 'https:') throw new Error(`Invalid source URL: ${id}`);
    accepted.push({ c: raw as Conversation, a });
  }

  accepted.sort((x, y) => y.c.totalTokens - x.c.totalTokens);
  console.log(JSON.stringify({
    accepted: accepted.map(x => x.c.conversationId), rejected
  }, null, 2));
  if (!accepted.length) throw new Error('No admitted conversations');

  const dataset = initDataset({ project: PROJECT, dataset: DATASET });
  for (const { c, a } of accepted) {
    dataset.insert({
      id: c.conversationId, input: c.firstInput!, expected: a.expected,
      metadata: {
        case_label: a.caseLabel, policy_version: a.policyVersion,
        production_output: c.lastOutput!, title: c.title, agents: c.flow,
        llm_calls: c.llmCalls, tool_calls: c.toolCalls,
        tool_errors: c.toolErrors, tools: c.toolNames,
        total_tokens: c.totalTokens, cost_usd: c.totalCost,
        sentry_url: c.webUrl
      }
    });
  }
  await dataset.flush();
  console.log(`Flushed ${accepted.length} admitted conversations`);
}
main().catch(error => { console.error(error); process.exitCode = 1; });

The script validates the whole batch before opening the dataset. Duplicate identities fail the import instead of allowing an arbitrary last row to win. Missing payloads, unapproved conversations and invalid metadata are reported with distinct reasons. Sorting by tokens follows the cookbook’s prioritization recommendation; it changes processing order, not the record identity or expected truth.

Run tsx import-conversations.ts if tsx is your configured runner. Read the admission report before treating the flush message as success. That message states how many inserts were flushed; it does not prove a score, a policy judgment or the dataset’s total cardinality. Open the disposable dataset and compare its IDs and payloads against the export and admissions file.

Repeat the same import without changing either file. The expected logical result is one item per conversation ID, because the documented inserts upsert. This does not promise that dataset version history remains unchanged. Also, the script does not delete an existing item when a later admission file rejects it. For that correction, stop using the contaminated disposable dataset and import the corrected selection into a fresh disposable dataset name.

For the populated refund example, inspect both admitted items by identity. Each should retain its own recorded answer while sharing the same policy sentence in the expected field. The wrong answer must still say that refunds are never available. If it now contains the policy sentence instead, the mapping has overwritten observation with truth. Correct that mapping before scoring; a perfect score on copied expected answers proves nothing about the original agent.

The recorded answer stays in production_output; expected truth comes from the admission record. The passthrough task returns the recording, not a new model response. Sources: Sentry cookbook and Braintrust API reference, read November 9, 2026.

Score the recording and challenge the scorer

1. Save the script below as score-recordings.ts.
2. Match its project and dataset names to the importer.
3. Run it using your existing TypeScript runner.
4. Inspect individual scores before interpreting an aggregate.

Braintrust documents a task receiving case input and hooks, including metadata, and scorer functions receiving output and expected. Sentry’s cookbook uses that contract for a passthrough returning production_output. This example adds a strict policy scorer and checks its behavior before starting the evaluation.

import { Eval, initDataset } from 'braintrust';

const PROJECT = 'my-project';
const DATASET = 'tutorial-refund-recordings-v1';
const truth = 'Eligible for a refund within 30 days.';
const exact = (output: unknown, expected: unknown) =>
  typeof output === 'string' && typeof expected === 'string' &&
  expected.length > 0 && output === expected ? 1 : 0;

const controls: [unknown, unknown, number][] = [
  [truth, truth, 1],
  ['Refunds are never available.', truth, 0],
  ['Not eligible. Eligible for a refund within 30 days.', truth, 0],
  ['', truth, 0],
  [undefined, undefined, 0]
];
for (const [output, expected, wanted] of controls) {
  if (exact(output, expected) !== wanted) {
    throw new Error('Policy scorer self-check failed');
  }
}

async function main() {
  await Eval(PROJECT, {
    data: initDataset({ project: PROJECT, dataset: DATASET }),
    task: async (_input, hooks) => {
      const output = hooks.metadata?.production_output;
      if (typeof output !== 'string' || output.trim().length === 0) {
        throw new Error('Missing recorded answer');
      }
      return output;
    },
    scores: [({ output, expected }) => ({
      name: 'policy_exact', score: exact(output, expected)
    })]
  });
}
main().catch(error => { console.error(error); process.exitCode = 1; });

Run tsx score-recordings.ts with the configured runner. The illustrative correct answer should receive 1 and the completed wrong answer 0. The incomplete conversation should have no dataset row from this import. If only those two complete cases are present, their arithmetic mean is 0.5; verify the individual rows before accepting that summary. Additional existing items would change the denominator.

Exact equality is intentionally strict. It catches a wrong answer that appends the correct sentence, which a substring scorer could falsely pass. It also rejects a semantically correct paraphrase. That is appropriate for this controlled example’s fixed output contract, but it is not a general measure of helpfulness. A broader policy needs structured expected decisions or carefully reviewed scoring rules. Do not silently relax equality just to raise a score.

The scorer controls include missing values because JavaScript equality alone would make undefined === undefined true. Our type and nonempty checks prevent that false pass. The negative control is as important as the positive one: a scorer that always returns 1 would look perfect on a dataset containing only correct answers.

Keep this historical experiment separate from a future live-agent comparison. Its task ignores the question and returns recorded metadata by design. Replacing that function with a model call changes what is being measured. The existing evaluation CI gates guide covers enforcement after you have a trustworthy dataset; it does not substitute for establishing truth here.

Read an individual failed row in this order: the independent expected sentence, the returned historical output, then the source transcript. That order helps distinguish a real wrong answer from an admission mistake or a formatting mismatch. If the exported answer is a correct paraphrase, a zero is consistent with this strict scorer. Record that limitation when reporting the experiment instead of presenting the mean as a general accuracy measurement.

Diagnose failures and recover cleanly

Observation	Interpretation	Recovery
No conversations appear	Tracing, conversation IDs, project selection or date range may be wrong.	Check the existing route and Sentry Conversations view before changing the importer.
missing-payload rejection	Input or output text was not available for this record.	Keep it excluded; inspect collection and scrubbing settings without reconstructing missing text.
not-approved-complete rejection	No explicit completed-transcript approval exists.	Inspect the transcript and update the admission only when completion is supported.
invalid-metadata rejection	Counts, arrays, cost or source fields violate this example’s contract.	Compare the export with documented fields and retain CLI totals rather than summing spans.
Wrong control receives 1	Scoring logic or expected truth is defective.	Stop interpreting the experiment and repair the scorer controls first.
Reimport produces more logical items	The dataset, project or identity mapping differs.	Compare actual IDs; isolate the corrected import in a fresh disposable dataset.

An empty export and an empty accepted set are different failures. The former directs attention toward recording and selection; the latter directs attention toward payloads and admission decisions. Preserve that distinction in debugging. Loosening admission until something uploads hides the reason the dataset was empty.

A network or SDK failure during flushing may leave some rows written. After correcting the problem, repeating the same IDs follows the documented upsert path. Do not generate replacement identities to get around the failure. If you cannot establish which records belong in the tutorial dataset, use a new disposable name and retain the previous dataset only for investigation under your normal retention rules.

The mapper handles one selected project. Its key is the conversation ID, following the cookbook. Before expanding into several projects, establish whether your application reuses IDs across them and define a deliberate composite identity. This guide does not claim that a conversation ID is globally unique across every project or organization. Keeping the scope narrow avoids that unsupported assumption.

Restore temporary instrumentation settings after the exercise and stop exporting tutorial data when the lesson is complete. Disabling future collection does not erase data already stored in either platform. Handle existing exports and disposable datasets through your established retention and deletion procedures; no undocumented deletion command is needed here.

Close the exercise by comparing the exported selection, the admission decisions and the dataset contents together. For this illustrative case, two completed conversations belong in the dataset and the interrupted conversation remains excluded. Reimport should preserve those two identities and their separate answers. The positive control should pass and the deliberate mistake should fail. These checks establish the behavior of this conversion and scorer; they do not establish how a new prompt or model will perform.

The practical result is a dataset whose records preserve what was asked, what was actually answered, why each case was admitted and which independent policy supplies truth. That separation makes a historical failure reproducible even when the model’s next answer changes. Fleet replay explains why retaining the original evidence matters, while AgentOps places this evaluation step within ongoing operation.

Sources

1. Sentry: Five things you can do with Sentry’s Agent Tracing — October 28, 2026 publisher-dated background.
2. Sentry CLI: agent-conversation — selectors, transcript viewing and output options.
3. Sentry: Next.js Agent Tracing — conversation IDs and collection controls.
4. Sentry: Conversations — grouping and transcript inspection.
5. Sentry: Conversation-to-evaluation cookbook — export schema, upserts and recorded-output evaluation.
6. Braintrust: TypeScript API reference — dataset and evaluation APIs.
7. Braintrust: Install and instrument — SDK installation and environment setup.