feat(evaluations): preserve the tool trajectory for judges

[donei003]

Now carries two commits: the trajectory capture, and #98 (merged in here) which made every judge path share one message_history.

Problem

A judge can only grade what it is shown. Handler packages record tool traffic onto OpenTelemetry spans and return only {output, usage}, so by the time a judge ran, the calls made on the way to that output were gone. "Did the agent call the right tool, in the right order, with the right arguments?" was an unaskable question of an SDK that had just run the agent that answered it.

Underneath that, the three judge paths had already drifted. Each built message_history with its own inline join:

Path	message_history was
Offline evals	row input + output + format block
Online inline (run_judges)	user input + output + format block
Online deferred (run_judge)	output + format block — no input at all

A deferred judge was grading a response with no request beside it. That is a pre-existing bug, fixed here as a consequence of unifying.

Approach

Capture by wrapping the tool map, which both paths already own — so every handler package is covered without changing any of them, and a custom caller-supplied handler is covered too. A handler still resolves a tool by the key the model named and calls it.

• Offline: once per row, in evaluations/runner.py.
• Online: once per invocation, in tracking.execute_and_track / execute_and_stream, which now return the rendered trajectory alongside response and track_data.

judge_scoring.build_message_history is the only place a history is built — in the module that already owns the {score, reasoning} contract, for exactly the same reason. It orders the conversation the way it happened: input → trajectory → output → format block, skipping empty parts. All three paths call it.

What a judge now sees:

Where is order A1?

Tools available: lookup_order, issue_refund
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"id":"A1"}
   result: order A1 shipped 2026-08-02
2. issue_refund
   arguments: {"id":"A1","amount":19.99}
   error: refund window closed

That order shipped on Aug 2 and is outside the refund window.

Your response MUST be in valid JSON format with the following structure:
...

Verified live against a real LaunchDarkly judge: the judge's reasoning cited the call, its arguments, and its result — all three only visible via the trajectory.

Properties pinned by tests

• The recorder observes; it never intervenes. A wrapped tool returns exactly what the original returned and raises exactly what the original raised. Calls past the 50-call cap still execute and are only counted — truncation drops the record, never the work, because a harness that changed the agent's behaviour would no longer be evaluating the agent.
• A recorder belongs to one invocation. Rows generate concurrently against one shared tool map, so a shared recorder would splice one row's calls into another's and hand the judge a conversation that never happened. Tested with two rows whose calls interleave.
• Order is call-start order, not completion order. A judge asked whether the agent searched before it refunded is reading a sequence.
• Inline and deferred produce byte-identical history for the same row. This is the property that makes a rubric portable between a production sample and a dataset replay.
• A tool result stays literal in the judge prompt. A tool result is a new injection surface alongside generated output, closed by the existing rule: the judge config is passed unrendered and the handler makes exactly one template pass. Tested with a tool returning the literal text {{expected_output}}.
• Back-compatible. A run with no observable tools adds no block, so a judge authored before this reads exactly the message_history it read before.
• Nothing is added to any event payload. The trajectory reaches LaunchDarkly only inside the prompt a judge was shown, never as a wire field the backend has not specified.

Design notes

• No standalone trajectory variable. An earlier revision exposed {{tool_trajectory}} too; it overlapped message_history and bought nothing, while inviting a rubric to interpolate both and pay for the trajectory twice. Confirmed live — a real judge config's own scaffolding already interpolates {{message_history}}. A test pins its absence. Since message_history is what judges cloned from the AI Library's default templates read, an existing judge becomes a trajectory judge by editing its rubric text alone.
• Native provider tools are skipped in both paths, and left out of "Tools available". Online, wrap_tool_handlers substitutes a callable tracking stub, so a native call is locally observable — but the stub returns nothing, so recording it would show a judge a call with an empty result while the provider's real result stayed invisible. Recording is therefore composed inside that wrapper, on the original map. Tests assert both the exclusion and that $ld:ai:tool_call still fires underneath.
• JudgeTask gains user_input and trajectory as plain strings — every field on it has to survive pickling to a worker thread; a test pins that.
• Graph-level judges get no trajectory, deliberately: graph_judge grades an answer produced across several nodes, and splicing their trajectories would describe a conversation that never happened. Per-node judges get their own node's.
• Bounds: 50 recorded calls per invocation, 2000 characters per rendered argument bag or result. A trajectory goes into a judge prompt, so an agent looping over a large result set would otherwise spend the judge's context window — and budget — on a tail no judge reads.
• trajectory.py sits at the package root, not under evaluations/, since it is no longer evaluations-specific.

Deliberately out of scope

Scorers cannot see the trajectory. Scorer.fn(row, output) is the contract, and the trajectory is not dataset-owned so it does not belong on DatasetRow. A deterministic check like "called lookup_order exactly once" is a natural follow-up but needs a contract change, not a quiet signature widening.

Validation

• uv run pytest -q — 1282 passed, 11 skipped
• uv run mypy packages/client/src/launchdarkly_ai_server — clean; ruff check / format --check — clean
• Exercised end-to-end in ai-sdk-evaluations-example (launchdarkly-labs/ai-sdk-evaluations-example#5), including a live run against a real judge

Language-agnostic spec: launchdarkly/ai-sdks-monorepo#13 — being restructured to describe the shared flow rather than the offline phase alone, now that this shape is settled. Submodule pointer: launchdarkly/ai-sdks-monorepo#14.

🤖 Generated with Claude Code

---

Note

Overview
Judges can now grade how an agent answered, not only the final text. The SDK records tool calls by wrapping the caller’s tool map (online in execute_and_track / execute_and_stream, offline per dataset row), renders them into a bounded transcript, and splices that into {{message_history}} between user input and model output—no new judge template variable and no new event payload fields.

judge_scoring.build_message_history becomes the single builder for inline run_judges, deferred run_judge (JudgeTask now carries user_input and a picklable trajectory string), and offline eval scoring—fixing the deferred path that previously omitted the user request. Graph per-node judges receive the streamed/captured trajectory; graph-level judges still do not.

Recording rules exclude synthetic __handoff_* tools and provider NativeTool calls from the transcript, cap volume (50 calls, 2k chars per value), and keep tool behavior and $ld:ai:tool_call tracking unchanged. Docs/READMEs describe trajectory rubrics; OTel graph spans rename from ld.ai.graph to launchdarkly.graph (attribute launchdarkly.graph.key). Package versions bump to 0.2.3 / 0.1.7.

^{Reviewed by Cursor Bugbot for commit 245a0a5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 2 new potential issues.

1 security issue and 1 flag not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Prefixed user tools vanish from judges

When a config defines a callable named __handoff_lookup, TrajectoryRecorder excludes its calls as synthetic. Judges see no use of that real tool, even though the model invoked it.

Learn more

The prefix test treats a name as a graph routing tool without checking whether graph routing actually created it. A regular AI Config may expose a callable whose key starts with __handoff_; execute_and_track passes its config's exposed keys to the recorder, but the prefix still excludes that callable. Its judge history therefore cannot describe the tool's actual calls.

Example: A normal config exposes __handoff_lookup and its handler invokes that tool to answer a user. The call runs, but the trajectory omits it entirely.

Recommended fix: Identify synthetic handoff keys in the graph routing caller and exclude only those generated keys. Do not infer synthetic origin solely from the tool name in TrajectoryRecorder.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Devin Review found 1 new potential issue.

1 security issue and 1 flag not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Native tool use appears absent to judges

When a config declares a NativeTool, exposed_tool_keys lists it although TrajectoryRecorder.wrap cannot record its calls. A judge sees “No tool calls were made” even when the agent used that tool.

Learn more

Native tools execute inside the provider rather than through a local function. TrajectoryRecorder.wrap skips their NativeTool sentinels, but the available-tool list is now built solely from config keys. A configured native tool is therefore listed without any way to record its use, so a judge can misread a successful native call as no tool call.

Example: A config declares web_search, and tool_handlers={"web_search": NativeTool("WebSearch")}. The provider uses WebSearch; the recorded trajectory says Tools available: web_search followed by No tool calls were made while producing the response.

Recommended fix: Exclude keys backed by NativeTool from the available-tool list at both execute_and_track and execute_and_stream. Keep config-declared keys with no implementation in that list, and preserve the existing handoff and registry-only filters.

---

Was this helpful? React with 👍 or 👎 to provide feedback.