Skip to content

agent_simulation: add the progression judge's findings to JobMetrics - #2005

Open
u9g wants to merge 1 commit into
mainfrom
feat/simulation-progression-findings
Open

u9g wants to merge 1 commit into
mainfrom
feat/simulation-progression-findings

Conversation

@u9g

@u9g u9g commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

SimulationRun.JobMetrics says which progression checks the judge flagged (unnecessary_tool_calls, information_loss, redundant_statements, poor_question_quality), but not what the agent actually did wrong. The judge already produces that for every flagged check: a 1–3 rating and 1–2 sentences of evidence citing transcript items. Today it only goes to the simulator's logs.

This adds JobMetrics.progression_findings (field 25). There's one ProgressionFinding per flagged check:

  • check: which of the four it is.
  • severity: MINOR (judge rating 2) or CLEAR (rating 1).
  • evidence: the judge's sentence, citing items as [item_id].
  • citations: the cited items in first-cited order, each with its item_id, kind (agent, caller, tool call or tool output) and text. The dashboard can show what was said without loading the transcript, and link each item into it by item_id.

Additive only; no reader of the existing fields changes.

Consumers:

  • livekit/agents-private: the simulate worker fills it, and agent-service decodes it from the stored metrics.
  • livekit/web: the run page's Conversation progression breakdown shows each finding with its cited lines (livekit/web#7351).

@changeset-bot

changeset-bot Bot commented Nov 20, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 842fb73

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
github.com/livekit/protocol Patch
@livekit/protocol Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant