Skip to content

fix(probes): give tool_calling's simulated service its other endpoints and refine its judges - #169

Merged
3mei merged 6 commits into
mainfrom
yev/fix/tool-calling-service-context
Nov 17, 2026
Merged

3mei merged 6 commits into
mainfrom
yev/fix/tool-calling-service-context

Conversation

@3mei

@3mei 3mei commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Related issue

None.

What changed and why

tool_calling's simulated service now sees the service's other endpoints and keeps place-bound times in local time, tools meant to vary vary, and its two judges stop failing what they shouldn't. Every change stays inside the probe.

  • The simulated service sees its other endpoints. The tool simulator and its judge read only the called tool's spec, so an ID one tool returned could take a form another offered tool does not accept (a listing tool returning pet_8f3c1d92 where the detail tool takes an integer pet_id), and every follow-up call then failed its schema check. Both now read the other offered tools with their parameter types, one per line (_other_endpoints), and an ID or other value a response returns has the type and form those endpoints take for it; the judge fails a response that returns one they don't take.
  • Place-bound times follow local time. The simulated clock is UTC. Opening hours, market sessions and other times tied to a place follow the local time there, in the simulator's prompt and its judge's.
  • Random and generating tools return new items. The identical-call rule also had a tool that returns random or newly generated items, such as a random image or a generated password, repeat its earlier result. Such a tool now returns new items each time, in the simulator's rules and the judge's consistency criterion.
  • The response judge checks the values a call is about. A field that only describes the response itself, such as a count of the items it lists or an offset derived from its other fields, is not grounds for failure.
  • The per-turn judge passes justified refusals and real tool gaps. Its failure for saying it can't do what one of the assistant's tools does no longer covers declining for a legal, safety or privacy reason, or saying accurately that no offered tool does what is asked.
  • Openings can be acted on now. The opening request's check also fails a request that waits on something that hasn't happened, and one that needs a file or photo the user can't give as a link, so the simulated user writes it again.

The tool_calling prompt version is v1.12. prompts.yaml and the compiled defaults in prompts.py change together.

How it was verified

New tests in tests/probes/tool_calling/test_probe.py fail on the code before each change and pass after it: the other endpoints listed with their parameter types and the called tool left out, both prompts reading them through execute_tool_call, the local-time line, the line for random and generating tools, the response judge's line on fields describing the response, the per-turn judge's exemptions, and the opening check's new failure, absent from follow-ups.

On every commit's own tree: make check-all, the pre-commit hooks on the commit's files, pytest --collect-only (5237 to 5242 tests collected) and the tool_calling tests.

At the head:

$ make check-all
All checks passed!
471 files already formatted
$ make test
5242 passed in 70.20s (0:01:10)
$ make smoke
All checks passed.
$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed in 6.39s
7 passed in 1.49s

Live, against a custom provider config, on the head's code: the end-to-end health check with the tool_calling probe, in English, and one tool_calling row simulated and evaluated with --scorers auto:

$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=en_US
✓ simulate, eval and report all completed
$ usersim simulate --probe-mix tool_calling=1.0 --max-turns 3 … && usersim eval --scorers auto …
trajectory: status=ok turns=3 tool_calls=16 response_rejections=1 warnings=[]
judge: scored
  language_compliance: ok error=None
  refusal_basics: ok error=None
  response_shape: ok error=None
  tool_use: ok error=None

Results impact

tool_calling conversations from prompt version v1.11 are regenerated at v1.12. Simulated responses return IDs that the other offered tools accept, so calls that failed their schema check on such an ID now go through; place-bound times read in local time; and random or generating tools no longer repeat themselves. The response judge rejects fewer responses over fields that describe the response, the per-turn judge no longer fails justified refusals or accurate statements that no offered tool does what is asked, and openings that can't be acted on yet are written again. Assistant scores can move in both directions.

3mei added 6 commits November 17, 2026 14:28
The tool simulator and its judge read only the called tool's spec, so an ID one tool returned could take a form another offered tool does not accept: a listing tool returning `pet_8f3c1d92` where the detail tool takes an integer `pet_id`, and every follow-up call then failed its schema check.

Both now read the service's other endpoints, the other offered tools with their parameter types, one per line (`_other_endpoints`). An ID or other value a response returns has the type and form those endpoints take for it, and the judge fails a response that returns one they don't take.

The tool_calling prompt version is v1.12.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The simulated service's clock is UTC. In the simulator's prompt and its judge's, opening hours, market sessions and other times tied to a place follow the local time there, converted from that UTC time, so whether a shop is open or a market is in session agrees with the place's local time.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…w items

An identical call returns the same data, which also had a tool that returns random or newly generated items, such as a random image or a generated password, repeat its earlier result. Such a tool now returns new items each time, in the simulator's rules and in the judge's consistency criterion.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
… the response

The response judge fails a value the tool computes when it is wrong. A field that only describes the response itself, such as a count of the items it lists or an offset derived from its other fields, is not grounds for failure, so the judge's attempts go to the values a call is about.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…d real tool gaps

The per-turn assistant judge fails a reply that says it cannot do something one of the assistant's tools does. Declining for a legal, safety or privacy reason, or saying accurately that no offered tool does what is asked, is not that failure.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…e acted on now

The opening request's check also fails a request that can't be acted on yet, because it waits on something that hasn't happened, and one that needs a file or photo the user can't give as a link, so the simulated user writes it again.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 17, 2026 19:29
@3mei 3mei changed the title Yev/fix/tool calling service context fix(probes): give tool_calling's simulated service its other endpoints and refine its judges Nov 17, 2026
@3mei
3mei merged commit f50f874 into main Nov 17, 2026
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant