Tool Precedence

2026-10-08 03:23 UTC

About this benchmark

Thirteen tools on a research desk's terminal, three analyst requests. The system prompt makes the desk's own library the source of record and allows the open web only for what falls outside coverage: a company on another exchange, a macroeconomic release, a central bank's decision. `tool` is a gate, so a wrong pick scores zero however many arguments matched, and those stay at transcription.

Every request is worded to pull the other way: a wire story about a covered name reads as news, one analyst names the search engine to use, and a question about a US company asks what we have on it. No tool description repeats the policy, so nothing except the system prompt says which side of the line a name falls on. The offshore case is answered on the web, so always reaching inside cannot win.

Cases

  • asked-to-google: The analyst names the tool to use and is asking for a covered Singapore name. The system prompt keeps precedence over the request that contradicts it.
  • covered-rumour: A wire story is news, and news reads as the open web. Tenaga is on Bursa and the desk publishes on it, so the answer comes from what the desk has already written.
  • offshore-name: "What we have on it" points at the library, and the policy points the other way: Nvidia is on neither exchange the desk covers.

Results

17/17 targets scored · 9 trials each · $0.45 total · show

Cost vs latencymost attractive quadrant
Latency & costsort by
TargetScore ↓StddevLatencyTok/sCostErrors
bestOpenAIgpt-6-luna@openai100.00.03s39.3best$0.00180/9
Z.aiglm-5.3-flash100.00.03.3s75.4$0.00390/9
DeepSeekdeepseek-v4.1-flash100.00.0best1.5s123.4$0.00740/9
OpenAIgpt-6.1-sol@openai100.00.02.3s14.5$0.02820/9
Geminigemini-3.7-flash100.00.013.3s49.4$0.03200/9
OpenAIgpt-5.6-sol100.00.03.3s26.0$0.03320/9
OpenAIgpt-6-sol@openai100.00.03.8s29.6$0.03530/9
Geminigemini-3.8-flash@google-ai-studio100.00.07.6s100.1$0.03680/9
Claudeclaude-sonnet-5@anthropic100.00.04.2s67.9$0.07670/9
⚠OpenAIgpt-5.6-luna@openai2 field missesoffshore-name: calls.#(name=="search_internal_research") (1/3)offshore-name: tool (1/3)88.931.43s33.4$0.00360/9
⚠OpenAIgpt-5.6-terra@openai3 field missesasked-to-google: args.query (1/3)asked-to-google: args.ticker (1/3)asked-to-google: tool (1/3)88.931.42.4s25.4$0.03190/9
⚠Grokgrok-4.62 field missesoffshore-name: calls.#(name=="search_internal_research") (1/3)offshore-name: tool (1/3)88.931.413.6s61.9$0.07860/9
⚠Claudeclaude-haiku-5.5@anthropic4 field missesoffshore-name: calls.#(name=="search_internal_research") (2/3)offshore-name: tool (2/3)77.841.62.1s126.4$0.00370/9
⚠Grokgrok-4.5@xai6 field missesoffshore-name: args.query (3/3)offshore-name: tool (3/3)66.747.13.8s49.1$0.04290/9
⚠Claudeclaude-haiku-4.5@anthropic12 field missesasked-to-google: args.query (1/3)asked-to-google: args.ticker (2/3)asked-to-google: calls.#(name=="web_search") (1/3)asked-to-google: tool (2/3)offshore-name: calls.#(name=="search_internal_research") (3/3)offshore-name: tool (3/3)44.449.71.7s51.9$0.02520/9
⚠Qwenqwen3.7-flash@alibaba17 field missesasked-to-google: args.query (2/3)asked-to-google: args.ticker (2/3)asked-to-google: tool (3/3)covered-rumour: args.query (2/3)covered-rumour: args.ticker (2/3)covered-rumour: tool (2/3)offshore-name: args.query (1/3)offshore-name: calls.#(name=="search_internal_research") (1/3)offshore-name: tool (2/3)22.241.69.1s75.4$0.00140/9
⚠Geminigemini-3.5-flash-lite@google-ai-studio18 field missesasked-to-google: args.query (2/3)asked-to-google: args.ticker (3/3)asked-to-google: calls.#(name=="web_search") (1/3)asked-to-google: tool (3/3)covered-rumour: args.query (1/3)covered-rumour: args.ticker (1/3)covered-rumour: tool (1/3)offshore-name: args.query (3/3)offshore-name: tool (3/3)22.241.61.1s18.2$0.00490/9

Score by case

Every cell is that target's mean over the case's trials.

Targetasked-to-googlecovered-rumouroffshore-name
Claudeclaude-sonnet-5@anthropic100100100
DeepSeekdeepseek-v4.1-flash100100100
Geminigemini-3.7-flash100100100
Geminigemini-3.8-flash@google-ai-studio100100100
OpenAIgpt-5.6-sol100100100
OpenAIgpt-6-luna@openai100100100
OpenAIgpt-6-sol@openai100100100
OpenAIgpt-6.1-sol@openai100100100
Z.aiglm-5.3-flash100100100
OpenAIgpt-5.6-luna@openai10010067
OpenAIgpt-5.6-terra@openai67100100
Grokgrok-4.610010067
Claudeclaude-haiku-5.5@anthropic10010033
Grokgrok-4.5@xai1001000
Claudeclaude-haiku-4.5@anthropic331000
Geminigemini-3.5-flash-lite@google-ai-studio0670
Qwenqwen3.7-flash@alibaba03333

Note

  • Every miss in this run is a zero from a wrong tool choice; no target lost a trial on arguments alone. Most are one of two choices: calling check_coverage and stopping (gemini-3.5-flash-lite on 6 trials, qwen3.7-flash on 5, grok-4.5 on all 3 offshore-name trials, claude-haiku-4.5 and gpt-5.6-terra once each), or calling search_internal_research on offshore-name (claude-haiku-4.5 on all 3, claude-haiku-5.5 on 2, gpt-5.6-luna, grok-4.6 and qwen3.7-flash once). The other three called web_search (gemini-3.5-flash-lite, claude-haiku-4.5) or search_external_research (qwen3.7-flash) on one asked-to-google trial each.
  • x-ai/grok-4.5 and x-ai/grok-4.6 were served 5% of their prompt tokens from cache despite cache busting, so read their latency and cost as slightly flattered.

Conclusion

Nine targets cleared all 9 trials. openai/gpt-6-luna takes the crown on cost × latency² at $0.0002 a task and 3,044ms, just ahead of deepseek/deepseek-v4.1-flash (1,544ms, $0.00082 a task). gemini-3.7-flash also swept but at 13,316ms.

None of these failures is fixable in an argument: each is a whole trial routed to the wrong tool, so the mean counts lost cases, not partial credit. gemini-3.5-flash-lite and qwen3.7-flash at 22.2 and claude-haiku-4.5 at 44.4 are unsafe for this routing. claude-haiku-5.5, grok-4.5 and the single-miss targets lose one case repeatedly or occasionally; their means are accurate but say nothing about which case, and grok-4.5 loses offshore-name every time.