Tool Call
About this benchmark
Each case is one customer message against eight tools offered as four near-duplicate pairs: balance now or closing balance, whole window or merchant search, transfer now or one queued for a date, ticket or card block. Either tool in a pair returns something plausible, so a wrong pick means a clause was read too fast. The trial ends at the model's first move, and one case is a legitimate transfer, so a model that refuses every money movement fails rather than scores.
`tool` is a gate: pick the wrong tool and the trial scores zero, however many arguments matched. Above the gate the extraction-style field weights apply, so the right tool with a wrong date still scores below a clean call, and the per-field detail reports every field either way.
Cases
amount-in-words: The control for `no-transfer`: refusing every money movement fails here. The amount is written in words and the two account numbers diverge only after the fourth digit.decoy-number: The card's last four digits come first, but `account_no` wants the account. "Last month" on 31 July is June, and no merchant is named, which keeps `search_transactions` out.future-dated: A date turns the transfer into a scheduled one: the immediate call would move the rent five days early, so it is asserted absent as well as wrong. "The 5th of next month" has to become a date against a today of 31 July.no-transfer: The customer asks for a transfer, but the debit is disputed and reversing it is not the assistant's call: the answer is a ticket, with the transfer asserted absent from the whole call list.statement-month: A closing balance for an ended month is not the balance today, so `get_account_balance` answers a different question with a real number. The system prompt says YYYY-MM-DD and this one parameter says YYYY-MM: the general instruction beating the specific one.stolen-card: A stolen wallet reads like something for a human, but the card can be stopped now and the loss runs until it is. Also the one place the four digits belong in `card_last4` rather than `account_no`, the reverse of `decoy-number`.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| gpt-6-luna@openai | 100.0 | 0.0 | 2.3s | 25.5 | $0.0020 | 0/18 |
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 6s | 77.6 | $0.0020 | 0/18 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 2.3s | 28.3 | $0.0043 | 0/18 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.2s | 47.0 | $0.0096 | 0/18 |
| deepseek-v4.1-flash | 100.0 | 0.0 | 982ms | 144.7 | $0.0113 | 0/18 |
| gpt-6.1-sol@openai | 100.0 | 0.0 | 2.4s | 17.7 | $0.0363 | 0/18 |
| gpt-6-sol@openai | 100.0 | 0.0 | 2.4s | 25.4 | $0.0397 | 0/18 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 2s | 29.6 | $0.0412 | 0/18 |
| gemini-3.8-flash@google-ai-studio | 100.0 | 0.0 | 4.8s | 94.0 | $0.0480 | 0/18 |
| grok-4.5@xai | 100.0 | 0.0 | 2.7s | 40.5 | $0.0635 | 0/18 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 3.2s | 50.6 | $0.1069 | 0/18 |
| ⚠glm-5.3-flash2 field missesdecoy-number: args.from_date (1/3)decoy-number: args.to_date (1/3) | 98.0 | 8.3 | 1.9s | 48.8 | $0.0045 | 0/18 |
| ⚠claude-haiku-5.5@anthropic2 field missesdecoy-number: args.from_date (1/3)decoy-number: args.to_date (1/3) | 98.0 | 8.3 | 1.8s | 104.7 | $0.0054 | 0/18 |
| ⚠claude-haiku-4.5@anthropic9 field missesdecoy-number: args.from_date (2/3)decoy-number: args.to_date (2/3)decoy-number: tool (2/3)future-dated: args.on_date (3/3) | 85.8 | 31.1 | 1.7s | 69.8 | $0.0449 | 0/18 |
Score by case
| Target | amount-in-words | decoy-number | future-dated | no-transfer | statement-month | stolen-card |
|---|---|---|---|---|---|---|
| claude-sonnet-5@anthropic | 100 | 100 | 100 | 100 | 100 | 100 |
| deepseek-v4.1-flash | 100 | 100 | 100 | 100 | 100 | 100 |
| gemini-3.5-flash-lite@google-ai-studio | 100 | 100 | 100 | 100 | 100 | 100 |
| gemini-3.8-flash@google-ai-studio | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-luna@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-terra@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6-luna@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6-sol@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6.1-sol@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| qwen3.7-flash@alibaba | 100 | 100 | 100 | 100 | 100 | 100 |
| grok-4.5@xai | 100 | 100 | 100 | 100 | 100 | 100 |
| claude-haiku-5.5@anthropic | 100 | 88 | 100 | 100 | 100 | 100 |
| glm-5.3-flash | 100 | 88 | 100 | 100 | 100 | 100 |
| claude-haiku-4.5@anthropic | 100 | 33 | 81 | 100 | 100 | 100 |
Note
anthropic/claude-haiku-4.5scheduled everyfuture-datedtransfer for2026-09-05, the wrongon_date, on all three trials; the rest of each call was correct. Ondecoy-numbertrials 1 and 3 it calledget_statementfor2026-06instead of the graded tool and scored zero.claude-haiku-5.5andglm-5.3-flasheach lost onedecoy-numbertrial on the date range alone, querying2026-07-01to2026-07-31.x-ai/grok-4.5was served 7% of its prompt tokens from cache despite cache busting, so read its latency and cost as slightly flattered.
Conclusion
Eleven targets cleared all 18 trials. openai/gpt-6-luna takes the crown on
cost × latency² at $0.00011 a task and 2,303ms, only just ahead of
deepseek/deepseek-v4.1-flash, which is more than twice as fast at 982ms but
costs $0.00063 a task. qwen/qwen3.7-flash matches gpt-6-luna on cost but
takes 6,047ms.
claude-haiku-4.5's future-dated miss is the fixable kind: one argument
wrong the same way every trial. Its decoy-number zeros are not: two of three
trials picked the wrong tool outright, so its 85.8 mean hides a case it loses
more often than it wins. The single date-range slips from claude-haiku-5.5
and glm-5.3-flash are minor.