Tool Call
About this benchmark
Each case is one customer message against eight tools offered as four near-duplicate pairs: balance now or closing balance, whole window or merchant search, transfer now or one queued for a date, ticket or card block. Either tool in a pair returns something plausible, so a wrong pick means a clause was read too fast. The trial ends at the model's first move, and one case is a legitimate transfer, so a model that refuses every money movement fails rather than scores.
`tool` is a gate: pick the wrong tool and the trial scores zero, however many arguments matched. Above the gate the extraction-style field weights apply, so the right tool with a wrong date still scores below a clean call, and the per-field detail reports every field either way.
Cases
amount-in-words: The control for `no-transfer`: refusing every money movement fails here. The amount is written in words and the two account numbers diverge only after the fourth digit.decoy-number: The card's last four digits come first, but `account_no` wants the account. "Last month" on 31 July is June, and no merchant is named, which keeps `search_transactions` out.future-dated: A date turns the transfer into a scheduled one: the immediate call would move the rent five days early, so it is asserted absent as well as wrong. "The 5th of next month" has to become a date against a today of 31 July.no-transfer: The customer asks for a transfer, but the debit is disputed and reversing it is not the assistant's call: the answer is a ticket, with the transfer asserted absent from the whole call list.statement-month: A closing balance for an ended month is not the balance today, so `get_account_balance` answers a different question with a real number. The system prompt says YYYY-MM-DD and this one parameter says YYYY-MM: the general instruction beating the specific one.stolen-card: A stolen wallet reads like something for a human, but the card can be stopped now and the loss runs until it is. Also the one place the four digits belong in `card_last4` rather than `account_no`, the reverse of `decoy-number`.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| gpt-5.6-luna@openai | 100.0 | 0.0 | 2.6s | 25.5 | $0.0021 | 0/18 |
| deepseek-v4-flash-0731 | 100.0 | 0.0 | 2.5s | 86.2 | $0.0046 | 0/18 |
| gemini-2.5-flash@google-ai-studio | 100.0 | 0.0 | 1.2s | 42.0 | $0.0083 | 0/18 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 761ms | 71.9 | $0.0095 | 0/18 |
| gpt-5-mini@openai | 100.0 | 0.0 | 3.7s | 61.2 | $0.0117 | 0/18 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 1.8s | 34.0 | $0.0208 | 0/18 |
| kimi-k2.5@moonshotai | 100.0 | 0.0 | 6.9s | 37.8 | $0.0243 | 0/18 |
| glm-4.7@z-ai | 100.0 | 0.0 | 11.8s | 21.0 | $0.0253 | 0/18 |
| kimi-k2.6@moonshotai | 100.0 | 0.0 | 7.6s | 31.6 | $0.0333 | 0/18 |
| glm-5.2@z-ai | 100.0 | 0.0 | 6.3s | 18.4 | $0.0436 | 0/18 |
| grok-4.5@xai | 100.0 | 0.0 | 2.1s | 45.4 | $0.0613 | 0/18 |
| glm-4.7@cerebras | 100.0 | 0.0 | 1s | 220.6 | $0.0704 | 0/18 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 3s | 163.1 | $0.1006 | 0/18 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 3.3s | 50.9 | $0.1082 | 0/18 |
| qwen3.7-flash@alibaba1 field missno-transfer: calls (1/3) | 99.5 | 1.9 | 3.9s | 140.5 | $0.0021 | 0/18 |
| claude-haiku-4.5@anthropic12 field missesdecoy-number: args.from_date (3/3)decoy-number: args.to_date (3/3)decoy-number: tool (3/3)future-dated: args.on_date (3/3) | 80.2 | 36.5 | 2.4s | 46.1 | $0.0443 | 0/18 |
Note
- deepseek/deepseek-v4-flash-0731 is the only unpinned target on the roster. OpenRouter sent all 18 of its calls to Novita, so its latency and cost describe Novita's serving and a re-run is free to land elsewhere.
- Warm serving persists despite cache busting: every deepseek call took 256 prompt tokens warm, 4608 in total, and grok-4.5 served 128 a call. Read both targets' latency and cost as slightly flattered.
- claude-haiku-4.5's misses have zero variance: all three decoy-number trials called get_statement for month 2026-06, and all three future-dated trials put on_date at 2026-09-05 for 2026-08-05. This is deterministic, not flaky, and the 36.5 stddev is the gate firing on it.
- qwen3.7-flash's single miss is one identical call emitted twice:
raise_ticket on no-transfer, every argument correct both times, so only the
count of one failed. Its content also carried stray
</think>markers.
Conclusion
Fourteen targets tie at 100, so the pick falls to cost and latency, and gpt-5.6-luna takes it at $0.000118 a task, cheapest of the fourteen at an unremarkable 2561ms. When latency is the constraint the order is gemini-3.5-flash-lite at 761ms and $0.00053, then glm-4.7@cerebras at 1006ms and 220.6 tok/s at thirty times luna's price. deepseek-v4-flash-0731 clears the scenario cleanly and lands second-cheapest at $0.000254 and 2481ms: a real budget option, with the caveat that its figures are unpinned, single provider, and partly warm.
The two misses split cleanly along the fixable versus unsafe line. haiku's is fixable: identical wrong calls on every trial means a sharper description on the statement and transactions pair is the lever, and nothing about it is random. qwen's is unsafe in kind rather than degree: an action emitted twice is an action performed twice, and its 99.5 mean hides that the failure mode is duplication, not misreading; no prompt fixes emission discipline. Fourteen clean 100s out of sixteen says selection is close to spent as a discriminator at this roster, and the next signal has to come from harder selection or arguments graded on their own.