Tool Haystack
About this benchmark
Fifty banking tools, one customer message, six cases. An earlier version offered thirty tools drawn from different corners of the bank and every target scored 100: when the answer is about a statement and the distractors are branch lookups, topic settles it without a description being read.
So the list is built in families. Ten tools take a card and its last four digits, eight report transactions, seven speak about statements, and the clause that separates them sits at the end of the description: a block is permanent, a freeze is reversible, a loss report changes nothing. Family members are scattered rather than grouped, and two cases are built so the tool whose name best fits the message is the wrong one.
`tool` is a gate, so a wrong pick scores zero however many arguments matched, and the arguments stay at transcription throughout. Three messages come from `tool-call` word for word, so the same request can be read at eight tools and at fifty.
Cases
hold-not-block: `stolen-card` read the other way, from the same eight card tools: the customer says not to cancel, so the answer is the hold they can lift.merchant-search: The customer names the merchant, which rules out listing the window. `list_recurring_charges` answers "a double bill" rather than the request.pending-not-posted: The merchant is named, so `search_transactions` looks right and is the one tool that cannot answer: it returns settled charges, and this one has not settled.standing-not-scheduled: `schedule_transfer` runs once on a named day. The money moves every month.statement-month: Six tools in the list speak about statements and only one returns what the month closed at.stolen-card: `report_card_lost` records the loss but leaves the card chargeable, while a temporary freeze is for a card expected back. This one will not be recovered.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 5.6s | 75.0 | $0.0050 | 0/18 |
| gpt-6-luna@openai | 100.0 | 0.0 | 2.5s | 22.5 | $0.0095 | 0/18 |
| claude-haiku-5.5@anthropic | 100.0 | 0.0 | 2s | 94.8 | $0.0189 | 0/18 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 2.6s | 24.0 | $0.0191 | 0/18 |
| glm-5.3-flash | 100.0 | 0.0 | 3s | 44.0 | $0.0219 | 0/18 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.2s | 43.9 | $0.0377 | 0/18 |
| deepseek-v4.1-flash | 100.0 | 0.0 | 1.2s | 113.4 | $0.0397 | 0/18 |
| gemini-3.8-flash@google-ai-studio | 100.0 | 0.0 | 4.8s | 97.4 | $0.1203 | 0/18 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 2.1s | 52.6 | $0.1483 | 0/18 |
| gpt-6.1-sol@openai | 100.0 | 0.0 | 2.4s | 16.0 | $0.1871 | 0/18 |
| gpt-6-sol@openai | 100.0 | 0.0 | 2.1s | 21.8 | $0.1875 | 0/18 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 2.3s | 24.2 | $0.1908 | 0/18 |
| grok-4.5@xai | 100.0 | 0.0 | 3.1s | 38.8 | $0.2580 | 0/18 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 2.6s | 53.2 | $0.3714 | 0/18 |
| ⚠qwen3.8-flash@alibaba19 field misseshold-not-block: args.account_no (1/2)merchant-search: args.account_no (2/3)merchant-search: args.from_date (2/3)merchant-search: args.merchant (1/3)merchant-search: args.to_date (2/3)merchant-search: calls (1/3)merchant-search: tool (2/3)pending-not-posted: args.account_no (2/3)standing-not-scheduled: args.day_of_month (1/3)standing-not-scheduled: args.from_account (1/3)standing-not-scheduled: args.start_date (1/3)standing-not-scheduled: args.to_account (1/3)stolen-card: args.account_no (1/3)stolen-card: args.card_last4 (1/3)errorPOST "https://openrouter.ai/api/v1/chat/completions": 429 Too Many Requests {"message":"Provider returned error","code":429,"metadata":{"raw":"qwen/qwen3.8-flash is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations","provider_name":"Alibaba","is_byok":false,"limit_source":"upstream_provider_shared_pool","remedy_hint":"Retry shortly, add your own provider key (https://openrouter.ai/settings/integrations), or route to another provider with provider routing: https://openrouter.ai/docs/features/provider-routing"}} | 77.5 | 33.2 | 20.5s | 40.9 | $0.0258 | 1/18 |
Score by case
| Target | hold-not-block | merchant-search | pending-not-posted | standing-not-scheduled | statement-month | stolen-card |
|---|---|---|---|---|---|---|
| claude-haiku-4.5@anthropic | 100 | 100 | 100 | 100 | 100 | 100 |
| claude-haiku-5.5@anthropic | 100 | 100 | 100 | 100 | 100 | 100 |
| claude-sonnet-5@anthropic | 100 | 100 | 100 | 100 | 100 | 100 |
| deepseek-v4.1-flash | 100 | 100 | 100 | 100 | 100 | 100 |
| gemini-3.5-flash-lite@google-ai-studio | 100 | 100 | 100 | 100 | 100 | 100 |
| gemini-3.8-flash@google-ai-studio | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-luna@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-terra@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6-luna@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6-sol@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| gpt-6.1-sol@openai | 100 | 100 | 100 | 100 | 100 | 100 |
| qwen3.7-flash@alibaba | 100 | 100 | 100 | 100 | 100 | 100 |
| grok-4.5@xai | 100 | 100 | 100 | 100 | 100 | 100 |
| glm-5.3-flash | 100 | 100 | 100 | 100 | 100 | 100 |
| qwen3.8-flash@alibaba | 91 | 33 | 78 | 84 | 100 | 83 |
Note
qwen/qwen3.8-flashwas rate-limited upstream by Alibaba (429) onhold-not-blocktrial 2, so its mean covers 17 trials. Of the rest,merchant-searchtrial 3 replied that no request had come through and made no call, and trial 2 calledsearch_transactions_all_accountsover a three-month range. Its other misses are wrong or missingaccount_noon four trials and every transfer field wrong onstanding-not-scheduledtrial 1.x-ai/grok-4.5was served 2% of its prompt tokens from cache despite cache busting, so read its latency and cost as slightly flattered.
Conclusion
Fourteen targets cleared all 18 trials, so cost and latency decide.
google/gemini-3.5-flash-lite takes the crown on cost × latency² at
$0.00209 a task and 1,173ms, with deepseek/deepseek-v4.1-flash close behind
(1,181ms, $0.00221 a task). openai/gpt-6-luna is a quarter of the price at
$0.00053 a task but takes 2,517ms, and qwen/qwen3.7-flash is cheapest at
$0.0050 for the run but takes 5,626ms.
qwen3.8-flash is the only failure and it is unsafe: its misses spread across
five of six cases with no single field in common, and it once ignored the
request entirely. Its 77.5 mean also omits one rate-limited trial.