Tool Haystack

2026-07-31 03:10 UTC · 14f11d7

About this benchmark

Fifty banking tools, one customer message, six cases. An earlier version offered thirty tools drawn from different corners of the bank and every target scored 100: when the answer is about a statement and the distractors are branch lookups, topic settles it without a description being read.

So the list is built in families. Ten tools take a card and its last four digits, eight report transactions, seven speak about statements, and the clause that separates them sits at the end of the description: a block is permanent, a freeze is reversible, a loss report changes nothing. Family members are scattered rather than grouped, and two cases are built so the tool whose name best fits the message is the wrong one.

`tool` is a gate, so a wrong pick scores zero however many arguments matched, and the arguments stay at transcription throughout. Three messages come from `tool-call` word for word, so the same request can be read at eight tools and at fifty.

Cases

hold-not-block: `stolen-card` read the other way, from the same eight card tools: the customer says not to cancel, so the answer is the hold they can lift.
merchant-search: The customer names the merchant, which rules out listing the window. `list_recurring_charges` answers "a double bill" rather than the request.
pending-not-posted: The merchant is named, so `search_transactions` looks right and is the one tool that cannot answer: it returns settled charges, and this one has not settled.
standing-not-scheduled: `schedule_transfer` runs once on a named day. The money moves every month.
statement-month: Six tools in the list speak about statements and only one returns what the month closed at.
stolen-card: `report_card_lost` records the loss but leaves the card chargeable, while a temporary freeze is for a card expected back. This one will not be recovered.

Results

14/14 targets scored · 6 trials each · $0.67 total

Cost vs latencymost attractive quadrant

Latency & costshorter is better

Target	Score ↓	Latency	Tok/s	Cost	Errors
qwen3.7-flash@alibaba	100.0	3.4s	121.7	$0.0016	0/6
gpt-5.6-luna@openai	100.0	2.3s	25.9	$0.0032	0/6
gpt-5-mini@openai	100.0	4.2s	56.2	$0.0088	0/6
gemini-2.5-flash@google-ai-studio	100.0	1.5s	33.5	$0.0108	0/6
gemini-3.5-flash-lite@google-ai-studio	100.0	1.3s	40.2	$0.0125	0/6
kimi-k2.5@moonshotai	100.0	10.1s	27.7	$0.0221	0/6
gpt-5.6-terra@openai	100.0	1.9s	30.1	$0.0316	0/6
claude-haiku-4.5@anthropic	100.0	2s	54.5	$0.0495	0/6
glm-5.2@z-ai	100.0	6.8s	15.0	$0.0582	0/6
deepseek-v4-pro@together	100.0	5.5s	35.3	$0.0753	0/6
gemini-3.6-flash@google-ai-studio	100.0	3.2s	170.5	$0.0836	0/6
grok-4.5@xai	100.0	2.5s	41.9	$0.0846	0/6
glm-4.7@cerebras	100.0	1.7s	172.4	$0.1028	0/6
claude-sonnet-5@anthropic	100.0	3s	45.5	$0.1240	0/6

Note

Roster changed since 30 July: glm-4.7@z-ai and kimi-k2.6@moonshotai dropped, deepseek-v4-pro added.
deepseek-v4-pro is pinned to @together. The first-party @deepseek pin returns 404 "No endpoints found" on this key, and unpinned it routed to BaseTen, giving up the deterministic routing every other target has. Read its cost and latency as Together's, not DeepSeek's.

Conclusion

Fourteen targets tie at 100, so cost and latency decide. qwen3.7-flash takes it at $0.00027 a task, twice cheaper than gpt-5.6-luna behind it; where latency matters more, gemini-3.5-flash-lite at 1294ms. deepseek-v4-pro clears the corpus on its first appearance but at $0.01255 a task and 5466ms, so nothing about this run recommends it over the field.

Third corpus, third flat column, and this one was built to fail. The list went from forty-four tools to fifty, and the six additions each shadow one of the six correct answers rather than filling the domain out: search_transactions_all_accounts and get_statement_closing_balance were written to match the customer's wording better than the tool the case wants. Neither was selected once. Across 84 gradable trials every target picked the right tool and made exactly one call, and no new distractor was chosen at all. List size is answered: 8, 30, 44 and 50 tools all produce the same column, and adding more is measuring nothing. The next signal has to come from cases where the correct move is not a single obvious call, from arguments that need resolution rather than transcription, or from turns where a wrong call compounds.