Tool Loop
About this benchmark
Twenty-nine tools, five requests, and no request any one tool can answer. The calls are answered from a lookup table beside each case, so the loop runs for real: the model calls, reads a result, and calls again with what the result told it.
Every chain is locked by an identifier the model cannot know in advance. Places are found by area, so coordinates have to be resolved first; a session is ranked by exchange code and priced by ticker; a fare needs two station ids. A call whose identifier the world does not recognise is refused rather than answered loosely, so guessing costs a turn and shows up in `rejected`.
The answer is a gate, so a right chain ending in a wrong number scores zero while the per-field detail still says which hop broke. Three cases leave the arithmetic to the model. The distractors are close rather than absent: council parking beside a venue's own car park, an e-hailing estimate beside a rail fare, and all four ranking measures populated, so the wrong one returns a plausible company rather than an error.
Cases
dividend-yield: No tool returns a yield: it is 0.63 declared against a last price of 9.84, divided by the model. `get_company_profile` answers for the same ticker with everything except that.index-close: Two calls are enough. Everything else the world answers here is real and adjacent, so a model that keeps going lands on constituents and a market status.mall-parking: `find_places` takes an area and not coordinates, so the geocode is unavoidable. The nearest mall is 0.4 km out and the famous one 2.9 km, and free bays are a subtraction the model does itself.most-traded-stock: Shares changing hands is `volume`, and all four measures are populated, so the wrong one returns a different company at a plausible price rather than an error. The volume list is unsorted and the leader sits third.station-fare: The two middle hops are independent, so grading matches on tool name rather than position. The nearest station is not the one sharing the area's name, and both routes to KL Sentral are priced.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 7.8s | 102.6 | $0.0074 | 0/15 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 10.5s | 18.1 | $0.0075 | 0/15 |
| gpt-5-mini@openai | 100.0 | 0.0 | 23s | 41.2 | $0.0417 | 0/15 |
| kimi-k2.5@moonshotai | 100.0 | 0.0 | 14.2s | 26.8 | $0.0574 | 0/15 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 8s | 19.2 | $0.0701 | 0/15 |
| glm-5.2@z-ai | 100.0 | 0.0 | 23.6s | 11.5 | $0.1328 | 0/15 |
| grok-4.5@xai | 100.0 | 0.0 | 6.6s | 34.3 | $0.1888 | 0/15 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 7.1s | 45.8 | $0.2670 | 0/15 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 7.6s | 102.4 | $0.3817 | 0/15 |
| glm-4.7@cerebras | 100.0 | 0.0 | 3s | 136.3 | $0.5223 | 0/15 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 8.1s | 36.8 | $0.6025 | 0/15 |
| deepseek-v4-flash-0731@deepseek3 field missesstation-fare: calls (1/3)station-fare: calls.#(name=="get_transit_fare").args.to_station_id (1/3)station-fare: rejected (1/3) | 97.8 | 8.3 | 5.4s | 67.7 | $0.0098 | 0/15 |
| gemini-2.5-flash@google-ai-studio10 field missesstation-fare: calls (1/3)station-fare: calls.#(name=="get_transit_fare").args.from_station_id (3/3)station-fare: content (3/3)station-fare: rejected (3/3) | 80.0 | 40.0 | 4.5s | 23.4 | $0.0417 | 0/15 |
| gemini-3.5-flash-lite@google-ai-studio6 field missesstation-fare: calls.#(name=="get_transit_fare").args.from_station_id (3/3)station-fare: content (3/3) | 80.0 | 40.0 | 3.3s | 32.7 | $0.0621 | 0/15 |
| deepseek-v4-pro@together6 field missesindex-close: content (1/3)mall-parking: calls (1/1)mall-parking: calls.#(name=="find_places").args.area (1/1)mall-parking: calls.#(name=="find_places").args.category (1/1)mall-parking: calls.#(name=="get_place_parking").args.place_id (1/1)mall-parking: content (1/1)errorPOST "https://openrouter.ai/api/v1/chat/completions": 429 Too Many Requests {"message":"Provider returned error","code":429,"metadata":{"raw":"deepseek/deepseek-v4-pro is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations","provider_name":"Together","is_byok":false,"limit_source":"upstream_provider_shared_pool","remedy_hint":"Retry shortly, add your own provider key (https://openrouter.ai/settings/integrations), or route to another provider with provider routing: https://openrouter.ai/docs/features/provider-routing"}} | 77.8 | 41.6 | 9.8s | 35.7 | $0.0858 | 6/15 |
Note
deepseek-v4-pro's mean covers nine trials, not fifteen: six calls died as upstream 429s from Together, every most-traded-stock trial among them, and two retries per call did not get through. Its two zero-scored survivors returned an empty final answer, one after a single call, so the 77.8 mostly measures a provider shedding load, not the model.gemini-2.5-flashmisspelt the from-station key the same way in every station-fare trial, had every fare call refused for it, and then told the user no route exists. The confident false negative is the finding; the refusals were its own doing.gemini-3.5-flash-literan a clean loop with nothing refused and quoted a fare from the wrong origin station, identically in all three trials. That is a deterministic wrong pick, not flakiness, despite the 40.0 in the stddev column.deepseek-v4-flash-0731's single sub-perfect trial passed a station name where an id belongs, was refused, then corrected itself on the next call and answered correctly; the 0.667 on that trial is shape penalties around a right answer.- 1.12M of 2.84M prompt tokens were served from cache despite cache busting,
and unevenly:
v4-flash-0731read 74% of its prompt tokens warm, the highest share on the roster, so its latency and cost are best-case numbers.
Conclusion
Eleven targets tie at 100 and the tie breaks on cost, where qwen3.7-flash
and gpt-5.6-luna are inseparable at $0.000495 and $0.000497 a task;
qwen takes it on latency, 7802ms against 10529ms, at 102.6 tok/s against
18.1. Where speed is the whole requirement, glm-4.7@cerebras answers in
2997ms for seventy times qwen's money. The new v4-flash-0731 lands at 97.8
for $0.00066 a task, third cheapest on the roster, with one recovered refusal
as its only blemish; given its warm cache share, read its speed and price as
a best case rather than a measurement.
The three targets below 100 need three different readings. flash-lite's
miss is fixable: the loop is well formed and the wrong origin is one visible
field a sharper prompt can move. 2.5-flash's is unsafe: it malformed an
argument, got refused, and converted its own failure into a confident "no
route" answer for the user, which no score gate separates from an honest dead
end. v4-pro's 77.8 should not be acted on from this run: six of fifteen
trials produced nothing to grade and two more came back empty, so re-run it
on another provider before reading anything into the number.