Tool Loop
About this benchmark
Twenty-nine tools, five requests, and no request any one tool can answer. The calls are answered from a lookup table beside each case, so the loop runs for real: the model calls, reads a result, and calls again with what the result told it.
Every chain is locked by an identifier the model cannot know in advance. Places are found by area, so coordinates have to be resolved first; a session is ranked by exchange code and priced by ticker; a fare needs two station ids. A call whose identifier the world does not recognise is refused rather than answered loosely, so guessing costs a turn and shows up in `rejected`.
The answer is a gate, so a right chain ending in a wrong number scores zero while the per-field detail still says which hop broke. Three cases leave the arithmetic to the model. The distractors are close rather than absent: council parking beside a venue's own car park, an e-hailing estimate beside a rail fare, and all four ranking measures populated, so the wrong one returns a plausible company rather than an error.
Cases
dividend-yield: No tool returns a yield: it is 0.63 declared against a last price of 9.84, divided by the model. `get_company_profile` answers for the same ticker with everything except that.index-close: Two calls are enough. Everything else the world answers here is real and adjacent, so a model that keeps going lands on constituents and a market status.mall-parking: `find_places` takes an area and not coordinates, so the geocode is unavoidable. The nearest mall is 0.4 km out and the famous one 2.9 km, and free bays are a subtraction the model does itself.most-traded-stock: Shares changing hands is `volume`, and all four measures are populated, so the wrong one returns a different company at a plausible price rather than an error. The volume list is unsorted and the leader sits third.station-fare: The two middle hops are independent, so grading matches on tool name rather than position. The nearest station is not the one sharing the area's name, and both routes to KL Sentral are priced.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 10.7s | 60.3 | $0.0047 | 0/15 |
| gpt-6-luna@openai | 100.0 | 0.0 | 8.9s | 21.1 | $0.0069 | 0/15 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 9.2s | 19.2 | $0.0141 | 0/15 |
| glm-5.3-flash@z-ai | 100.0 | 0.0 | 17.8s | 19.2 | $0.0152 | 0/15 |
| deepseek-v4.1-flash | 100.0 | 0.0 | 3.1s | 87.5 | $0.0232 | 0/15 |
| claude-haiku-5.5@anthropic | 100.0 | 0.0 | 5.5s | 74.5 | $0.0309 | 0/15 |
| gpt-6.1-sol@openai | 100.0 | 0.0 | 9.5s | 12.2 | $0.1154 | 0/15 |
| gpt-6-sol@openai | 100.0 | 0.0 | 9.5s | 16.1 | $0.1319 | 0/15 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 7.9s | 19.5 | $0.1361 | 0/15 |
| grok-4.5@xai | 100.0 | 0.0 | 7.8s | 28.9 | $0.2011 | 0/15 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 6.9s | 45.8 | $0.2606 | 0/15 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 7.2s | 41.2 | $0.6029 | 0/15 |
| ⚠gemini-3.8-flash@google-ai-studio3 field missesdividend-yield: calls (1/3)station-fare: calls (2/3) | 98.3 | 3.4 | 14s | 55.1 | $0.2088 | 0/15 |
| ⚠gemini-3.5-flash-lite@google-ai-studio2 field missesstation-fare: calls.#(name=="get_transit_fare").args.from_station_id (1/3)station-fare: content (1/3) | 93.3 | 24.9 | 4.4s | 24.7 | $0.0627 | 0/15 |
| ⚠qwen3.8-flash@alibaba43 field missesdividend-yield: calls (2/3)dividend-yield: calls.#(name=="get_dividend_history").args.ticker (2/3)dividend-yield: calls.#(name=="get_quote").args.ticker (2/3)dividend-yield: calls.#(name=="search_ticker").args.name (2/3)dividend-yield: calls.0.name (2/3)dividend-yield: content (2/3)index-close: calls (2/2)index-close: calls.#(name=="get_index_level").args.index (1/2)index-close: calls.#(name=="list_exchanges").args.country (1/2)index-close: calls.0.name (1/2)index-close: content (1/2)mall-parking: calls (3/3)mall-parking: calls.#(name=="find_places").args.area (3/3)mall-parking: calls.#(name=="find_places").args.category (3/3)mall-parking: calls.#(name=="get_place_parking").args.place_id (3/3)mall-parking: calls.0.name (3/3)mall-parking: content (3/3)station-fare: calls (1/1)station-fare: calls.#(name=="find_transit_stations").args.area (1/1)station-fare: calls.#(name=="get_transit_fare").args.from_station_id (1/1)station-fare: calls.#(name=="get_transit_fare").args.to_station_id (1/1)station-fare: calls.#(name=="reverse_geocode").args.lat (1/1)station-fare: calls.#(name=="search_station").args.name (1/1)station-fare: content (1/1)errorPOST "https://openrouter.ai/api/v1/chat/completions": 429 Too Many Requests {"message":"Provider returned error","code":429,"metadata":{"raw":"qwen/qwen3.8-flash is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations","provider_name":"Alibaba","is_byok":false,"limit_source":"upstream_provider_shared_pool","remedy_hint":"Retry shortly, add your own provider key (https://openrouter.ai/settings/integrations), or route to another provider with provider routing: https://openrouter.ai/docs/features/provider-routing"}} | 20.0 | 37.7 | 16.9s | 28.1 | $0.0077 | 6/15 |
Score by case
| Target | dividend-yield | index-close | mall-parking | most-traded-stock | station-fare |
|---|---|---|---|---|---|
| claude-haiku-4.5@anthropic | 100 | 100 | 100 | 100 | 100 |
| claude-haiku-5.5@anthropic | 100 | 100 | 100 | 100 | 100 |
| claude-sonnet-5@anthropic | 100 | 100 | 100 | 100 | 100 |
| deepseek-v4.1-flash | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-luna@openai | 100 | 100 | 100 | 100 | 100 |
| gpt-5.6-terra@openai | 100 | 100 | 100 | 100 | 100 |
| gpt-6-luna@openai | 100 | 100 | 100 | 100 | 100 |
| gpt-6-sol@openai | 100 | 100 | 100 | 100 | 100 |
| gpt-6.1-sol@openai | 100 | 100 | 100 | 100 | 100 |
| qwen3.7-flash@alibaba | 100 | 100 | 100 | 100 | 100 |
| grok-4.5@xai | 100 | 100 | 100 | 100 | 100 |
| glm-5.3-flash@z-ai | 100 | 100 | 100 | 100 | 100 |
| gemini-3.8-flash@google-ai-studio | 97 | 100 | 100 | 100 | 94 |
| gemini-3.5-flash-lite@google-ai-studio | 100 | 100 | 100 | 100 | 67 |
| qwen3.8-flash@alibaba | 33 | 40 | 0 | err | 0 |
Note
qwen/qwen3.8-flashwas rate-limited upstream by Alibaba (429) on 6 of 15 trials, all threemost-traded-stocktrials among them, so it never produced an answer for that case. In 7 of its other 9 trials it replied that no request had come through, asked what the user wanted, and made no tool call. Its 20.0 measures that, not the task.google/gemini-3.5-flash-litequoted the fare from the wrong origin station onstation-faretrial 2; its loop was clean and nothing was refused.google/gemini-3.8-flashgave the right answer in all 15 trials. Its three partial scores are on the call list only.- Most targets read 68% to 73% of their prompt tokens from cache despite cache busting. Anthropic and Google targets read none, so their latency and cost are not best-case numbers the way the rest are.
Conclusion
Twelve targets tie at 100. deepseek/deepseek-v4.1-flash takes the crown on
cost × latency² at $0.00155 a task and 3,086ms, the fastest target to clear
every trial. qwen/qwen3.7-flash ($0.00031 a task) and openai/gpt-6-luna
($0.00046) are cheaper but take 10,723ms and 8,884ms. All three read 68% to 72%
of their prompt tokens from cache, so treat those prices as a best case;
claude-haiku-5.5 at 5,533ms and $0.0021 a task read none.
gemini-3.5-flash-lite's miss is one wrong answer on one trial, confidently
stated; its 93.3 is honest but the failure is silent. qwen3.8-flash is
unusable here: rate limits aside, most of its served trials ignored the
question.