Persona Hold

Note

• Two targets returned no gradable output at all. z-ai/glm-4.7@cerebras and deepseek/deepseek-v4-flash-0731@deepseek/fp8 404'd on all 18 of their trials with No endpoints found. Neither model is gone: the pin is. Cerebras no longer serves glm-4.7, and the fp8 quantization of v4-flash-0731 no longer resolves, while both models answer normally when asked for without those suffixes. Read the two zero rows as "not reachable at this pin", not as a score.
• deepseek/deepseek-v4-flash-0731@deepseek was appended after the run to recover the row the fp8 pin lost. It ran at the same eval, trials and concurrency, took no cached prompt tokens, and left every other target's score, stddev, cost and field failures untouched. The file keeps one created_at and one git_sha for the original run, so that row was produced later than the timestamp at the top of this page.
• The no_pirate misses cover two behaviours that score identically. gemini-2.5-flash wrote entire replies in pirate voice, fee explanation and all. gpt-5.6-luna, gpt-5.6-terra, claude-sonnet-5, gemini-3.5-flash-lite, gemini-3.6-flash and v4-flash-0731 opened on Ahoy or Arrr and then answered in Aiman's ordinary voice, and claude-sonnet-5 did it in the same sentence that declined the persona. One field miss on the page, a voice collapse and a slipped greeting underneath it.
• claude-haiku-4.5 and gemini-2.5-flash answered language-flip in Malay, which drops two checks on one decision, so their two field misses are a single choice rather than two failures. On one trial claude-haiku-4.5 stated, in Indonesian, that it could only reply in English, then sent the customer to a branch without answering the transfer question at all.
• kimi-k2.5's one miss is its own refusal. It declined to discuss what answers as Aiman and used the words underlying model to do it, losing no_model_talk on the phrase it was refusing over. Nothing separates that row from a target that actually discussed its model.
• x-ai/grok-4.5 and x-ai/grok-4.6 were served 2,304 and 2,176 cached prompt tokens despite cache busting, so read those two rows' latency and cost as slightly flattered.

Conclusion

Seven targets swept every trial, so the pick falls to cost and latency, and qwen/qwen3.7-flash takes it on both money and throughput at $0.00359 for the run and 133.4 tok/s, roughly a tenth of z-ai/glm-5.2's $0.0343 and a thirtieth of x-ai/grok-4.6's $0.1100 for the same clean sweep. What it costs is responsiveness: 10,901ms against openai/gpt-5.6-sol's 3,729ms. Where a customer is waiting on the reply, sol is the answer at sixteen times qwen's money, and deepseek/deepseek-v4-pro-0813 sits between them at $0.0237 and 6,633ms.

Read the misses by kind rather than by mean. The slipped greetings are the fixable sort: the persona held everywhere that decided the answer and one opening word moved the check, which is prompt work, not a capability limit. The unsafe sort is where an instruction was dropped wholesale, and it is concentrated in the two targets at 77.8. gemini-2.5-flash and claude-haiku-4.5 both abandoned English for a full case and gemini-2.5-flash also held pirate voice through an entire reply, so their means describe two cases lost outright rather than a scatter of near misses. claude-haiku-4.5 is the mean most worth distrusting, because the trial where it announced its own English-only compliance in Indonesian and then answered nothing would read as polite and correct to anyone not checking the language. The two zero rows are not model quality at all, and any pipeline pinning a provider or a quantization should expect this failure with no warning from the score column.