Plate Recognition

2026-10-08 02:36 UTC

About this benchmark

Reads the plate off the vehicle in a 1920x1015 barrier camera still and returns it as one string. One field, so every trial scores 0 or 1 and the mean across four cases and five trials is a plain success rate.

The frames are real captures from a Malaysian multi-storey carpark: sun glare on a bonnet, a plate recessed behind a grille with dealer text under it, a two-line plate on a dirty bumper, and a small plate on an unlit level. Malaysian plates carry no state colour coding or hyphens, so there is no format prior to lean on when the pixels are ambiguous.

Each fixture is cropped along the bottom to drop a burnt-in camera overlay that printed the plate in clean text, so the plate can only be read from the plate. Faces and a passing motorcyclist are pixelated in the bytes the models are sent.

Cases

  • bmw-grille: BMW iX at a barrier, VJP 9981, with a dealer strip under the digits inside the plate frame.
  • honda-glare: Honda City on a sunlit rooftop level, VJF 5591, the plate shadowed under a hard windscreen flare.
  • kancil-dim: Perodua Kancil on an unlit level, WNH 8994, on a slanted decorative plate face that turns the leading W into an M.
  • proton-twoline: Proton Saga at an entry lane, BLM 8686, on a two-line plate with letters above digits.

Results

17/18 targets scored · 12 trials each · $0.50 total · show

Cost vs latencymost attractive quadrant
Latency & costsort by
TargetScore ↓StddevLatencyTok/sCostErrors
bestQwenqwen3.7-flash@alibaba100.00.03s12.7best$0.00080/12
Claudeclaude-haiku-5.5@anthropic100.00.03.9s4.3$0.00360/12
⚠Qwenqwen3.8-flash@alibabaerrorPOST "https://openrouter.ai/api/v1/chat/completions": 429 Too Many Requests {"message":"Provider returned error","code":429,"metadata":{"raw":"qwen/qwen3.8-flash is temporarily rate-limited upstream. Please retry shortly, or add your own key to accumulate your rate limits: https://openrouter.ai/settings/integrations","provider_name":"Alibaba","is_byok":false,"limit_source":"upstream_provider_shared_pool","remedy_hint":"Retry shortly, add your own provider key (https://openrouter.ai/settings/integrations), or route to another provider with provider routing: https://openrouter.ai/docs/features/provider-routing"}}100.00.011.7s9.1$0.00372/12
OpenAIgpt-6-luna@openai100.00.05.4s10.2$0.00400/12
Geminigemini-3.5-flash-lite@google-ai-studio100.00.01.7s9.9$0.00480/12
DeepSeekdeepseek-v4.1-flash100.00.0best1.4s107.0$0.00590/12
OpenAIgpt-5-mini@openai100.00.05.3s22.8$0.01020/12
Geminigemini-3.8-flash@google-ai-studio100.00.03.4s45.0$0.01750/12
OpenAIgpt-5.4-mini@openai100.00.03.7s4.6$0.02290/12
Claudeclaude-sonnet-5@anthropic100.00.03.9s4.3$0.07200/12
Grokgrok-4.6@xai100.00.07s37.9$0.07340/12
OpenAIgpt-5.6-sol@openai100.00.03.6s4.7$0.07540/12
OpenAIgpt-6.1-sol@openai100.00.04.8s7.2$0.07750/12
OpenAIgpt-6-sol@openai100.00.05s16.0$0.08290/12
⚠OpenAIgpt-5.6-luna@openai3 field misseskancil-dim: plate_number (3/3)75.043.33.9s10.0$0.00790/12
⚠Claudeclaude-haiku-4.5@anthropic3 field missesproton-twoline: plate_number (3/3)75.043.34.1s3.5$0.02290/12
⚠Qwenqwen3.7-plus@alibaba9 field missesbmw-grille: plate_number (2/3)honda-glare: plate_number (3/3)kancil-dim: plate_number (2/3)proton-twoline: plate_number (2/3)25.043.34.6s45.2$0.01130/12
⚠MoonshotAIkimi-k2.5@moonshotaierrorPOST "https://openrouter.ai/api/v1/chat/completions": 404 Not Found {"message":"No endpoints found for moonshotai/kimi-k2.5. Every candidate endpoint was removed during routing: Filter by Image Support removed siliconflow/int4; Filter by Parameters removed amazon-bedrock/us-east-2; Filter by Fallback removed atlas-cloud/int4, venice, novita.","code":404,"metadata":{"routing_funnel":[{"step":"Initial Endpoints","endpoint_count":5},{"step":"Filter by Image Support","endpoint_count":4},{"step":"Filter by Parameters","endpoint_count":3},{"step":"Filter by Fallback","endpoint_count":0}],"failed_routing_step":"Filter by Fallback"}}n/an/an/an/a$0.000012/12

Score by case

Every cell is that target's mean over the case's trials, and ✕ marks a reply that was never a document, whatever it scored.

Targetbmw-grillehonda-glarekancil-dimproton-twoline
Claudeclaude-haiku-5.5@anthropic100100100100
Claudeclaude-sonnet-5@anthropic100100100100
DeepSeekdeepseek-v4.1-flash100100100100
Geminigemini-3.5-flash-lite@google-ai-studio100100100100
Geminigemini-3.8-flash@google-ai-studio100100100100
OpenAIgpt-5-mini@openai100100100100
OpenAIgpt-5.4-mini@openai100100100100
OpenAIgpt-5.6-sol@openai100100100100
OpenAIgpt-6-luna@openai100100100100
OpenAIgpt-6-sol@openai100100100100
OpenAIgpt-6.1-sol@openai100100100100
Qwenqwen3.7-flash@alibaba100100100100
Qwenqwen3.8-flash@alibaba100100100100
Grokgrok-4.6@xai100100100100
Claudeclaude-haiku-4.5@anthropic1001001000
OpenAIgpt-5.6-luna@openai1001000100
Qwenqwen3.7-plus@alibaba33✕3333
MoonshotAIkimi-k2.5@moonshotaierrerrerrerr

Note

  • moonshotai/kimi-k2.5 produced no gradable output. All 12 calls returned 404 No endpoints found: OpenRouter's routing funnel went from five endpoints to zero, and the last three were removed at the fallback filter. Its 0.0 is 12 routing failures, not 12 wrong plates.
  • qwen/qwen3.8-flash lost two of its 12 calls (proton-twoline trials 2 and 3) to 429 Too Many Requests from the upstream provider. Its 100.0 is the mean of the 10 trials that returned, and proton-twoline was graded on one trial only.
  • qwen/qwen3.7-plus read the correct plate on all 12 trials and still scores 25.0. It wraps the object in an array ([{"plate_number": ...}]) on 9 trials; the 3 trials that returned a bare object scored 1. The score measures output shape, not reading.
  • The other two misses are real misreads, the same way on every trial of one case. openai/gpt-5.6-luna reads kancil-dim as MNH8994 on all three trials. anthropic/claude-haiku-4.5 reads the proton-twoline digits as 9595 (twice) and 9535 with a P prefix (once).

Conclusion

Fourteen targets read every graded plate, so the choice comes down to cost and speed. qwen/qwen3.7-flash is the cheapest clean sweep at $0.0008 for the run, at 2,951ms. If latency matters, deepseek/deepseek-v4.1-flash is the fastest at 1,369ms for $0.0059, with gemini-3.5-flash-lite close behind at 1,688ms for $0.0048. anthropic/claude-haiku-5.5 ($0.0036, 3,937ms) and gpt-6-luna ($0.0040) are also clean and cheap. The sol models, sonnet-5 and grok-4.6 get the same result for $0.072 to $0.083, which buys nothing here.

qwen3.7-plus is fixable: every plate is right, and an array unwrap in the caller recovers all of it, so its mean understates it. gpt-5.6-luna and haiku-4.5 are the unsafe kind: their 75.0 hides one case each that they get wrong on every trial, with well-formed JSON and a confident wrong plate, so nothing in the output flags the error.