Plate Recognition

2026-09-03 03:10 UTC

About this benchmark

Reads the plate off the vehicle in a 1920x1015 barrier camera still and returns it as one string. One field, so every trial scores 0 or 1 and the mean across four cases and five trials is a plain success rate.

The frames are real captures from a Malaysian multi-storey carpark: sun glare on a bonnet, a plate recessed behind a grille with dealer text under it, a two-line plate on a dirty bumper, and a small plate on an unlit level. Malaysian plates carry no state colour coding or hyphens, so there is no format prior to lean on when the pixels are ambiguous.

Each fixture is cropped along the bottom to drop a burnt-in camera overlay that printed the plate in clean text, so the plate can only be read from the plate. Faces and a passing motorcyclist are pixelated in the bytes the models are sent.

Cases

  • bmw-grille: BMW iX at a barrier, VJP 9981, with a dealer strip under the digits inside the plate frame.
  • honda-glare: Honda City on a sunlit rooftop level, VJF 5591, the plate shadowed under a hard windscreen flare.
  • kancil-dim: Perodua Kancil on an unlit level, WNH 8994, on a slanted decorative plate face that turns the leading W into an M.
  • proton-twoline: Proton Saga at an entry lane, BLM 8686, on a two-line plate with letters above digits.

Results

15/16 targets scored · 12 trials each · $0.47 total · show

Cost vs latencymost attractive quadrant
Latency & costsort by
TargetScoreStddevLatencyTok/sCostErrors
bestQwenqwen3.7-flash@alibaba100.00.03.3s24.4best$0.00090/12
Qwenqwen3.8-flash@alibaba100.00.06.3s15.9$0.00420/12
Geminigemini-3.5-flash-lite@google-ai-studio100.00.0best1.6s10.3$0.00480/12
OpenAIgpt-5-mini@openai100.00.03.3s34.5$0.01010/12
Geminigemini-3.8-flash@google-ai-studio100.00.03.3s36.9$0.01620/12
Geminigemini-3.6-flash@google-ai-studio100.00.02.7s69.2$0.01900/12
OpenAIgpt-5.4-mini@openai100.00.01.9s8.8$0.02290/12
Grokgrok-4.5@xai100.00.01.8s6.1$0.05460/12
Claudeclaude-sonnet-5@anthropic100.00.03.7s4.6$0.07210/12
Grokgrok-4.6@xai100.00.06.1s48.4$0.07450/12
OpenAIgpt-5.6-sol@openai100.00.02.2s7.7$0.07540/12
OpenAIgpt-5.6-terra@openai1 field missproton-twoline: plate_number (1/3)91.727.61.9s9.1$0.07580/12
OpenAIgpt-5.6-luna@openai3 field misseskancil-dim: plate_number (3/3)75.043.32.4s13.7$0.00780/12
Claudeclaude-haiku-4.5@anthropic3 field missesproton-twoline: plate_number (3/3)75.043.32.3s6.0$0.02290/12
Qwenqwen3.7-plus@alibaba4 field misseshonda-glare: plate_number (2/3)kancil-dim: plate_number (1/3)proton-twoline: plate_number (1/3)66.747.15.3s33.7$0.01090/12
MoonshotAIkimi-k2.5@moonshotaierrorPOST "https://openrouter.ai/api/v1/chat/completions": 404 Not Found {"message":"No endpoints found for moonshotai/kimi-k2.5.","code":404,"metadata":{"routing_funnel":[{"step":"Initial Endpoints","endpoint_count":8},{"step":"Filter by Image Support","endpoint_count":7},{"step":"Filter by Parameters","endpoint_count":4},{"step":"Apply Status Sorting","endpoint_count":4},{"step":"Filter by Fallback","endpoint_count":0}]}}n/an/an/an/a$0.000012/12

Score by case

Every cell is that target's mean over the case's trials, and ✕ marks a reply that was never a document, whatever it scored.

Targetbmw-grillehonda-glarekancil-dimproton-twoline
Claudeclaude-sonnet-5@anthropic100100100100
Geminigemini-3.5-flash-lite@google-ai-studio100100100100
Geminigemini-3.6-flash@google-ai-studio100100100100
Geminigemini-3.8-flash@google-ai-studio100100100100
OpenAIgpt-5-mini@openai100100100100
OpenAIgpt-5.4-mini@openai100100100100
OpenAIgpt-5.6-sol@openai100100100100
Qwenqwen3.7-flash@alibaba100100100100
Qwenqwen3.8-flash@alibaba100100100100
Grokgrok-4.5@xai100100100100
Grokgrok-4.6@xai100100100100
OpenAIgpt-5.6-terra@openai10010010067
Claudeclaude-haiku-4.5@anthropic1001001000
OpenAIgpt-5.6-luna@openai1001000100
Qwenqwen3.7-plus@alibaba100336767
MoonshotAIkimi-k2.5@moonshotaierrerrerrerr

Note

  • moonshotai/kimi-k2.5 produced no gradable output. Every call returned 404 No endpoints found for moonshotai/kimi-k2.5, and the routing funnel in the error shows why: eight endpoints, seven after the image-support filter, four after parameters, then zero once the fallback filter ran. Retries did not clear it. Its 0.0 is twelve routing failures, not twelve wrong plates, so nothing on this page measures the model.
  • qwen/qwen3.7-plus read all four plates correctly on all twelve trials. Its four zeros are the envelope: it returned [{"plate_number": "VJF5591"}], a one-element array, where the bare object was asked for. The same slug returns the bare object on the other eight trials, including trials of the same case, so the score measures how often it settles on the requested shape and not whether it can read anything.
  • anthropic/claude-haiku-4.5 misses proton-twoline on all three trials with three different answers, PLM65S, PLM9505 and PLM9595 against a true BLM8686. The leading B comes back as P every time while the digits move, so repeated calls would not converge on the right plate.
  • openai/gpt-5.6-luna returns MNH8994 for kancil-dim on all three trials against a true WNH8994, and openai/gpt-5.6-terra returns DLM8686 for proton-twoline on one of three against BLM8686. Both are wrong at a fixed string, so a retry buys nothing on luna and one in three on terra.
  • qwen/qwen3.8-flash, x-ai/grok-4.5 and x-ai/grok-4.6 were served cached prompt tokens despite cache_bust: true, 1920, 768 and 1152 respectively, so read those three latencies and costs as flattered rather than measured.

Conclusion

Eleven of the sixteen targets swept every trial, so cost and latency decide. google/gemini-3.5-flash-lite is the fastest clean row at 1,628ms for $0.0048, and qwen/qwen3.7-flash is the cheapest by a wide margin at $0.0009 but takes 3,275ms. Take flash-lite where a car is waiting at the barrier and qwen3.7-flash for batch work off a recorded feed, where the five-fold price gap matters and the extra 1.6 seconds does not. x-ai/grok-4.5 looks competitive on speed at 1,790ms, but at $0.0546 it is eleven times flash-lite for the same perfect score, and its number is one of the cache-flattered three. Everything else in the tie above a cent, including openai/gpt-5.6-sol, anthropic/claude-sonnet-5 and x-ai/grok-4.6 near $0.075, is paying for nothing this scenario detects.

The failures split three ways and only one is about seeing. qwen3.7-plus is the fixable one: a schema-repair step or a stricter decode would rescue all four of its zeros, because the correct plate is already inside the response. luna and terra describe coverage rather than reliability, one plate each that they read wrong the same way every time. haiku-4.5 is the unsafe kind, and its mean flatters it: the three clean cases hide a case where it is confidently wrong at a different string on every attempt, which is worse for a caller than a stable error it could learn to catch. kimi-k2.5 needs a working route before anyone reads a number next to it.