Plate Recognition
About this benchmark
Reads the plate off the vehicle in a 1920x1015 barrier camera still and returns it as one string. One field, so every trial scores 0 or 1 and the mean across four cases and five trials is a plain success rate.
The frames are real captures from a Malaysian multi-storey carpark: sun glare on a bonnet, a plate recessed behind a grille with dealer text under it, a two-line plate on a dirty bumper, and a small plate on an unlit level. Malaysian plates carry no state colour coding or hyphens, so there is no format prior to lean on when the pixels are ambiguous.
Each fixture is cropped along the bottom to drop a burnt-in camera overlay that printed the plate in clean text, so the plate can only be read from the plate. Faces and a passing motorcyclist are pixelated in the bytes the models are sent.
Cases
bmw-grille: BMW iX at a barrier, VJP 9981, with a dealer strip under the digits inside the plate frame.honda-glare: Honda City on a sunlit rooftop level, VJF 5591, the plate shadowed under a hard windscreen flare.kancil-dim: Perodua Kancil on an unlit level, WNH 8994, on a slanted decorative plate face that turns the leading W into an M.proton-twoline: Proton Saga at an entry lane, BLM 8686, on a two-line plate with letters above digits.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| qwen3.7-flash@alibaba | 100.0 | 0.0 | 3.3s | 24.4 | $0.0009 | 0/12 |
| qwen3.8-flash@alibaba | 100.0 | 0.0 | 6.3s | 15.9 | $0.0042 | 0/12 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 1.6s | 10.3 | $0.0048 | 0/12 |
| gpt-5-mini@openai | 100.0 | 0.0 | 3.3s | 34.5 | $0.0101 | 0/12 |
| gemini-3.8-flash@google-ai-studio | 100.0 | 0.0 | 3.3s | 36.9 | $0.0162 | 0/12 |
| gemini-3.6-flash@google-ai-studio | 100.0 | 0.0 | 2.7s | 69.2 | $0.0190 | 0/12 |
| gpt-5.4-mini@openai | 100.0 | 0.0 | 1.9s | 8.8 | $0.0229 | 0/12 |
| grok-4.5@xai | 100.0 | 0.0 | 1.8s | 6.1 | $0.0546 | 0/12 |
| claude-sonnet-5@anthropic | 100.0 | 0.0 | 3.7s | 4.6 | $0.0721 | 0/12 |
| grok-4.6@xai | 100.0 | 0.0 | 6.1s | 48.4 | $0.0745 | 0/12 |
| gpt-5.6-sol@openai | 100.0 | 0.0 | 2.2s | 7.7 | $0.0754 | 0/12 |
| ⚠gpt-5.6-terra@openai1 field missproton-twoline: plate_number (1/3) | 91.7 | 27.6 | 1.9s | 9.1 | $0.0758 | 0/12 |
| ⚠gpt-5.6-luna@openai3 field misseskancil-dim: plate_number (3/3) | 75.0 | 43.3 | 2.4s | 13.7 | $0.0078 | 0/12 |
| ⚠claude-haiku-4.5@anthropic3 field missesproton-twoline: plate_number (3/3) | 75.0 | 43.3 | 2.3s | 6.0 | $0.0229 | 0/12 |
| ⚠qwen3.7-plus@alibaba4 field misseshonda-glare: plate_number (2/3)kancil-dim: plate_number (1/3)proton-twoline: plate_number (1/3) | 66.7 | 47.1 | 5.3s | 33.7 | $0.0109 | 0/12 |
| ⚠kimi-k2.5@moonshotaierrorPOST "https://openrouter.ai/api/v1/chat/completions": 404 Not Found {"message":"No endpoints found for moonshotai/kimi-k2.5.","code":404,"metadata":{"routing_funnel":[{"step":"Initial Endpoints","endpoint_count":8},{"step":"Filter by Image Support","endpoint_count":7},{"step":"Filter by Parameters","endpoint_count":4},{"step":"Apply Status Sorting","endpoint_count":4},{"step":"Filter by Fallback","endpoint_count":0}]}} | n/a | n/a | n/a | n/a | $0.0000 | 12/12 |
Score by case
| Target | bmw-grille | honda-glare | kancil-dim | proton-twoline |
|---|---|---|---|---|
| claude-sonnet-5@anthropic | 100 | 100 | 100 | 100 |
| gemini-3.5-flash-lite@google-ai-studio | 100 | 100 | 100 | 100 |
| gemini-3.6-flash@google-ai-studio | 100 | 100 | 100 | 100 |
| gemini-3.8-flash@google-ai-studio | 100 | 100 | 100 | 100 |
| gpt-5-mini@openai | 100 | 100 | 100 | 100 |
| gpt-5.4-mini@openai | 100 | 100 | 100 | 100 |
| gpt-5.6-sol@openai | 100 | 100 | 100 | 100 |
| qwen3.7-flash@alibaba | 100 | 100 | 100 | 100 |
| qwen3.8-flash@alibaba | 100 | 100 | 100 | 100 |
| grok-4.5@xai | 100 | 100 | 100 | 100 |
| grok-4.6@xai | 100 | 100 | 100 | 100 |
| gpt-5.6-terra@openai | 100 | 100 | 100 | 67 |
| claude-haiku-4.5@anthropic | 100 | 100 | 100 | 0 |
| gpt-5.6-luna@openai | 100 | 100 | 0 | 100 |
| qwen3.7-plus@alibaba | 100 | 33 | 67 | 67 |
| kimi-k2.5@moonshotai | err | err | err | err |
Note
moonshotai/kimi-k2.5produced no gradable output. Every call returned404 No endpoints found for moonshotai/kimi-k2.5, and the routing funnel in the error shows why: eight endpoints, seven after the image-support filter, four after parameters, then zero once the fallback filter ran. Retries did not clear it. Its 0.0 is twelve routing failures, not twelve wrong plates, so nothing on this page measures the model.qwen/qwen3.7-plusread all four plates correctly on all twelve trials. Its four zeros are the envelope: it returned[{"plate_number": "VJF5591"}], a one-element array, where the bare object was asked for. The same slug returns the bare object on the other eight trials, including trials of the same case, so the score measures how often it settles on the requested shape and not whether it can read anything.anthropic/claude-haiku-4.5missesproton-twolineon all three trials with three different answers,PLM65S,PLM9505andPLM9595against a trueBLM8686. The leadingBcomes back asPevery time while the digits move, so repeated calls would not converge on the right plate.openai/gpt-5.6-lunareturnsMNH8994forkancil-dimon all three trials against a trueWNH8994, andopenai/gpt-5.6-terrareturnsDLM8686forproton-twolineon one of three againstBLM8686. Both are wrong at a fixed string, so a retry buys nothing on luna and one in three on terra.qwen/qwen3.8-flash,x-ai/grok-4.5andx-ai/grok-4.6were served cached prompt tokens despitecache_bust: true, 1920, 768 and 1152 respectively, so read those three latencies and costs as flattered rather than measured.
Conclusion
Eleven of the sixteen targets swept every trial, so cost and latency decide.
google/gemini-3.5-flash-lite is the fastest clean row at 1,628ms for $0.0048,
and qwen/qwen3.7-flash is the cheapest by a wide margin at $0.0009 but takes
3,275ms. Take flash-lite where a car is waiting at the barrier and qwen3.7-flash
for batch work off a recorded feed, where the five-fold price gap matters and
the extra 1.6 seconds does not. x-ai/grok-4.5 looks competitive on speed at
1,790ms, but at $0.0546 it is eleven times flash-lite for the same perfect
score, and its number is one of the cache-flattered three. Everything else in
the tie above a cent, including openai/gpt-5.6-sol, anthropic/claude-sonnet-5
and x-ai/grok-4.6 near $0.075, is paying for nothing this scenario detects.
The failures split three ways and only one is about seeing. qwen3.7-plus is
the fixable one: a schema-repair step or a stricter decode would rescue all four
of its zeros, because the correct plate is already inside the response. luna
and terra describe coverage rather than reliability, one plate each that they
read wrong the same way every time. haiku-4.5 is the unsafe kind, and its
mean flatters it: the three clean cases hide a case where it is confidently
wrong at a different string on every attempt, which is worse for a caller than
a stable error it could learn to catch. kimi-k2.5 needs a working route
before anyone reads a number next to it.