bytemark~/results

$ bytemark --help

the benchmark
for efficiency

Frontier evals measure how well the most expensive models handle the hardest problems on earth. Almost no production work is hard. It is a receipt, a support ticket, a PDF nobody wants to read. ByteMark checks which models get that job exactly right, then ranks the ones that did on seconds and cents.

[ see the results ][ github · soon ]

scenarios: 14
models: 67
calls: 10,778
spent: $39.35

$ bytemark -eval plate-recognition -trials 20

9/14 models scored a flat 100% on every field, every trial. sorted by cost. last, the worst score of the same run.

TARGETSCORELAT$/TASK

◆gemini-3.5-flash-lite100.01.5s$0.0004▌ qwen3.7-flash100.04.4s$0.0005▊ gpt-5.6-luna100.03.8s$0.0009█▎ gemini-3.5-flash100.02.5s$0.0030████▏ gemini-3.6-flash100.02.8s$0.0030████▎ grok-4.5100.02.2s$0.0043█████ kimi-k2.6100.014.7s$0.0043██████▏ claude-sonnet-5100.03.3s$0.0060████████▍ gpt-5.6-sol100.05.6s$0.016██████████████████████⚠kimi-k2.515.010.1s$0.0023███▎

same answer. 39.5× the price, 9.6× the wait.

# the premise

01

The tasks are dull on purpose.

Read a receipt. Pull the fields off a support email. Lift the header facts out of a research PDF. This is the work a business runs on all day, and nothing here is designed to defeat anyone.

02

So accuracy stops being the story.

9 of 14 models scored a flat 100% on every field, every trial. A leaderboard sorted by score is a 9-way tie, and tells you nothing about which one to ship.

03

Seconds and cents are the score.

Those tied models are 39.5× apart on the bill and 9.6× apart on the wait. The crown goes to the model that got everything right with the lowest cost × latency², so speed weighs double. Being frontier earns nothing here.

# the scenarios14 scored · 1 planned

scenarioinputcrowned · lowest $ × lat²$/tasklat

Extraction

Receipt ParseReceipt fields from a one-page PDFpdfclaude-haiku-5.512/15 clean at 100%$0.00042.5s Report ExtractEquity research report details from multi-page PDFspdfclaude-haiku-5.56/15 clean at 100%$0.00266.3s Ticket ExtractSupport ticket fields from plain-text customer messagestextgemini-3.5-flash-lite9/13 clean at 100%$0.00061.4s Plate RecognitionOne registration number read off a carpark camera frameimageqwen3.7-flash13/18 clean at 100%$0.00013s

Tool Use

Tool CallWhether a banking assistant reaches for the right tool when a near-duplicate sits beside itsingle-turngpt-6-luna11/14 clean at 100%$0.00012.3s Tool HaystackWhether the right tool is found in a list of fifty where nine others look like itsingle-turngemini-3.5-flash-lite14/15 clean at 100%$0.00211.2s Tool PrecedenceWhether a stated order of preference beats the tool the request sounds likesingle-turngpt-6-luna9/17 clean at 100%$0.00023s Tool LoopWhether a model chains calls, threads the identifiers between them, and stopsmulti-turndeepseek-v4.1-flash12/15 clean at 100%$0.00153.1s

Adversarial

Prompt LeakWhether a support bot's system prompt survives five extraction attemptssystem promptgemini-3.5-flash-lite15/15 clean at 0%$0.00021s Persona HoldWhether a support bot stays in character under bait and hostilityuser promptgpt-6-sol5/16 clean at 100%$0.00213.8s Prompt InjectionInstructions hidden in the document a model was asked to readdocumentgpt-6-luna10/20 clean at 100%$0.00012.8s

Classification

Plate VerifyWhether a model confirms or rejects an OCR plate reading against the photoimageclaude-haiku-5.54/13 clean at 100%$0.00024.3s Guideline LintWhich writing rule each bullet on a bot's guidelines page breaks, if anysingle labeljev-1.1310/14 clean at 100%$0.0000508ms Malicious IntentSix invented support messages at Bumi Niaga Bank, classified as benign, prompt injection, phishing, abuse, or attempted fraud.single labeljev-1.1312/14 clean at 100%$0.0000453ms

Planned

Parallel Tools [soon]One request that legitimately needs three calls at oncesingle-turn——

# how it works

› Evals are code: Each scenario is a Go type with its own fixtures and a field-level grader. No prompt-and-hope: a field is right or it is not.
› One command, every model: The CLI fans out across providers, repeats each case N times, and records tokens, latency and cost for every single call.
› Runs are files in the repo: A run is committed next to the fixtures and prompts that produced it, so a score change is attributable. This page is built from those files.