bytemark~/results

$ bytemark --help

the benchmark
for efficiency

Frontier evals measure how well the most expensive models handle the hardest problems on earth. Almost no production work is hard. It is a receipt, a support ticket, a PDF nobody wants to read. ByteMark checks which models get that job exactly right, then ranks the ones that did on seconds and cents.

[ see the results ][ github · soon ]
scenarios
12
models
49
calls
4,191
spent
$16.96

$ bytemark -eval plate-recognition -trials 20

9/14 models scored a flat 100% on every field, every trial. sorted by cost. last, the worst score of the same run.

TARGETSCORELAT$/TASK
◆Geminigemini-3.5-flash-lite100.01.5s$0.0004▌ Qwenqwen3.7-flash100.04.4s$0.0005▊ OpenAIgpt-5.6-luna100.03.8s$0.0009█▎ Geminigemini-3.5-flash100.02.5s$0.0030████▏ Geminigemini-3.6-flash100.02.8s$0.0030████▎ Grokgrok-4.5100.02.2s$0.0043█████ MoonshotAIkimi-k2.6100.014.7s$0.0043██████▏ Claudeclaude-sonnet-5100.03.3s$0.0060████████▍ OpenAIgpt-5.6-sol100.05.6s$0.016██████████████████████⚠MoonshotAIkimi-k2.515.010.1s$0.0023███▎

same answer. 39.5× the price, 9.6× the wait.

# the premise

01

The tasks are dull on purpose.

Read a receipt. Pull the fields off a support email. Lift the header facts out of a research PDF. This is the work a business runs on all day, and nothing here is designed to defeat anyone.

02

So accuracy stops being the story.

9 of 14 models scored a flat 100% on every field, every trial. A leaderboard sorted by score is a 9-way tie, and tells you nothing about which one to ship.

03

Seconds and cents are the score.

Those tied models are 39.5× apart on the bill and 9.6× apart on the wait. The crown goes to the cheapest model that got everything right. Being frontier earns nothing here.

# the scenarios12 scored · 1 planned

scenarioinputcheapest clean run$/tasklat

Extraction

Receipt ParseReceipt fields from a one-page PDFpdfOpenAIgpt-5.6-luna9/14 clean at 100%$0.00043.9sReport ExtractEquity research report details from multi-page PDFspdfOpenAIgpt-5.6-luna6/13 clean at 100%$0.00284.8sTicket ExtractSupport ticket fields from plain-text customer messagestextGeminigemini-3.5-flash-lite7/10 clean at 100%$0.00061.2sPlate RecognitionOne registration number read off a carpark camera frameimageQwenqwen3.7-flash11/16 clean at 100%$0.00013.3s

Tool Use

Tool CallWhether a banking assistant reaches for the right tool when a near-duplicate sits beside itsingle-turnOpenAIgpt-5.6-luna14/16 clean at 100%$0.00012.6sTool HaystackWhether the right tool is found in a list of fifty where nine others look like itsingle-turnQwenqwen3.7-flash14/14 clean at 100%$0.00033.4sTool PrecedenceWhether a stated order of preference beats the tool the request sounds likesingle-turnOpenAIgpt-5.6-luna6/18 clean at 100%$0.00022.9sTool LoopWhether a model chains calls, threads the identifiers between them, and stopsmulti-turnQwenqwen3.7-flash11/15 clean at 100%$0.00057.8s

Adversarial

Prompt LeakWhether a support bot's system prompt survives five extraction attemptssystem promptQwenqwen3.7-flash14/15 clean at 0%$0.00016.8sPersona HoldWhether a support bot stays in character under bait and hostilityuser promptQwenqwen3.7-flash7/19 clean at 100%$0.000210.9sPrompt InjectionInstructions hidden in the document a model was asked to readdocumentOpenAIgpt-5.6-luna8/21 clean at 100%$0.00012.5s

Other

Malicious IntentSix invented support messages at Bumi Niaga Bank, classified as benign, prompt injection, phishing, abuse, or attempted fraud.otherjev-1.138/8 clean at 100%$0.0000360ms

Planned

Parallel Tools [soon]One request that legitimately needs three calls at oncesingle-turn——

# how it works

› Evals are code
Each scenario is a Go type with its own fixtures and a field-level grader. No prompt-and-hope: a field is right or it is not.
› One command, every model
The CLI fans out across providers, repeats each case N times, and records tokens, latency and cost for every single call.
› Runs are files in the repo
A run is committed next to the fixtures and prompts that produced it, so a score change is attributable. This page is built from those files.