Receipt Parse

2026-09-03 02:58 UTC

About this benchmark

Reads a short, clean PDF receipt and fills a fixed JSON schema: invoice and receipt numbers, date paid, vendor, bill-to name and email, currency, subtotal, discount, total, amount paid, payment method and the line-item table. The model is told to use only values printed on the page.

This is the floor of the document suite: it asks nothing of long-context reading, only whether a model can hold a whole schema at once without dropping or inventing a field. Grading is per field: exact match on identifiers and dates, case-insensitive containment on names, a numeric tolerance on money, and absolute value on the discount so a sign convention does not decide the score.

Cases

  • receipt-a: One synthetic single-page receipt from Northwind Labs billed to Jane Doe, in USD. A 50% discount line takes a $22.00 subtotal down to an $11.00 total, so a model that reads the discount as positive, or reports the subtotal as the total, is caught.

Results

13/14 targets scored · 3 trials each · $0.14 total · show

Cost vs latencymost attractive quadrant
Latency & costsort by
TargetScore ↓StddevLatencyTok/sCostErrors
bestOpenAIgpt-5.6-luna@openai100.00.03.9s54.5best$0.00110/3
Geminigemini-3.5-flash-lite@google-ai-studio100.00.0best1.6s132.2$0.00210/3
OpenAIgpt-5.6-terra@openai100.00.02.4s58.5$0.00790/3
⚠Qwenqwen3.7-plus@alibabaerrorPOST "https://openrouter.ai/api/v1/chat/completions": 400 Bad Request {"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly.","code":400,"metadata":{"provider_name":null}}100.00.036.9s55.9$0.00951/3
Claudeclaude-haiku-4.5@anthropic100.00.05.2s37.9$0.01050/3
Geminigemini-3.7-flash100.00.011s69.3$0.01100/3
Geminigemini-3.8-flash100.00.06.9s149.7$0.01400/3
Geminigemini-3.6-flash@google-ai-studio100.00.08.1s187.2$0.01830/3
Grokgrok-4.5@xai100.00.08.5s68.8$0.02080/3
Grokgrok-4.6@xai100.00.015.2s67.5$0.02890/3
⚠Metamuse-glimmer-30b2 field missesreceipt-a: date_paid (2/2)errorPOST "https://openrouter.ai/api/v1/chat/completions": 400 Bad Request {"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly.","code":400,"metadata":{"provider_name":null,"previous_errors":[{"code":400,"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly."},{"code":400,"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly."},{"code":400,"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly."}]}}93.80.014.6s92.8$0.00831/3
⚠Z.aiglm-5v-turbo@z-ai16 field missesreceipt-a: amount_paid (1/1)receipt-a: bill_to_email (1/1)receipt-a: bill_to_name (1/1)receipt-a: currency (1/1)receipt-a: date_paid (1/1)receipt-a: discount (1/1)receipt-a: invoice_number (1/1)receipt-a: line_items.0.amount (1/1)receipt-a: line_items.0.description (1/1)receipt-a: line_items.0.quantity (1/1)receipt-a: line_items.0.unit_price (1/1)receipt-a: payment_method (1/1)receipt-a: receipt_number (1/1)receipt-a: subtotal (1/1)receipt-a: total (1/1)receipt-a: vendor_name (1/1)errorPOST "https://openrouter.ai/api/v1/chat/completions": 400 Bad Request {"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly.","code":400,"metadata":{"provider_name":null}}0.00.010.1s53.1$0.00462/3
⚠Z.aiglm-4.6v@z-ai32 field missesreceipt-a: amount_paid (2/2)receipt-a: bill_to_email (2/2)receipt-a: bill_to_name (2/2)receipt-a: currency (2/2)receipt-a: date_paid (2/2)receipt-a: discount (2/2)receipt-a: invoice_number (2/2)receipt-a: line_items.0.amount (2/2)receipt-a: line_items.0.description (2/2)receipt-a: line_items.0.quantity (2/2)receipt-a: line_items.0.unit_price (2/2)receipt-a: payment_method (2/2)receipt-a: receipt_number (2/2)receipt-a: subtotal (2/2)receipt-a: total (2/2)receipt-a: vendor_name (2/2)errorPOST "https://openrouter.ai/api/v1/chat/completions": 400 Bad Request {"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly.","code":400,"metadata":{"provider_name":null}}0.00.021.8s41.9$0.00581/3
⚠Qwenqwen3.8-flasherrorPOST "https://openrouter.ai/api/v1/chat/completions": 400 Bad Request {"message":"Failed to parse the file: The document parsing engine is currently rate limited. Please retry shortly.","code":400,"metadata":{"provider_name":null}}n/an/an/an/a$0.00003/3

Note

  • OpenRouter's document parsing engine was rate limited during the run and returned 400 Failed to parse the file before any target saw the PDF. Five rows lost trials to it: qwen/qwen3.8-flash all three, z-ai/glm-5v-turbo two, and qwen/qwen3.7-plus, z-ai/glm-4.6v and meta/muse-glimmer-30b one each. Retries did not clear it. Every error on this page is that one upstream fault, not a model refusing or timing out.
  • qwen/qwen3.8-flash has no gradable output at all. Its 0.0 is three errored trials, not three wrong answers, and nothing on this page measures the model. A single call on the same slug immediately before the run parsed the document and scored every field, so read the row as unmeasured.
  • The two Z.AI zeros are schema breaks, not reading failures. glm-4.6v and glm-5v-turbo both recovered the receipt correctly and then returned it under their own key layout: bill_to.name and customer.name where bill_to_name was asked for, from.company and vendor.name where vendor_name was, items and discounts where line_items and discount were. The values in those objects are right, including the receipt and invoice numbers, the email and every amount. All 16 fields miss on path, so the row reads as a model that cannot see the document when it is a model that will not fill the shape.
  • meta/muse-glimmer-30b's only miss is date_paid, on both trials that landed. It returned June 3, 2026 where 2026-06-03 was required. The date is correct; the format is not.
  • x-ai/grok-4.5, x-ai/grok-4.6, meta/muse-glimmer-30b, z-ai/glm-4.6v and z-ai/glm-5v-turbo were served cached prompt tokens despite cache busting, 256 and 256 for the two grok rows and 96, 11 and 2 for the rest, so read those latencies and costs as slightly flattered.
  • git_sha is 0e9e555-dirty. Untracked run files were in the tree when the binary was built, so the stamp records the commit but the tree does not match it. No source under cmd/ or internal/ differs from 0e9e555.

Conclusion

Ten of the fourteen targets swept every trial, so the pick falls to cost and speed. google/gemini-3.5-flash-lite takes both ends of it at 1,612ms and $0.0021, the fastest row on the page and within a fifth of a cent of the cheapest. openai/gpt-5.6-luna is cheaper still at $0.0011 but takes 3,929ms, so it is the choice only where nobody is waiting on the reply. Everything above $0.01 for the same clean sweep, including both grok rows at $0.0208 and $0.0289, is paying for nothing this scenario can detect.

The failures split three ways and only one of them is about capability. muse-glimmer's date is fixable in the prompt: the model read the receipt and formatted the answer its own way. The Z.AI rows are the unsafe kind, and their zeros understate rather than overstate them, because a consumer that keys off the requested field names gets nothing back from a response that actually contains every correct value. The rate-limited trials are not a model signal at all, and qwen/qwen3.8-flash needs a re-run before anyone reads a number next to it.