Receipt Parse
About this benchmark
Reads a short, clean PDF receipt and fills a fixed JSON schema: invoice and receipt numbers, date paid, vendor, bill-to name and email, currency, subtotal, discount, total, amount paid, payment method and the line-item table. The model is told to use only values printed on the page.
This is the floor of the document suite: it asks nothing of long-context reading, only whether a model can hold a whole schema at once without dropping or inventing a field. Grading is per field: exact match on identifiers and dates, case-insensitive containment on names, a numeric tolerance on money, and absolute value on the discount so a sign convention does not decide the score.
Cases
receipt-a: One synthetic single-page receipt from Northwind Labs billed to Jane Doe, in USD. A 50% discount line takes a $22.00 subtotal down to an $11.00 total, so a model that reads the discount as positive, or reports the subtotal as the total, is caught.
Results
| Target | Score ↓ | Stddev | Latency | Tok/s | Cost | Errors |
|---|---|---|---|---|---|---|
| gpt-6-luna@openai | 100.0 | 0.0 | 3.6s | 72.9 | $0.0006 | 0/3 |
| gpt-5.6-luna@openai | 100.0 | 0.0 | 3.7s | 53.5 | $0.0010 | 0/3 |
| claude-haiku-5.5@anthropic | 100.0 | 0.0 | 2.5s | 89.2 | $0.0012 | 0/3 |
| gemini-3.5-flash-lite@google-ai-studio | 100.0 | 0.0 | 2s | 131.6 | $0.0025 | 0/3 |
| gpt-6.1-sol@openai | 100.0 | 0.0 | 4.2s | 38.8 | $0.0079 | 0/3 |
| gpt-5.6-terra@openai | 100.0 | 0.0 | 2.1s | 66.9 | $0.0082 | 0/3 |
| gpt-6-sol@openai | 100.0 | 0.0 | 4.8s | 43.7 | $0.0094 | 0/3 |
| claude-haiku-4.5@anthropic | 100.0 | 0.0 | 6s | 31.1 | $0.0104 | 0/3 |
| gemini-3.7-flash | 100.0 | 0.0 | 7.6s | 122.8 | $0.0129 | 0/3 |
| gemini-3.8-flash | 100.0 | 0.0 | 11.5s | 98.4 | $0.0151 | 0/3 |
| grok-4.5@xai | 100.0 | 0.0 | 6.7s | 64.7 | $0.0179 | 0/3 |
| grok-4.6@xai | 100.0 | 0.0 | 12.5s | 111.9 | $0.0354 | 0/3 |
| ⚠qwen3.8-flash3 field missesreceipt-a: date_paid (3/3) | 93.8 | 0.0 | 112.7s | 50.2 | $0.0142 | 0/3 |
| ⚠qwen3.7-plus@alibaba3 field missesreceipt-a: date_paid (3/3) | 93.8 | 0.0 | 47s | 55.3 | $0.0164 | 0/3 |
| ⚠deepseek-v4.1-flash14 field missesreceipt-a: bill_to_email (1/3)receipt-a: bill_to_name (1/3)receipt-a: date_paid (2/3)receipt-a: discount (1/3)receipt-a: invoice_number (1/3)receipt-a: line_items.0.amount (1/3)receipt-a: line_items.0.description (1/3)receipt-a: line_items.0.unit_price (1/3)receipt-a: payment_method (1/3)receipt-a: receipt_number (1/3)receipt-a: subtotal (1/3)receipt-a: total (1/3)receipt-a: vendor_name (1/3) | 70.8 | 36.9 | 13.4s | 262.0 | $0.0190 | 0/3 |
Note
qwen/qwen3.7-plusandqwen/qwen3.8-flashmiss onlydate_paid, on all three trials each. Both returnedJune 3, 2026where2026-06-03was required. The date is correct; the format is not.deepseek/deepseek-v4.1-flashtrial 2 returned a receipt that is not in the document: a different vendor (Acme Corp.), customer, invoice number, two line items and an emptydate_paid. Its schema was intact, so the output parsed cleanly and scored 0.19 on the few fields it happened to match. Trial 3 read the right receipt and missed onlydate_paidon the same format.
Conclusion
Twelve targets swept all three trials, so the pick falls to cost and speed.
openai/gpt-6-luna is the cheapest at $0.00055 for the run, at 3,638ms.
anthropic/claude-haiku-5.5 costs about twice that ($0.00116) and answers
faster at 2,505ms; google/gemini-3.5-flash-lite is the fastest clean sweep at
1,969ms for $0.00245. The sol models and everything above $0.008 buy nothing
this scenario can detect.
Both qwen rows are fixable: one field, wrong format, every trial, with the
values read correctly. deepseek-v4.1-flash is unsafe. Its 70.8 mean averages
one clean trial, one date-format miss and one invented receipt, and the invented
one is well-formed JSON a consumer would accept without complaint.