Ticket Extract

2026-10-08 02:45 UTC

About this benchmark

Turns a customer support email into a structured ticket: ticket id, customer name and email, product, severity in the customer’s own words, the date the problem started, order number, requested action, and one entry per affected device with serial and model.

The input is plain text pasted into the prompt, so nothing here depends on PDF handling; it isolates comprehension and instruction-following. Each message carries a deliberate trap: numbers that look like the one being asked for, a date the customer corrects mid-thread, and a device list whose length is stated in prose rather than tabulated.

Cases

  • ticket-a: Two Meridian G4 gateways, urgent, RMA requested. The sender opens by correcting the start date (the fault began 2 June, not the 6th) and an invoice number sits right beside the order number.
  • ticket-b: One Beacon M2 sensor, low priority, replacement requested. The unit was a free sample, so there is no order number and the field must stay empty rather than borrow the ticket reference.
  • ticket-c: Five Halyard R1 controllers bricked by a firmware rollout, high severity, rollback requested. Serials are split across two sentences and a sixth serial is named only to be excluded, so the expected device count is five.

Results

12/13 targets scored · 9 trials each · $0.20 total · show

Cost vs latencymost attractive quadrant
Latency & costsort by
TargetScore ↓StddevLatencyTok/sCostErrors
OpenAIgpt-6-luna@openai100.00.03.3s74.9best$0.00150/9
OpenAIgpt-5.6-luna@openai100.00.02.6s62.3$0.00260/9
bestGeminigemini-3.5-flash-lite@google-ai-studio100.00.0best1.4s150.6$0.00570/9
Claudeclaude-haiku-4.5@anthropic100.00.04.8s30.2$0.01450/9
OpenAIgpt-6.1-sol@openai100.00.03.6s40.8$0.02210/9
OpenAIgpt-5.6-terra@openai100.00.02.1s62.6$0.02300/9
OpenAIgpt-6-sol@openai100.00.03.6s52.9$0.02610/9
Geminigemini-3.8-flash@google-ai-studio100.00.06.5s113.7$0.02720/9
Grokgrok-4.5@xai100.00.06.6s64.3$0.03560/9
⚠Claudeclaude-haiku-5.5@anthropic8 field missesticket-c: affected_devices (2/3)ticket-c: affected_devices.0.model (2/3)ticket-c: affected_devices.0.serial (2/3)ticket-c: affected_devices.4.serial (2/3)92.613.93.2s122.8$0.00280/9
⚠Qwenqwen3.7-plus@alibaba9 field missesticket-a: reported_at (3/3)ticket-b: reported_at (3/3)ticket-c: reported_at (3/3)90.90.744.3s57.4$0.03020/9
⚠Qwenqwen3.7-flash@alibaba14 field missesticket-a: product (3/3)ticket-a: reported_at (3/3)ticket-b: product (2/3)ticket-b: reported_at (3/3)ticket-c: reported_at (3/3)85.65.135.2s90.0$0.00380/9
⚠MoonshotAIkimi-k2.5@moonshotaierrorPOST "https://openrouter.ai/api/v1/chat/completions": 404 Not Found {"message":"No endpoints found for moonshotai/kimi-k2.5. Every candidate endpoint was removed during routing: Filter by Parameters removed amazon-bedrock/us-east-2; Filter by Fallback removed siliconflow/int4, atlas-cloud/int4, venice, novita.","code":404,"metadata":{"routing_funnel":[{"step":"Initial Endpoints","endpoint_count":5},{"step":"Filter by Parameters","endpoint_count":4},{"step":"Filter by Fallback","endpoint_count":0}],"failed_routing_step":"Filter by Fallback"}}n/an/an/an/a$0.00009/9

Score by case

Every cell is that target's mean over the case's trials.

Targetticket-aticket-bticket-c
Claudeclaude-haiku-4.5@anthropic100100100
Geminigemini-3.5-flash-lite@google-ai-studio100100100
Geminigemini-3.8-flash@google-ai-studio100100100
OpenAIgpt-5.6-luna@openai100100100
OpenAIgpt-5.6-terra@openai100100100
OpenAIgpt-6-luna@openai100100100
OpenAIgpt-6-sol@openai100100100
OpenAIgpt-6.1-sol@openai100100100
Grokgrok-4.5@xai100100100
Claudeclaude-haiku-5.5@anthropic10010078
Qwenqwen3.7-plus@alibaba919092
Qwenqwen3.7-flash@alibaba828392
MoonshotAIkimi-k2.5@moonshotaierrerrerr

Note

  • git_sha reads fdd9ef5-dirty. The binary was built with uncommitted changes to internal/model/openrouter that route four more slug prefixes to the decisions endpoint. No target in this run matches those prefixes, so every call took the same chat path as a clean build of fdd9ef5, and the scores are what that commit produces.
  • moonshotai/kimi-k2.5 produced no gradable output. All 9 calls returned 404 No endpoints found: OpenRouter's routing funnel went from five endpoints to zero, with the last four removed at the fallback filter. Its 0.0 is 9 routing failures, not 9 wrong extractions.
  • anthropic/claude-haiku-5.5 returned an empty affected_devices array on ticket-c trials 2 and 3. Trial 1 listed all five devices. Every other field on those trials is correct, so the output is well-formed and gives no sign that a list is missing.
  • qwen/qwen3.7-plus misses reported_at on all 9 trials and nothing else. Every trial returned a date that is not the expected one, written as ISO on 5 trials and as Tue, 9 June 2026 14:22 style text on 4.
  • qwen/qwen3.7-flash misses reported_at on all 9 trials as well. On 7 the date itself is wrong; on 2 (ticket-a trial 3, ticket-b trial 3) the date is right but written as 2 June 2026 instead of ISO. It also shortened product to a generic noun (Gateway, sensor) on 5 trials, dropping the model name.

Conclusion

Nine targets extracted every field on all 9 trials, so cost and speed decide. openai/gpt-6-luna is the pick: the cheapest clean sweep at $0.0015 for the run, at 3,276ms. openai/gpt-5.6-luna costs $0.0026 and answers faster at 2,617ms. If latency comes first, gemini-3.5-flash-lite is the fastest clean sweep at 1,440ms for $0.0057. grok-4.5, gemini-3.8-flash, gpt-6-sol, gpt-6.1-sol and gpt-5.6-terra get the same result for $0.022 to $0.036.

The qwen misses are consistent but not cleanly fixable: reported_at fails on every trial, and the value is mostly the wrong date, not a format slip, so a date normaliser in the caller would recover only 2 of 18 trials. claude-haiku-5.5 is the unsafe one despite its 92.6: two of three ticket-c trials silently dropped a whole list, and the mean hides that the same input gave a full answer once and an empty one twice.