Report Extract

2026-08-14 05:21 UTC

About this benchmark

Extracts the header facts of a real equity research report (publisher, date, report type, subject, ticker, recommendation, currency, share and target price, previous target, shares issued, market cap), plus three lists: the credited analysts, the major shareholders, and any peer or sector comparison table.

These are dense multi-column PDFs with tables, disclaimers and footers, and the facts are scattered across pages rather than sitting on page one. The hard part is not reading but discipline: not attributing a peer company’s rating or target price to the subject, crediting only the report’s own authors, and leaving fields null when the document simply does not state them. Fields are weighted, with target price counting heaviest.

Cases

dialog-uobkh: UOB Kay Hian company report on Dialog (DLG MK), 4 pages, BUY with a MYR 2.70 target against a MYR 2.29 share price. No previous target stated.input withheld
greatec-maybank: Maybank company report on Greatech, 10 pages, the longest document in the suite. BUY, with the target raised from MYR 2.40 to MYR 3.66, so both the new and the superseded target must be picked up.input withheld
plantation-mbsb: MBSB sector report on Plantation, 5 pages, POSITIVE. The null-discipline case: a sector report has no ticker, no share price and no target price, but its comparison table is full of per-company ratings and targets waiting to be mistaken for the subject’s.input withheld

Results

13/13 targets scored · 9 trials each · $2.26 total · show

Cost vs latencymost attractive quadrant

Latency & costsort by

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
gpt-5.6-luna@openai	100.0	0.0	4.8s	71.3	$0.0254	0/9
gemini-3.7-flash	100.0	0.0	6.8s	139.3	$0.0311	0/9
gemini-3.6-flash@google-ai-studio	100.0	0.0	7.3s	174.7	$0.0667	0/9
gemini-3.5-flash@google-ai-studio	100.0	0.0	11.4s	143.6	$0.1789	0/9
gpt-5.6-terra@openai	100.0	0.0	4.1s	56.9	$0.2479	0/9
grok-4.5@xai	100.0	0.0	10.7s	62.8	$0.2597	0/9
⚠grok-4.6@xai3 field misses	98.9	1.5	28.8s	71.6	$0.4875	0/9
⚠gemini-3.5-flash-lite@google-ai-studio5 field misses	98.3	1.5	2.1s	132.8	$0.0156	0/9
⚠qwen3.7-plus@alibaba8 field misses	95.9	4.6	31.9s	45.6	$0.1583	0/9
⚠muse-glimmer-30b30 field misses	85.3	8.2	57.9s	52.0	$0.1675	0/9
⚠claude-haiku-4.5@anthropic44 field misses	77.7	30.7	4.6s	49.2	$0.1807	0/9
⚠glm-4.6v@z-ai143 field misses	27.9	8.1	211s	36.5	$0.1978	0/9
⚠glm-5v-turbo@z-ai126 field misseserror	17.5	13.5	32.7s	82.1	$0.2446	2/9

Score by case

Every cell is that target's mean over the case's trials, and ✕ marks a reply that was never a document, whatever it scored.

Target	dialog-uobkh	greatec-maybank	plantation-mbsb
gemini-3.5-flash@google-ai-studio	100	100	100
gemini-3.6-flash@google-ai-studio	100	100	100
gemini-3.7-flash	100	100	100
gpt-5.6-luna@openai	100	100	100
gpt-5.6-terra@openai	100	100	100
grok-4.5@xai	100	100	100
grok-4.6@xai	97	100	100
gemini-3.5-flash-lite@google-ai-studio	98	100	97
qwen3.7-plus@alibaba	92	100	95
muse-glimmer-30b	87	90	78
claude-haiku-4.5@anthropic	100	99	34
glm-4.6v@z-ai	26	23	35
glm-5v-turbo@z-ai	19	4	35

Note

z-ai/glm-4.6v and z-ai/glm-5v-turbo scored on shape rather than on reading. Both answered with their own key names and nesting, report_metadata wrapping source and date in one, analyst, rating and a target_price object carrying value and currency in the other, in place of the schema's flat fields. The values underneath are the ones printed on the page, UOB Kay Hian, DLG MK, a RM2.70 target, so 27.9 and 17.5 say nothing about what either target could read off these documents.
z-ai/glm-5v-turbo lost 2 of its 9 trials to a 400 from the provider, Failed to parse doc.pdf, on dialog-uobkh and plantation-mbsb. Errored trials are excluded from the mean, so its score rests on 7 trials and two of its three cases on 2 apiece.
Every currency miss in the run is the same one. google/gemini-3.5-flash-lite, qwen/qwen3.7-plus and meta/muse-glimmer-30b wrote RM where the schema asks for the ISO code, 12 trials between them, and for flash-lite that single substitution is its entire deficit.
anthropic/claude-haiku-4.5's 77.7 is one case collapsing, not a weakness spread across the run: 100.0 on dialog-uobkh, 98.9 on greatec-maybank, 34.3 on plantation-mbsb, and the low case fails the same way on all three trials. Its output holds the schema shape throughout. It names a company in subject, sets report_type to company, and fills share_price 6.03 and target_price 7.00, which is the failure the 30.7 stddev is pointing at.
x-ai/grok-4.6's whole deficit is one character. It wrote EPE for the first major shareholder on dialog-uobkh on all three trials, and lost nothing else anywhere in the run.
x-ai/grok-4.6 was served 99,456 cached prompt tokens despite cache_bust being on, against 1,152 for x-ai/grok-4.5 and double digits for the two z-ai targets. Its cost and latency are the highest in the run with that discount already applied.

Conclusion

Six targets lost no field on any trial, so the pick falls to cost and latency. openai/gpt-5.6-luna takes it at $0.0254 for the run at 4812ms, roughly a tenth of the $0.2597 and $0.2479 that x-ai/grok-4.5 and openai/gpt-5.6-terra spend for the same perfect score. google/gemini-3.7-flash is the alternative at $0.0311 and 6811ms if throughput matters, at 139 tok/s against luna's 71. Below the clean group, google/gemini-3.5-flash-lite is the interesting one: 98.3 at $0.0156 and 2093ms, the cheapest and by far the fastest target in the run, and its only loss is a currency string.

The failures split three ways. Flash-lite, qwen and muse-glimmer are fixable by a prompt that names the currency format, and for flash-lite that alone would close the gap to the leaders. Haiku is the unsafe read: its mean of 77.7 is two clean cases averaged with one that fails identically every time, and a pipeline sampling one case would see either 100 or 34 with nothing in between to warn it. The two z-ai targets are unusable rather than weak, because a consumer keying off score alone will read them as unable to extract from these PDFs when what they cannot do is return the requested shape.