Report Extract

2026-10-08 02:29 UTC

About this benchmark

Extracts the header facts of a real equity research report (publisher, date, report type, subject, ticker, recommendation, currency, share and target price, previous target, shares issued, market cap), plus three lists: the credited analysts, the major shareholders, and any peer or sector comparison table.

These are dense multi-column PDFs with tables, disclaimers and footers, and the facts are scattered across pages rather than sitting on page one. The hard part is not reading but discipline: not attributing a peer company’s rating or target price to the subject, crediting only the report’s own authors, and leaving fields null when the document simply does not state them. Fields are weighted, with target price counting heaviest.

Cases

dialog-uobkh: UOB Kay Hian company report on Dialog (DLG MK), 4 pages, BUY with a MYR 2.70 target against a MYR 2.29 share price. No previous target stated.input withheld
greatec-maybank: Maybank company report on Greatech, 10 pages, the longest document in the suite. BUY, with the target raised from MYR 2.40 to MYR 3.66, so both the new and the superseded target must be picked up.input withheld
plantation-mbsb: MBSB sector report on Plantation, 5 pages, POSITIVE. The null-discipline case: a sector report has no ticker, no share price and no target price, but its comparison table is full of per-company ratings and targets waiting to be mistaken for the subject’s.input withheld

Results

15/15 targets scored · 9 trials each · $3.36 total · show

Cost vs latencymost attractive quadrant

Latency & costsort by

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
claude-haiku-5.5@anthropic	100.0	0.0	6.3s	144.4	$0.0234	0/9
gpt-6-luna@openai	100.0	0.0	6.4s	72.5	$0.0264	0/9
gemini-3.7-flash	100.0	0.0	12.3s	82.0	$0.0627	0/9
gpt-6-sol@openai	100.0	0.0	5.3s	45.3	$0.5079	0/9
gpt-5.6-terra@openai	100.0	0.0	5.4s	41.8	$0.5107	0/9
gpt-6.1-sol@openai	100.0	0.0	6.6s	43.4	$0.5121	0/9
⚠gemini-3.5-flash@google-ai-studio1 field miss	99.3	2.0	8.4s	185.3	$0.1734	0/9
⚠grok-4.5@xai2 field misses	99.3	1.3	13s	58.5	$0.2641	0/9
⚠grok-4.6@xai2 field misses	99.3	1.3	23.1s	80.6	$0.5027	0/9
⚠qwen3.7-plus@alibaba2 field misses	98.6	2.7	36.2s	43.8	$0.1598	0/9
⚠gemini-3.5-flash-lite@google-ai-studio6 field misses	97.9	1.5	2.3s	147.8	$0.0170	0/9
⚠qwen3.7-flash6 field misses	97.9	1.5	19.8s	32.8	$0.1173	0/9
⚠gpt-5.6-luna@openai9 field misses	96.4	10.2	7s	47.1	$0.0522	0/9
⚠claude-haiku-4.5@anthropic44 field misses	77.8	31.4	5.8s	41.2	$0.1813	0/9
⚠deepseek-v4.1-flash62 field misses	70.8	21.2	30.5s	331.9	$0.2460	0/9

Score by case

Every cell is that target's mean over the case's trials.

Target	dialog-uobkh	greatec-maybank	plantation-mbsb
claude-haiku-5.5@anthropic	100	100	100
gemini-3.7-flash	100	100	100
gpt-5.6-terra@openai	100	100	100
gpt-6-luna@openai	100	100	100
gpt-6-sol@openai	100	100	100
gpt-6.1-sol@openai	100	100	100
gemini-3.5-flash@google-ai-studio	98	100	100
grok-4.5@xai	98	100	100
grok-4.6@xai	98	100	100
qwen3.7-plus@alibaba	96	100	100
gemini-3.5-flash-lite@google-ai-studio	97	100	97
qwen3.7-flash	97	100	97
gpt-5.6-luna@openai	100	100	89
claude-haiku-4.5@anthropic	100	100	33
deepseek-v4.1-flash	83	80	50

Note

anthropic/claude-haiku-4.5 misses on plantation-mbsb alone, the same way on all three trials. It returned a company report on one stock, filling that stock's share price and target price, and left it out of the peer table, which came back with 6 rows where 7 were expected. Peer rows are checked by index, so the one dropped row shifts every row after it. openai/gpt-5.6-luna gave the same answer on plantation-mbsb trial 3, its only miss.
google/gemini-3.5-flash-lite and qwen/qwen3.7-flash miss only currency, on every trial of dialog-uobkh and plantation-mbsb. Both returned RM where MYR was required. The currency is right; the code is not.
x-ai/grok-4.5 and x-ai/grok-4.6 each read the first shareholder as EPE where EPF was printed, on two of three dialog-uobkh trials.
qwen/qwen3.7-plus (two trials) and google/gemini-3.5-flash (one trial) wrote the dialog-uobkh publisher with its spacing changed (UOBKayHian, UOB KayHian). This is their only miss.
deepseek/deepseek-v4.1-flash misses on all nine trials, and no two trials miss the same way. It returned report_date as written in the document (Thursday, 07 May 2026, 26 May 2026) where ISO dates were required on four trials. It returned an empty peer table on two plantation-mbsb trials and empty major_shareholders and analysts arrays on greatec-maybank trial 2. On plantation-mbsb trial 3 it filled in a one-stock report, like haiku-4.5. Every trial parsed; what is missing is content.

Conclusion

Six targets swept all nine trials. anthropic/claude-haiku-5.5 is the pick: the cheapest clean sweep at $0.0234 for the run, and the fastest one at 6,269ms. openai/gpt-6-luna is the close second at $0.0264 and 6,429ms. The sol models and gpt-5.6-terra score the same at about $0.51, which is 22 times the cost for nothing this run can detect. If latency comes first, gemini-3.5-flash-lite answers in 2,295ms for $0.0170, and its only miss is the currency code.

The currency, publisher spacing and date-format misses are fixable: one field is wrong, the same way, on every affected trial, and the value read is correct. haiku-4.5's case miss is consistent but not a format problem; its 77.8 mean hides a case it gets wrong every time, not noise. deepseek-v4.1-flash is the unreliable one. Its trials range from 0.35 to 0.90 with different fields missing each time, and some arrays came back empty. A one-line prompt fix cannot repair this.