Guideline Lint

2026-10-08 02:51 UTC

About this benchmark

Staff write a page of bullets that Bumi Niaga Bank's support bot reads on every chat. A linter checks it one bullet at a time, so each case is a single bullet and its section heading, not the page. The model names the one writing rule it breaks, or `none`: addressed to staff, too long, only prohibits, restates the base prompt, or holds a credential or personal data. Exact label scores one.

Five controls break nothing but look as if they do: a bullet naming a team, two related steps in one bullet, a "never" with an action attached, a specific apology, and a public hotline. A repeated bullet needs the whole page to spot, so that rule is left out. The grade ignores confidence, so a hedged right answer scores as a sure one.

Cases

chained-steps: Seven loosely related steps chained into one bullet.
customer-nric: A named customer and her NRIC number; no password to spot.
named-team: Control: names a team, but the instruction is to the bot.
never-then-do: Control: opens with "never" but gives the action to take.
only-avoid: Three prohibitions and nothing for the bot to do instead.
portal-password: A back-office login and password inside a bullet every chat sees.
public-hotline: Control: a public business number is not personal data.
related-pair: Control: two steps, but closely related actions may share a bullet.
restates-base: Tone and no-fabrication, which the base prompt already covers.
specific-apology: Control: about tone, but specific to delayed transfers and tied to a tool.
staff-voice: Addresses support agents about the bot instead of instructing the bot.

Results

14/14 targets scored · 33 trials each · $0.10 total · show

Cost vs latencymost attractive quadrant

Latency & costsort by

Target	Score ↓	Stddev	Latency	Tok/s	Cost	Errors
pplx-decider-v1.1-27b@perplexity	100.0	0.0	952ms	1.0	$0.0003	0/33
jev-1.13@typesafe	100.0	0.0	508ms	131.2	$0.0010	0/33
solar-decide@upstage	100.0	0.0	8.3s	0.1	$0.0012	0/33
gpt-6-luna@openai	100.0	0.0	2.4s	19.8	$0.0022	0/33
claude-haiku-5.5@anthropic	100.0	0.0	1.6s	8.2	$0.0029	0/33
gpt-5.6-luna@openai	100.0	0.0	1.3s	10.6	$0.0034	0/33
qwen3.7-flash@alibaba	100.0	0.0	12.1s	74.4	$0.0043	0/33
⚠qwen3.8-flash@alibabaerror	100.0	0.0	15.6s	13.8	$0.0049	4/33
deepseek-v4.1-flash	100.0	0.0	1.2s	129.0	$0.0100	0/33
claude-haiku-4.5@anthropic	100.0	0.0	1.3s	7.5	$0.0215	0/33
gemini-3.8-flash@google-ai-studio	100.0	0.0	4s	53.5	$0.0364	0/33
⚠gemini-3.5-flash-lite@google-ai-studio	97.0	17.1	1.1s	9.3	$0.0048	0/33
⚠gpt-6-luna-decisions@openai	81.8	38.6	565ms	0.0	$0.0016	0/33
⚠solar-decide-flash@upstage	39.4	48.9	736ms	1.4	$0.0012	0/33

Score by case

Every cell is that target's mean over the case's trials.

Target	chained-steps	customer-nric	named-team	never-then-do	only-avoid	portal-password	public-hotline	related-pair	restates-base	specific-apology	staff-voice
claude-haiku-4.5@anthropic	100	100	100	100	100	100	100	100	100	100	100
claude-haiku-5.5@anthropic	100	100	100	100	100	100	100	100	100	100	100
deepseek-v4.1-flash	100	100	100	100	100	100	100	100	100	100	100
gemini-3.8-flash@google-ai-studio	100	100	100	100	100	100	100	100	100	100	100
gpt-5.6-luna@openai	100	100	100	100	100	100	100	100	100	100	100
gpt-6-luna@openai	100	100	100	100	100	100	100	100	100	100	100
pplx-decider-v1.1-27b@perplexity	100	100	100	100	100	100	100	100	100	100	100
qwen3.7-flash@alibaba	100	100	100	100	100	100	100	100	100	100	100
qwen3.8-flash@alibaba	100	100	100	100	100	100	100	100	100	100	100
jev-1.13@typesafe	100	100	100	100	100	100	100	100	100	100	100
solar-decide@upstage	100	100	100	100	100	100	100	100	100	100	100
gemini-3.5-flash-lite@google-ai-studio	100	100	100	67	100	100	100	100	100	100	100
gpt-6-luna-decisions@openai	100	100	100	100	0	100	100	100	0	100	100
solar-decide-flash@upstage	33	100	0	0	100	100	0	0	0	0	100

Note

git_sha reads d6e778b-dirty, but no tracked file differed from d6e778b at build time. The only change in the tree was an untracked run file from another scenario's run, which Go's VCS stamp counts as modified. d6e778b is the commit that routes the Perplexity, Upstage and OpenAI decision slugs to the decisions endpoint; without it those four targets would have gone to chat completions.
upstage/solar-decide-flash@upstage never returned none in 33 trials. All 15 trials of the five none cases came back flagged: 10 as staff_voice, 3 as too_long, 2 as credentials. It also labelled restates-base as staff_voice on all three trials. Its winning probability on those misses sits between 0.44 and 0.76.
openai/gpt-6-luna-decisions@openai misses the same two cases on every trial, both as none. On only-avoid it is a near tie: none 0.37 to 0.43 against only_avoid 0.35 to 0.40. On restates-base it is not: none 0.74 to 0.79. Its correct labels are low too, down to 0.32 on specific-apology and 0.50 to 0.56 on staff-voice.
typesafe/jev-1.13@typesafe picked the right label on all 33 trials with two close calls: chained-steps at 0.69 to 0.76 and named-team at 0.54 to 0.71. perplexity/pplx-decider-v1.1-27b@perplexity never went below 0.96 on any trial, and upstage/solar-decide@upstage never below 0.84.
qwen/qwen3.8-flash@alibaba errored on 4 of 33 trials, every one a 429 rate limit from the pinned Alibaba provider. Its 100 is the mean of the 29 trials that returned.
google/gemini-3.5-flash-lite@google-ai-studio missed one trial: never-then-do on trial 1, labelled only_avoid instead of none.

Conclusion

Eleven targets swept every trial, so cost, latency and confidence decide it. perplexity/pplx-decider-v1.1-27b costs $0.0003 for 33 trials at 952ms, and every label it returned carries 0.96 or more, so a linter can act on its probability without a review band. typesafe/jev-1.13 is faster at 508ms for $0.0010, but its probability drops to 0.54 on a correct label. Pick the Perplexity decider; take Jev only when sub-600ms latency matters. Among chat models, openai/gpt-6-luna is cheapest at $0.0022 but takes 2,440ms, and anthropic/claude-haiku-5.5 is the balance at 1,633ms for $0.0029.

The decision models are not interchangeable. solar-decide-flash is unsafe as a linter: it never says none, so it would flag every clean bullet, and its 39.4 is just the cases where a rule was broken and it happened to name the right one. gpt-6-luna-decisions is consistent rather than flaky, but its blind spot lets two of five rule violations through as clean, and its low probabilities on correct labels mean no cutoff separates right from wrong. Qwen 3.8's row is rate limiting, not judgement, and the gemini-3.5-flash-lite miss is a one-off on a control.