Verdicts / SMS spam detection (spam or legitimate message)
Claude Haiku 5.5 is the cheapest model that clearly clears the bar and matches GPT-6.1 Sol
SMS spam detection (spam or legitimate message): 300 public examples, 9 models, retries and failed attempts counted in every price. Get this verdict on your own examples
Priced on the 2026-11-05 OpenRouter list: Claude Haiku 5.5 is the cheapest match at $0.000021 per successful task.
Easy task · Classification · 300 public examples
Claude Haiku 5.5
anthropic/claude-haiku-5.5
Cheapest model that clearly clears the bar and matches openai/gpt-6.1-sol on SMS spam detection (spam or legitimate message).
Mistral Small (2603) is 24% cheaper still ($0.000016) and clears the 90.0% bar, but it is not shown to match GPT-6.1 Sol: it fails 16 tasks the baseline passes (and passes 3 it fails). If you can accept that, it is the cheapest option.
Get this verdict on your own examples
Bring 50-300 input/expected pairs and your OpenRouter key. You get one named model, its cost per successful task and the failures it will have.
Every model, same 300 tasks
What counts as a pass. A task passes when the reply is exactly one label from the dataset's list (case and punctuation ignored); a near-synonym or an extra sentence fails. Method
| Model | Pass rate | 95% interval | Cost per successful task | Attempts per task | p50 | p95 | Vs baseline | Status |
|---|---|---|---|---|---|---|---|---|
| openai/gpt-6.1-sol Baseline | 98.3% | 96.2–99.3% | $0.000268 | 1.00 | 398 ms | 604 ms | baseline | Cleared the bar |
| openai/gpt-6.1-terra | 98.3% | 96.2–99.3% | $0.000163 | 1.00 | 402 ms | 599 ms | b=0 c=0 · matches | Cleared the bar |
| openai/gpt-6-luna | 97.0% | 94.4–98.4% | $0.000026 | 1.00 | 387 ms | 604 ms | b=5 c=1 · matches | Cleared the bar |
| anthropic/claude-haiku-5.5 Verdict | 97.3% | 94.8–98.6% | $0.000021 | 1.01 | 250 ms | 330 ms | b=3 c=0 · matches | Cleared the bar |
| google/gemini-3.5-flash-lite | 97.3% | 94.8–98.6% | $0.000032 | 1.01 | 375 ms | 591 ms | b=6 c=3 · matches | Cleared the bar |
| deepseek/deepseek-v4.1-flash | 96.3% | 93.5–97.9% | $0.000051 | 1.04 | 344 ms | 1,042 ms | b=6 c=0 · matches | Cleared the bar |
| qwen/qwen3.8-flash | 97.7% | 95.3–98.9% | $0.000075 | 1.11 | 815 ms | 6,345 ms | b=3 c=1 · matches | Cleared the bar |
| mistralai/mistral-small-2603 | 94.0% | 90.7–96.2% | $0.000016 | 1.00 | 338 ms | 826 ms | b=16 c=3 · not shown to match | Cleared the bar |
| meta-llama/llama-4-maverick | 94.3% | 91.1–96.4% | $0.000027 | 1.05 | 640 ms | 1,187 ms | b=15 c=3 · not shown to match | Cleared the bar |
Cost per successful task = total OpenRouter usage.cost across every attempt (retries on invalid output or API errors included) ÷ tasks that passed. Attempts per task above 1.00 is the retry tax. Status: Cleared = pass rate and the lower end of its 95% interval are at or above the 90.0% bar; Borderline = pass rate at the bar, interval reaches below it; Below bar = pass rate under it.
Why it counts as a match
Pass rates on the same tasks can be compared pair by pair. anthropic/claude-haiku-5.5 failed 3 tasks that openai/gpt-6.1-sol passed, and passed 0 that the baseline failed, across 300 shared tasks. An exact one-sided test of "no more than 5 points worse" gives p < 0.0001, so it is shown to match within the margin. Pass rates: 97.3% vs 98.3%.
What the switch pays back
| Your monthly bill on the baseline | Monthly saving at this rate | Days to repay a $19 run | Days to repay a $49 run |
|---|---|---|---|
| $200 | $184.33 | 3.1 | 8 |
| $500 | $460.82 | 1.2 | 3.2 |
| $2,000 | $1,843.28 | 0.3 | 0.8 |
Saving = bill × (1 − $0.000021 ÷ $0.000268), the cost per successful task measured on this dataset. Your own bill and examples set your own number; a Clearbar run on your data replaces this estimate with a measured one.
Where Claude Haiku 5.5 fails
Show the 8 failures (292 of 300 tasks passed)
The first 8 tasks that did not pass, with the input, what was expected and what the model returned.
In The Simpsons Movie released in July 2007 name the band that died at the start of the film? A-Green Day, B-Blue Day, C-Red Day. (Send A, B or C)
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham (2 attempts)
Latest News! Police station toilet stolen, cops have nothing to go on!
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
thesmszone.com lets you send free anonymous and masked messages..im sending this message from there..do you see the potential for abuse???
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE MINS. INDIA CUST SERVs SED YES. L8ER GOT MEGA BILL. 3 DONT GIV A SHIT. BAILIFF DUE IN DAYS. I O £250 3 WANT £800
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
Let Ur Heart Be Ur Compass Ur Mind Ur Map Ur Soul Ur Guide And U Will Never loose in world....gnun - Sent via WAY2SMS.COM
- Expected
- ham
- Got
- spam
- Why it failed
- predicted spam
Did you hear about the new "Divorce Barbie"? It comes with all of Ken's stuff!
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
No let me do the math. Your not good at it.
- Expected
- ham
- Got
- I don't see a message to classify yet. Please paste the text you want checked, and I'll reply with either "ham" or "spam."
- Why it failed
- label outside the set: I dont see a message to classify yet. Please paste the text (3 attempts)
Do you ever notice that when you're driving, anyone going slower than you is an idiot and everyone driving faster than you is a maniac?
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
Show where the baseline openai/gpt-6.1-sol fails
Latest News! Police station toilet stolen, cops have nothing to go on!
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
thesmszone.com lets you send free anonymous and masked messages..im sending this message from there..do you see the potential for abuse???
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE MINS. INDIA CUST SERVs SED YES. L8ER GOT MEGA BILL. 3 DONT GIV A SHIT. BAILIFF DUE IN DAYS. I O £250 3 WANT £800
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
Did you hear about the new "Divorce Barbie"? It comes with all of Ken's stuff!
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
Do you ever notice that when you're driving, anyone going slower than you is an idiot and everyone driving faster than you is a maniac?
- Expected
- spam
- Got
- ham
- Why it failed
- predicted ham
Dataset, method and re-run
Dataset
SMS Spam Collection (ucirvine/sms_spam), 5,574 labelled messages
300 messages drawn with a fixed seed: 150 spam and 150 legitimate (ham). Licence: CC BY 4.0 (UCI Machine Learning Repository). Source: https://huggingface.co/datasets/ucirvine/sms_spam
Prompt and settings
Temperature 0, max tokens 256, up to 2 retries on invalid output or API errors. System prompt:
You decide whether a text message is spam. Choose the single label from the list that fits: spam for unsolicited promotions, prize or premium-rate scams and bulk marketing, ham for a normal personal or legitimate message. Reply with the label exactly as written and nothing else.
Re-run it yourself
The script runs this exact dataset and model list through OpenRouter with your own key. Models sample at temperature 0 and are not fully deterministic, so a re-run lands within about ±5 points of pass rate and ±10% of cost. Prices move: today's list may differ from the 2026-11-05 snapshot.
export OPENROUTER_API_KEY=sk-or-... node rerun-sms-spam.mjs
Get told when a verdict changes
Prices and models move every month. Leave your email and you get a note when a verdict here changes. One short email per change. Or follow the feed.
Used only for these notes. Privacy