Verdicts / Product-review sentiment (positive or negative)
mistralai/mistral-small-3.2-24b-instruct is the cheapest model that clearly clears the bar and matches openai/gpt-4.1
Product-review sentiment (positive or negative): 300 public examples, 8 models, retries and failed attempts counted in every price. Get this verdict on your own examples
Prices re-checked 2026-11-04: Mistral Small 3.2 24B is still the cheapest match at $0.000011 per successful task.
Easy task · Classification · 300 public examples
Mistral Small 3.2 24B
mistralai/mistral-small-3.2-24b-instruct
Cheapest model that clearly clears the bar and matches openai/gpt-4.1 on Product-review sentiment (positive or negative).
Get this verdict on your own examples
Bring 50-300 input/expected pairs and your OpenRouter key. You get one named model, its cost per successful task and the failures it will have.
Every model, same 300 tasks
What counts as a pass. A task passes when the reply is exactly one label from the dataset's list (case and punctuation ignored); a near-synonym or an extra sentence fails. Method
No model in this run passed 3 of the 300 rows (1.0%), so 99.0% is the most any of these 8 models could score here; 3 of the baseline's 6 failures are those rows. They are usually labels that overlap or an expected answer that is open to debate; the failures below show them.
| Model | Pass rate | 95% interval | Cost per successful task | Attempts per task | p50 | p95 | Vs baseline | Status |
|---|---|---|---|---|---|---|---|---|
| openai/gpt-4.1 Baseline | 98.0% | 95.7–99.1% | $0.000283 | 1.00 | 416 ms | 607 ms | baseline | Cleared the bar |
| openai/gpt-4.1-mini | 98.3% | 96.2–99.3% | $0.000056 | 1.00 | 637 ms | 946 ms | b=0 c=1 · matches | Cleared the bar |
| openai/gpt-4.1-nano | 97.0% | 94.4–98.4% | $0.000014 | 1.01 | 577 ms | 1,283 ms | b=5 c=2 · matches | Cleared the bar |
| anthropic/claude-haiku-5.5 | 97.3% | 94.8–98.6% | $0.000023 | 1.01 | 270 ms | 347 ms | b=2 c=0 · matches | Cleared the bar |
| google/gemini-2.5-flash-lite | 86.3% | 82.0–89.8% | $0.000013 | 1.37 | 405 ms | 732 ms | b=37 c=2 · not shown to match | Below bar |
| meta-llama/llama-3.3-70b-instruct | 98.7% | 96.6–99.5% | $0.000026 | 1.00 | 251 ms | 737 ms | b=0 c=2 · matches | Cleared the bar |
| mistralai/mistral-small-3.2-24b-instruct Verdict | 98.3% | 96.2–99.3% | $0.000011 | 1.00 | 245 ms | 751 ms | b=1 c=2 · matches | Cleared the bar |
| deepseek/deepseek-v4-flash | 90.7% | 86.8–93.5% | $0.000061 | 1.21 | 489 ms | 2,634 ms | b=22 c=0 · not shown to match | Borderline |
Cost per successful task = total OpenRouter usage.cost across every attempt (retries on invalid output or API errors included) ÷ tasks that passed. Attempts per task above 1.00 is the retry tax. Status: Cleared = pass rate and the lower end of its 95% interval are at or above the 88.0% bar; Borderline = pass rate at the bar, interval reaches below it; Below bar = pass rate under it.
Why it counts as a match
Pass rates on the same tasks can be compared pair by pair. mistralai/mistral-small-3.2-24b-instruct failed 1 task that openai/gpt-4.1 passed, and passed 2 that the baseline failed, across 300 shared tasks. An exact one-sided test of "no more than 5 points worse" gives p < 0.0001, so it is shown to match within the margin. Pass rates: 98.3% vs 98.0%.
What the switch pays back
| Your monthly bill on the baseline | Monthly saving at this rate | Days to repay a $19 run | Days to repay a $49 run |
|---|---|---|---|
| $200 | $192.23 | 3 | 7.6 |
| $500 | $480.57 | 1.2 | 3.1 |
| $2,000 | $1,922.26 | 0.3 | 0.8 |
Saving = bill × (1 − $0.000011 ÷ $0.000283), the cost per successful task measured on this dataset. Your own bill and examples set your own number; a Clearbar run on your data replaces this estimate with a measured one.
Where mistralai/mistral-small-3.2-24b-instruct fails
Show the 5 failures (295 of 300 tasks passed)
The first 5 tasks that did not pass, with the input, what was expected and what the model returned.
Hmmmmm, maybe my WIFE should be rating this... She has used them for many moons. Quality not all that great and they are throw aways after a couple years, so I just keep buying replacements over and over again. :) But they do have some bang for the buck!!!
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Too much irrelevant trivia The idea of emphasizing the rivals in Lincoln's cabinet is different from anything else I've read about that period. However, Goodwin uses up too many pages on the socializing in Washington..especially with Kate Chase. 757 pages is is a big bite, and some trimming tto the identified issue wou …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Not that great "Freewheel Burning" and "Some Heads are Gonna Roll" are both very good songs, but other than that I didn't really like this album. It is still much better than Point of Entry, Rocka Rolla, and Turbo. I suggest you get Sad Wings of Destiny, Painkiller, British Steel, Stained Class, and Hell Bent for Leath …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
A Sad State but true. This is a very well written story of a cops everyday life in LA. Very sad that things have to be the way they are but he tells it like it really is. He is misjudged, mistreated and underrated which causes him to doubt himself and everyone else. He fights to stay true to all he believes and has bee …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Ok but It Could Have Been Better. i think that the book was ok but it could have been better. Like the fact that Glory won the Breeders Cup Classic by 31 lengths! I mean not even most REAL racehorses today could do such a thing. Also Cindy is a brat and that is something I can agree with. it's like "oh Samantha are you …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Show where the baseline openai/gpt-4.1 fails
Hmmmmm, maybe my WIFE should be rating this... She has used them for many moons. Quality not all that great and they are throw aways after a couple years, so I just keep buying replacements over and over again. :) But they do have some bang for the buck!!!
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Too much irrelevant trivia The idea of emphasizing the rivals in Lincoln's cabinet is different from anything else I've read about that period. However, Goodwin uses up too many pages on the socializing in Washington..especially with Kate Chase. 757 pages is is a big bite, and some trimming tto the identified issue wou …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Not that great "Freewheel Burning" and "Some Heads are Gonna Roll" are both very good songs, but other than that I didn't really like this album. It is still much better than Point of Entry, Rocka Rolla, and Turbo. I suggest you get Sad Wings of Destiny, Painkiller, British Steel, Stained Class, and Hell Bent for Leath …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
I love mandatory fees The regulatory cost recovery fee rocks! I haven't had any problems with the charge being added to my bill. I was afraid that this was one of those specials that only select customers would win, but fortunately I was able to qualify without filling out any forms. Thank you Amazon for not doing any …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Ok but It Could Have Been Better. i think that the book was ok but it could have been better. Like the fact that Glory won the Breeders Cup Classic by 31 lengths! I mean not even most REAL racehorses today could do such a thing. Also Cindy is a brat and that is something I can agree with. it's like "oh Samantha are you …
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
DVD already Put the show on DVD already!!! They haven't even syndicated it...is someone holding the originals ransom or something?
- Expected
- positive
- Got
- negative
- Why it failed
- predicted negative
Dataset, method and re-run
Dataset
Amazon Polarity (fancyzhx/amazon_polarity), test split
300 rows drawn with a fixed seed from 1,200 rows at 12 seeded pages of the 400,000-row test split. Licence: Apache-2.0. Source: https://huggingface.co/datasets/fancyzhx/amazon_polarity
Prompt and settings
Temperature 0, max tokens 256, up to 2 retries on invalid output or API errors. System prompt:
You label the sentiment of a product review. Choose the single label from the list that best matches the reviewer's overall opinion. Reply with the label exactly as written and nothing else.
Re-run it yourself
The script runs this exact dataset and model list through OpenRouter with your own key. Models sample at temperature 0 and are not fully deterministic, so a re-run lands within about ±5 points of pass rate and ±10% of cost. Prices move: today's list may differ from the 2026-11-04 snapshot.
export OPENROUTER_API_KEY=sk-or-... node rerun-amazon-sentiment.mjs
Get told when a verdict changes
Prices and models move every month. Leave your email and you get a note when a verdict here changes. One short email per change. Or follow the feed.
Used only for these notes. Privacy