Pro run
$19 per run
- Up to 150 of your examples
- Up to 8 models plus your current one as the baseline
- Verdict with cost per successful task, retry tax, failures and payback
We'll email you the day it opens. No payment now. Privacy
Verdicts / Banking customer-message intent classification (77 intents)
meta-llama/llama-3.3-70b-instruct is the cheapest model that matches openai/gpt-4.1
Banking customer-message intent classification (77 intents): 150 public examples, 8 models, retries and failed attempts counted in every price. Checked 2026-11-04.
Classification · 150 public examples
meta-llama/llama-3.3-70b-instruct
Matches openai/gpt-4.1 on Banking customer-message intent classification (77 intents), at a fraction of the price.
$0.000082per successful task
93.4%under openai/gpt-4.1 ($0.001235)
77.3%pass rate vs 76.7% for the baseline
Every model, same 150 tasks
| Model | Pass rate | 95% interval | Cost per successful task | Attempts per task | p50 | p95 | Vs baseline | Status |
|---|---|---|---|---|---|---|---|---|
| openai/gpt-4.1 Baseline | 76.7% | 69.3–82.7% | $0.001235 | 1.00 | 407 ms | 625 ms | baseline | Borderline |
| openai/gpt-4.1-mini | 75.3% | 67.9–81.5% | $0.000251 | 1.00 | 698 ms | 1,085 ms | b=8 c=6 · not shown to match | Borderline |
| openai/gpt-4.1-nano | 58.0% | 50.0–65.6% | $0.000085 | 1.04 | 911 ms | 1,404 ms | b=32 c=4 · not shown to match | Below bar |
| anthropic/claude-haiku-5.5 | 80.7% | 73.6–86.2% | $0.000140 | 1.04 | 301 ms | 1,339 ms | b=4 c=10 · matches | Cleared the bar |
| google/gemini-2.5-flash-lite | 66.0% | 58.1–73.1% | $0.000085 | 1.24 | 427 ms | 604 ms | b=19 c=3 · not shown to match | Below bar |
| meta-llama/llama-3.3-70b-instruct Verdict | 77.3% | 70.0–83.3% | $0.000082 | 1.00 | 458 ms | 3,285 ms | b=6 c=7 · matches | Borderline |
| mistralai/mistral-small-3.2-24b-instruct | 67.3% | 59.5–74.3% | $0.000070 | 1.01 | 631 ms | 5,664 ms | b=17 c=3 · not shown to match | Below bar |
| deepseek/deepseek-v4-flash | 75.3% | 67.9–81.5% | $0.000106 | 1.19 | 333 ms | 3,054 ms | b=6 c=4 · matches | Borderline |
Cost per successful task = total OpenRouter usage.cost across every attempt (retries on invalid output or API errors included) ÷ tasks that passed. Attempts per task above 1.00 is the retry tax. Status: Cleared = pass rate and the lower end of its 95% interval are at or above the 70.0% bar; Borderline = pass rate at the bar, interval reaches below it; Below bar = pass rate under it.
Why it counts as a match
Pass rates on the same tasks can be compared pair by pair. meta-llama/llama-3.3-70b-instruct failed 6 tasks that openai/gpt-4.1 passed, and passed 7 that the baseline failed, across 150 shared tasks. An exact one-sided test of "no more than 5 points worse" gives p = 0.009649, so it is shown to match within the margin. Pass rates: 77.3% vs 76.7%.
What the switch pays back
| Your monthly bill on the baseline | Monthly saving at this rate | Days to repay a $19 run | Days to repay a $49 run |
|---|---|---|---|
| $200 | $186.72 | 3.1 | 7.9 |
| $500 | $466.80 | 1.2 | 3.1 |
| $2,000 | $1,867.21 | 0.3 | 0.8 |
Saving = bill × (1 − $0.000082 ÷ $0.001235), the cost per successful task measured on this dataset. Your own bill and examples set your own number; a Clearbar run on your data replaces this estimate with a measured one.
Where meta-llama/llama-3.3-70b-instruct fails
116 of 150 tasks passed. These are the first 12 that did not, with the input, what was expected and what the model returned.
Do you accept other currencies besides US Dollars?
- Expected
- supported_cards_and_currencies
- Got
- fiat_currency_support
- Why it failed
- predicted fiat_currency_support
Help! My wallet was stolen and someone is taking money out. I need this money! What can I do?
- Expected
- cash_withdrawal_not_recognised
- Got
- lost_or_stolen_card
- Why it failed
- predicted lost_or_stolen_card
How long does a card delivery take?
- Expected
- card_arrival
- Got
- card_delivery_estimate
- Why it failed
- predicted card_delivery_estimate
In what ATM can I change my pin?
- Expected
- change_pin
- Got
- atm_support
- Why it failed
- predicted atm_support
Can you tell me the total cost of a transfer?
- Expected
- top_up_by_bank_transfer_charge
- Got
- transfer_fee_charged
- Why it failed
- predicted transfer_fee_charged
Some of my card payments look different than where I purchased products from. Why is that?
- Expected
- card_payment_not_recognised
- Got
- declined_card_payment
- Why it failed
- predicted declined_card_payment
How do I set-up my PIN for the new card?
- Expected
- get_physical_card
- Got
- change_pin
- Why it failed
- predicted change_pin
I would appreciate that someone let me know when there is an extra charge for payments. I was checking the app earlier and saying a charge on one of my payments that was additional that no one made me aware of before.
- Expected
- card_payment_fee_charged
- Got
- extra_charge_on_statement
- Why it failed
- predicted extra_charge_on_statement
When will the card arrive?
- Expected
- card_delivery_estimate
- Got
- card_arrival
- Why it failed
- predicted card_arrival
how much do you charge to accept transfers
- Expected
- top_up_by_bank_transfer_charge
- Got
- transfer_fee_charged
- Why it failed
- predicted transfer_fee_charged
Can I also transfer with SWIFT?
- Expected
- top_up_by_bank_transfer_charge
- Got
- transfer_into_account
- Why it failed
- predicted transfer_into_account
Can I give my friends access to my account so they can top it up for me?
- Expected
- topping_up_by_card
- Got
- beneficiary_not_allowed
- Why it failed
- predicted beneficiary_not_allowed
Where the baseline openai/gpt-4.1 fails
Do you accept other currencies besides US Dollars?
- Expected
- supported_cards_and_currencies
- Got
- fiat_currency_support
- Why it failed
- predicted fiat_currency_support
Help! My wallet was stolen and someone is taking money out. I need this money! What can I do?
- Expected
- cash_withdrawal_not_recognised
- Got
- lost_or_stolen_card
- Why it failed
- predicted lost_or_stolen_card
How long does a card delivery take?
- Expected
- card_arrival
- Got
- card_delivery_estimate
- Why it failed
- predicted card_delivery_estimate
Can you tell me the total cost of a transfer?
- Expected
- top_up_by_bank_transfer_charge
- Got
- transfer_fee_charged
- Why it failed
- predicted transfer_fee_charged
Some of my card payments look different than where I purchased products from. Why is that?
- Expected
- card_payment_not_recognised
- Got
- extra_charge_on_statement
- Why it failed
- predicted extra_charge_on_statement
How do I set-up my PIN for the new card?
- Expected
- get_physical_card
- Got
- change_pin
- Why it failed
- predicted change_pin
I would appreciate that someone let me know when there is an extra charge for payments. I was checking the app earlier and saying a charge on one of my payments that was additional that no one made me aware of before.
- Expected
- card_payment_fee_charged
- Got
- extra_charge_on_statement
- Why it failed
- predicted extra_charge_on_statement
When will the card arrive?
- Expected
- card_delivery_estimate
- Got
- card_arrival
- Why it failed
- predicted card_arrival
how much do you charge to accept transfers
- Expected
- top_up_by_bank_transfer_charge
- Got
- transfer_fee_charged
- Why it failed
- predicted transfer_fee_charged
Can I also transfer with SWIFT?
- Expected
- top_up_by_bank_transfer_charge
- Got
- country_support
- Why it failed
- predicted country_support
Can I give my friends access to my account so they can top it up for me?
- Expected
- topping_up_by_card
- Got
- beneficiary_not_allowed
- Why it failed
- predicted beneficiary_not_allowed
Are there restrictions for my disposable card since it does not seem to be working?
- Expected
- virtual_card_not_working
- Got
- disposable_card_limits
- Why it failed
- predicted disposable_card_limits
Dataset, method and re-run
Dataset
BANKING77 (PolyAI/banking77), test split
150 rows drawn with a fixed seed from the 3,080-row test split. Licence: CC BY 4.0. Source: https://huggingface.co/datasets/PolyAI/banking77
Prompt and settings
Temperature 0, max tokens 256, up to 2 retries on invalid output or API errors. System prompt:
You route banking customer messages. Choose the single intent from the label list that best matches the message. Reply with the label exactly as written and nothing else.
Re-run it yourself
The script runs this exact dataset and model list through OpenRouter with your own key. Models sample at temperature 0 and are not fully deterministic, so a re-run lands within about ±5 points of pass rate and ±10% of cost. Prices move: today's list may differ from the 2026-11-04 snapshot.
export OPENROUTER_API_KEY=sk-or-... node rerun-banking77-intent.mjs
Reproduced 2026-11-04
The downloadable script was run again from an empty folder with an OpenRouter key from the environment, for $0.2323 in total. 6 of 8 models landed within ±5 points of pass rate and ±10% of cost per successful task; the ones that moved further:
- google/gemini-2.5-flash-lite: pass rate 66.0% to 58.0% (-8 points), cost per successful task $0.000085 to $0.000082 (-3.5%)
- meta-llama/llama-3.3-70b-instruct: pass rate 77.3% to 76.0% (-1.3 points), cost per successful task $0.000082 to $0.000097 (+18.3%)
Verdict on the re-run: deepseek/deepseek-v4-flash as the cheapest match; the pick above was meta-llama/llama-3.3-70b-instruct. Models close to the match line can swap places between runs, which is why each row shows its paired test.
Get told when a verdict changes
Prices and models move every month. Leave your email and we'll send a note when a verdict here changes. One short email per change. Or follow the feed.
Used only for these notes. Privacy
Get this verdict on your own examples
Bring 50-300 input/expected pairs and your OpenRouter key. You get one named model, its cost per successful task and the failures it will have.
Deep run
$49 per run
- Up to 300 of your examples
- Up to 15 models plus your current one
- 3 re-checks within 30 days when prices or models move
We'll email you the day it opens. No payment now. Privacy
Model watch
$9 per month
- Your finalists re-checked when OpenRouter prices or models change
- A dated change list: who got cheaper, who now clears your bar
- Cached results reused, so only new or repriced cells run
We'll email you the day it opens. No payment now. Privacy