Verdicts / Tool-call arguments
GPT-6 Luna is the cheapest model that clearly clears the bar and matches GPT-6.1 Sol
Tool-call arguments: 300 public examples, 9 models, retries and failed attempts counted in every price. Get this verdict on your own examples
Priced on the 2026-11-05 OpenRouter list: GPT-6 Luna is the cheapest match at $0.000051 per successful task.
Easy task · Tool-call arguments · 300 public examples
GPT-6 Luna
openai/gpt-6-luna
Cheapest model that clearly clears the bar and matches openai/gpt-6.1-sol on Tool-call arguments.
Claude Haiku 5.5 is within $0.000013 per successful task (25% more), passes 94.7% against 93.0% and answers 2.1x faster (p95 372 ms against 773 ms). If that matters more than the last fraction of a cent, it is the model to pick.
Get this verdict on your own examples
Bring 50-300 input/expected pairs and your OpenRouter key. You get one named model, its cost per successful task and the failures it will have.
Every model, same 300 tasks
What counts as a pass. A task passes when the reply is one JSON tool call that names the right function, carries every expected argument with the right value (numbers exactly; text ignoring case and spacing) and sends no argument the function does not declare; "Euros" for "EUR" fails. Method
| Model | Pass rate | 95% interval | Cost per successful task | Attempts per task | p50 | p95 | Vs baseline | Status |
|---|---|---|---|---|---|---|---|---|
| openai/gpt-6.1-sol Baseline | 94.0% | 90.7–96.2% | $0.000769 | 1.00 | 414 ms | 752 ms | baseline | Cleared the bar |
| openai/gpt-6.1-terra | 93.7% | 90.3–95.9% | $0.000425 | 1.00 | 422 ms | 740 ms | b=2 c=1 · matches | Cleared the bar |
| openai/gpt-6-luna Verdict | 93.0% | 89.5–95.4% | $0.000051 | 1.00 | 406 ms | 773 ms | b=5 c=2 · matches | Cleared the bar |
| anthropic/claude-haiku-5.5 | 94.7% | 91.5–96.7% | $0.000064 | 1.00 | 248 ms | 372 ms | b=4 c=6 · matches | Cleared the bar |
| google/gemini-3.5-flash-lite | 95.0% | 91.9–97.0% | $0.000157 | 1.00 | 391 ms | 614 ms | b=3 c=6 · matches | Cleared the bar |
| deepseek/deepseek-v4.1-flash | 96.0% | 93.1–97.7% | $0.000072 | 1.01 | 368 ms | 1,159 ms | b=2 c=8 · matches | Cleared the bar |
| qwen/qwen3.8-flash | 93.0% | 89.5–95.4% | $0.000093 | 1.02 | 803 ms | 5,838 ms | b=4 c=1 · matches | Cleared the bar |
| mistralai/mistral-small-2603 | 70.7% | 65.3–75.5% | $0.000098 | 1.45 | 341 ms | 1,757 ms | b=74 c=4 · not shown to match | Below bar |
| meta-llama/llama-4-maverick | 94.7% | 91.5–96.7% | $0.000080 | 1.00 | 583 ms | 1,008 ms | b=6 c=8 · matches | Cleared the bar |
Cost per successful task = total OpenRouter usage.cost across every attempt (retries on invalid output or API errors included) ÷ tasks that passed. Attempts per task above 1.00 is the retry tax. Status: Cleared = pass rate and the lower end of its 95% interval are at or above the 85.0% bar; Borderline = pass rate at the bar, interval reaches below it; Below bar = pass rate under it.
Show where Mistral Small (2603) fails
Of Mistral Small (2603)'s 88 failures, 58 returned an empty reply, 14 gave the right value in different words from the label, 10 gave a different value, 5 wrote a number differently from the label (0.2 for 20) and 1 left out an argument. 14 of them are tasks GPT-6.1 Sol also fails.
USER: Hi, I need a random number between 1 and 100.
- Kind
- Empty reply
- Expected
- {"name":"generate_random_number","arguments":{"min":1,"max":100}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: I want to watch a movie. Can you suggest me some? ASSISTANT: Of course! Could you please tell me your preferred genre and the year of release you're interested in? USER: I like action movies. I would prefer something released after 2010.
- Kind
- Empty reply
- Expected
- {"name":"search_movies","arguments":{"genre":"action","year":2010}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: Can you get me the latest sports news from the United States?
- Kind
- Different value
- Expected
- {"name":"get_news","arguments":{"category":"sports","country":"United States"}}
- Got
- {"name": "get_news", "arguments": {"category": "sports", "country": "US"}}
- Why it failed
- wrong: country
USER: Hi, I need help with calculating a tip. My bill amount is $50 and I want to give a 20% tip.
- Kind
- Empty reply
- Expected
- {"name":"calculate_tip","arguments":{"bill_amount":50,"tip_percentage":20}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: Hi, I need to calculate the area of a rectangle. The length is 10 units and the width is 5 units.
- Kind
- Empty reply
- Expected
- {"name":"calculate_rectangle_area","arguments":{"length":10,"width":5}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: Hi, I need some help with a loan I'm considering. ASSISTANT: Of course, I'd be happy to assist you. Could you please provide me with the principal amount of the loan, the annual interest rate, and the loan term in years? USER: Sure, the principal amount is $200,000, the annual interest rate is 3.5%, and the loan …
- Kind
- Empty reply
- Expected
- {"name":"calculate_loan_payment","arguments":{"principal":200000,"interest_rate":3.5,"loan_term":30}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: I am looking for a vegan Italian recipe with tomatoes and basil. Can you help me find one? ASSISTANT: Sure, I can help you with that. Let me find the best options for you.
- Kind
- Empty reply
- Expected
- {"name":"search_recipes","arguments":{"keywords":["tomatoes","basil"],"cuisine":"Italian","dietary_restrictions":["vegan"]}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: Hi, I need some help with my mortgage calculations. ASSISTANT: Of course, I'd be happy to help with that. Could you please provide me with the loan amount, the annual interest rate, and the term of the loan in years? USER: Sure, the loan amount is $200,000, the annual interest rate is 3.5%, and the loan term is 3 …
- Kind
- Number written differently (0.2 vs 20)
- Expected
- {"name":"calculate_mortgage_payment","arguments":{"loan_amount":200000,"interest_rate":3.5,"loan_term":30}}
- Got
- {"name": "calculate_mortgage_payment", "arguments": {"loan_amount": 200000, "interest_rate": 0.035, "loan_term": 30}}
- Why it failed
- wrong: interest_rate
USER: What does the word 'serendipity' mean?
- Kind
- Empty reply
- Expected
- {"name":"get_definition","arguments":{"word":"serendipity"}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: I want to listen to some music. Can you suggest me some songs from the artist 'Coldplay'?
- Kind
- Empty reply
- Expected
- {"name":"search_music","arguments":{"artist":"Coldplay"}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
USER: Hi, I need help with calculating my tax. My income is $50000 and the tax rate is 20%.
- Kind
- Number written differently (0.2 vs 20)
- Expected
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":20}}
- Got
- {"name": "calculate_tax", "arguments": {"income": 50000, "tax_rate": 0.2}}
- Why it failed
- wrong: tax_rate
USER: Hi, I need to know the distance between New York and Los Angeles.
- Kind
- Empty reply
- Expected
- {"name":"calculate_distance","arguments":{"location1":"New York","location2":"Los Angeles"}}
- Got
- (empty)
- Why it failed
- empty output (3 attempts)
Why it counts as a match
Pass rates on the same tasks can be compared pair by pair. openai/gpt-6-luna failed 5 tasks that openai/gpt-6.1-sol passed, and passed 2 that the baseline failed, across 300 shared tasks. An exact one-sided test of "no more than 5 points worse" gives p < 0.0001, so it is shown to match within the margin. Pass rates: 93.0% vs 94.0%.
What the switch pays back
| Your monthly bill on the baseline | Monthly saving at this rate | Days to repay a $19 run | Days to repay a $49 run |
|---|---|---|---|
| $200 | $186.74 | 3.1 | 7.9 |
| $500 | $466.84 | 1.2 | 3.1 |
| $2,000 | $1,867.36 | 0.3 | 0.8 |
Saving = bill × (1 − $0.000051 ÷ $0.000769), the cost per successful task measured on this dataset. Your own bill and examples set your own number; a Clearbar run on your data replaces this estimate with a measured one.
Where GPT-6 Luna fails
Show the 12 failures (279 of 300 tasks passed)
Of GPT-6 Luna's 21 failures, 16 gave the right value in different words from the label, 2 wrote a number differently from the label (0.2 for 20), 2 gave a different value and 1 left out an argument. 16 of them are tasks GPT-6.1 Sol also fails.
The first 12 tasks that did not pass, with the input, what was expected and what the model returned.
USER: I want to watch a movie. Can you suggest me some? ASSISTANT: Of course! Could you please tell me your preferred genre and the year of release you're interested in? USER: I like action movies. I would prefer something released after 2010.
- Kind
- Different value
- Expected
- {"name":"search_movies","arguments":{"genre":"action","year":2010}}
- Got
- {"name":"search_movies","arguments":{"genre":"action","year":2011}}
- Why it failed
- wrong: year
USER: I am looking for a book but I can't remember the title. The author's name is George Orwell and it's a dystopian novel.
- Kind
- Wording differs from the label
- Expected
- {"name":"search_books","arguments":{"author":"George Orwell","genre":"dystopian"}}
- Got
- {"name":"search_books","arguments":{"author":"George Orwell","genre":"dystopian novel"}}
- Why it failed
- wrong: genre
USER: Hi, I need help with calculating my tax. My income is $50000 and the tax rate is 20%.
- Kind
- Number written differently (0.2 vs 20)
- Expected
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":20}}
- Got
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":0.2}}
- Why it failed
- wrong: tax_rate
USER: I want to watch a movie tonight. Can you suggest me some good ones? ASSISTANT: Sure, I can help with that. Can you tell me your preferences? Like any specific genre, actor or year of release? USER: I like action movies. And I prefer movies released after 2010.
- Kind
- Missing argument
- Expected
- {"name":"search_movies","arguments":{"search_query":"action","year":2010}}
- Got
- {"name":"search_movies","arguments":{"search_query":"action movies released after 2010"}}
- Why it failed
- wrong: search_query, year
USER: I am looking for a book but I can't remember the full title. It's something like "To Kill a..." and it's by Harper Lee.
- Kind
- Wording differs from the label
- Expected
- {"name":"search_book","arguments":{"title":"To Kill a","author":"Harper Lee"}}
- Got
- {"name":"search_book","arguments":{"title":"To Kill a Mockingbird","author":"Harper Lee"}}
- Why it failed
- wrong: title
USER: I need to create an event for my birthday party. ASSISTANT: Sure, I can help with that. Could you please provide me with the date and location for your birthday party? USER: The date is 2022-05-15 and the location is my house.
- Kind
- Wording differs from the label
- Expected
- {"name":"create_event","arguments":{"title":"My Birthday Party","date":"2022-05-15","location":"My House"}}
- Got
- {"name":"create_event","arguments":{"title":"Birthday Party","date":"2022-05-15","location":"my house"}}
- Why it failed
- wrong: title
USER: I am looking for a book but I can't remember the full title. It's something like "To Kill a..." and it's by Harper Lee. Can you help me find it?
- Kind
- Wording differs from the label
- Expected
- {"name":"search_book","arguments":{"title":"To Kill a...","author":"Harper Lee"}}
- Got
- {"name":"search_book","arguments":{"title":"To Kill a Mockingbird","author":"Harper Lee"}}
- Why it failed
- wrong: title
USER: Hi, I had a cheeseburger for lunch today. It was about 563 calories. Can you log this for me? ASSISTANT: Sure, I can help with that. Let me log this information for you.
- Kind
- Wording differs from the label
- Expected
- {"name":"track_calories","arguments":{"meal":"cheeseburger","calories":563,"date":"today"}}
- Got
- {"name":"track_calories","arguments":{"meal":"Cheeseburger for lunch","calories":563,"date":"today"}}
- Why it failed
- wrong: meal
USER: I need to schedule a meeting for tomorrow. ASSISTANT: Sure, I can help with that. Could you please provide me with the details of the meeting such as the name of the event, start time, end time, and location? USER: The meeting is called "Project Discussion". It will start at 10:00 AM and end at 11:30 AM. The loca …
- Kind
- Wording differs from the label
- Expected
- {"name":"create_calendar_event","arguments":{"event_name":"Project Discussion","start_time":"10:00 AM","end_time":"11:30 AM","location":"Conference Room"}}
- Got
- {"name":"create_calendar_event","arguments":{"event_name":"Project Discussion","start_time":"tomorrow at 10:00 AM","end_time":"tomorrow at 11:30 AM","location":"Conference Room"}}
- Why it failed
- wrong: start_time, end_time
USER: I need to create a new event in my calendar. ASSISTANT: Sure, I can help with that. Could you please provide me with the title, start time, end time, and location of the event? USER: The title is "Team Meeting", it will start at 10:00 AM and end at 11:00 AM tomorrow. The location is the conference room.
- Kind
- Wording differs from the label
- Expected
- {"name":"create_calendar_event","arguments":{"event_title":"Team Meeting","start_time":"10:00 AM","end_time":"11:00 AM","location":"conference room"}}
- Got
- {"name":"create_calendar_event","arguments":{"event_title":"Team Meeting","start_time":"tomorrow 10:00 AM","end_time":"tomorrow 11:00 AM","location":"the conference room"}}
- Why it failed
- wrong: start_time, end_time, location
USER: I am looking for a book but I can't remember the exact title. It's something about a mockingbird.
- Kind
- Wording differs from the label
- Expected
- {"name":"search_books","arguments":{"query":"mockingbird"}}
- Got
- {"name":"search_books","arguments":{"query":"book about a mockingbird"}}
- Why it failed
- wrong: query
USER: Hi, I have a sentence in French that I need translated to English. The sentence is "Je suis très heureux de vous rencontrer".
- Kind
- Wording differs from the label
- Expected
- {"name":"translate_text","arguments":{"text":"Je suis très heureux de vous rencontrer","source_language":"fr","target_language":"en"}}
- Got
- {"name":"translate_text","arguments":{"text":"Je suis très heureux de vous rencontrer","source_language":"French","target_language":"English"}}
- Why it failed
- wrong: source_language, target_language
Show where the baseline openai/gpt-6.1-sol fails
Of GPT-6.1 Sol's 18 failures, 13 gave the right value in different words from the label, 3 wrote a number differently from the label (0.2 for 20) and 2 left out an argument.
USER: Hi, I need help with calculating my tax. My income is $50000 and the tax rate is 20%.
- Kind
- Number written differently (0.2 vs 20)
- Expected
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":20}}
- Got
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":0.2}}
- Why it failed
- wrong: tax_rate
USER: I want to watch a movie tonight. Can you suggest me some good ones? ASSISTANT: Sure, I can help with that. Can you tell me your preferences? Like any specific genre, actor or year of release? USER: I like action movies. And I prefer movies released after 2010.
- Kind
- Missing argument
- Expected
- {"name":"search_movies","arguments":{"search_query":"action","year":2010}}
- Got
- {"name":"search_movies","arguments":{"search_query":"action movies released after 2010"}}
- Why it failed
- wrong: search_query, year
USER: I am looking for a book but I can't remember the full title. It's something like "To Kill a..." and it's by Harper Lee.
- Kind
- Wording differs from the label
- Expected
- {"name":"search_book","arguments":{"title":"To Kill a","author":"Harper Lee"}}
- Got
- {"name":"search_book","arguments":{"title":"To Kill a...","author":"Harper Lee"}}
- Why it failed
- wrong: title
USER: Hi, I need to calculate my tax. My income is $50000 and the tax rate is 20%.
- Kind
- Number written differently (0.2 vs 20)
- Expected
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":20}}
- Got
- {"name":"calculate_tax","arguments":{"income":50000,"tax_rate":0.2}}
- Why it failed
- wrong: tax_rate
USER: I need to create an event for my birthday party. ASSISTANT: Sure, I can help with that. Could you please provide me with the date and location for your birthday party? USER: The date is 2022-05-15 and the location is my house.
- Kind
- Wording differs from the label
- Expected
- {"name":"create_event","arguments":{"title":"My Birthday Party","date":"2022-05-15","location":"My House"}}
- Got
- {"name":"create_event","arguments":{"title":"Birthday party","date":"2022-05-15","location":"my house"}}
- Why it failed
- wrong: title
USER: Hi, I had a cheeseburger for lunch today. It was about 563 calories. Can you log this for me? ASSISTANT: Sure, I can help with that. Let me log this information for you.
- Kind
- Wording differs from the label
- Expected
- {"name":"track_calories","arguments":{"meal":"cheeseburger","calories":563,"date":"today"}}
- Got
- {"name":"track_calories","arguments":{"meal":"cheeseburger for lunch","calories":563,"date":"today"}}
- Why it failed
- wrong: meal
USER: I need to schedule a meeting for tomorrow. ASSISTANT: Sure, I can help with that. Could you please provide me with the details of the meeting such as the name of the event, start time, end time, and location? USER: The meeting is called "Project Discussion". It will start at 10:00 AM and end at 11:30 AM. The loca …
- Kind
- Wording differs from the label
- Expected
- {"name":"create_calendar_event","arguments":{"event_name":"Project Discussion","start_time":"10:00 AM","end_time":"11:30 AM","location":"Conference Room"}}
- Got
- {"name":"create_calendar_event","arguments":{"event_name":"Project Discussion","start_time":"tomorrow at 10:00 AM","end_time":"tomorrow at 11:30 AM","location":"Conference Room"}}
- Why it failed
- wrong: start_time, end_time
USER: I need to create a new event in my calendar. ASSISTANT: Sure, I can help with that. Could you please provide me with the title, start time, end time, and location of the event? USER: The title is "Team Meeting", it will start at 10:00 AM and end at 11:00 AM tomorrow. The location is the conference room.
- Kind
- Wording differs from the label
- Expected
- {"name":"create_calendar_event","arguments":{"event_title":"Team Meeting","start_time":"10:00 AM","end_time":"11:00 AM","location":"conference room"}}
- Got
- {"name":"create_calendar_event","arguments":{"event_title":"Team Meeting","start_time":"tomorrow at 10:00 AM","end_time":"tomorrow at 11:00 AM","location":"conference room"}}
- Why it failed
- wrong: start_time, end_time
USER: I'm in the mood for Italian food. Can you find any Italian restaurants near me?
- Kind
- Missing argument
- Expected
- {"name":"search_restaurants","arguments":{"cuisine":"Italian","location":"near me"}}
- Got
- {"name":"search_restaurants","arguments":{"cuisine":"Italian"}}
- Why it failed
- wrong: location
USER: Hi, I have a sentence in French that I need translated to English. The sentence is "Je suis très heureux de vous rencontrer".
- Kind
- Wording differs from the label
- Expected
- {"name":"translate_text","arguments":{"text":"Je suis très heureux de vous rencontrer","source_language":"fr","target_language":"en"}}
- Got
- {"name":"translate_text","arguments":{"text":"Je suis très heureux de vous rencontrer","source_language":"French","target_language":"English"}}
- Why it failed
- wrong: source_language, target_language
USER: I need to create an invoice for a customer named John Doe. He bought 2 apples for $1 each and 3 bananas for $0.5 each. The tax rate is 10%.
- Kind
- Wording differs from the label
- Expected
- {"name":"create_invoice","arguments":{"customer_name":"John Doe","items":[{"name":"apple","price":1,"quantity":2},{"name":"banana","price":0.5,"quantity":3}],"tax_rate":10}}
- Got
- {"name":"create_invoice","arguments":{"customer_name":"John Doe","items":[{"name":"apples","price":1,"quantity":2},{"name":"bananas","price":0.5,"quantity":3}],"tax_rate":10}}
- Why it failed
- wrong: items
USER: Hey, I need to calculate my carbon footprint for driving my car for 5 hours. Can you help me with that? ASSISTANT: Sure, I can help you with that. Let me calculate it for you.
- Kind
- Wording differs from the label
- Expected
- {"name":"calculate_carbon_footprint","arguments":{"activity":"driving","duration":5}}
- Got
- {"name":"calculate_carbon_footprint","arguments":{"activity":"driving my car","duration":5}}
- Why it failed
- wrong: activity
Dataset, method and re-run
Dataset
glaive-function-calling-v2 (glaiveai/glaive-function-calling-v2)
Task: Tool-call argument checking: pick the right function and fill its arguments.
300 rows drawn with a fixed seed from the eligible rows found in 36 random pages (3,600 rows) of the 112,960-row train split, at most 5 per function; eligible = every argument value is stated in the conversation. Licence: Apache-2.0. Source: https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2
Prompt and settings
Temperature 0, max tokens 400, up to 2 retries on invalid output or API errors. System prompt:
You are given tool definitions and a conversation. Reply with the one tool call the assistant should make next, as a single JSON object {"name": "<tool name>", "arguments": {<parameter>: <value>}} and nothing else. Use only the tools and parameters declared, and fill each value from what the user said.Re-run it yourself
The script runs this exact dataset and model list through OpenRouter with your own key. Models sample at temperature 0 and are not fully deterministic, so a re-run lands within about ±5 points of pass rate and ±10% of cost. Prices move: today's list may differ from the 2026-11-05 snapshot.
export OPENROUTER_API_KEY=sk-or-... node rerun-glaive-tool-call.mjs
Get told when a verdict changes
Prices and models move every month. Leave your email and you get a note when a verdict here changes. One short email per change. Or follow the feed.
Used only for these notes. Privacy