NL2SQL Reliability Study

The question

AI can turn a plain-English question, like "which customers spent the most last month?", into SQL, the language databases understand. Most leaderboards score this by asking each question once. But AI has randomness built in: ask again and you can get a different answer.

A dashboard that refreshes every hour asks the same question 24 times a day. So the number that matters isn't "can it do this?" but "does it do this every time?" I asked every question ten times to find out.

One character, half the time

A real question from the test: "What's the French name of this card?" The database has an English table (T1) and a translations table (T2). Across ten identical tries the AI wrote this five times:

SELECT T2.name ...   -- French name, correct

and this the other five:

SELECT T1.name ...   -- English name, wrong

Same prompt, one character different. The wrong one runs perfectly and returns a reasonable-looking answer. Nothing crashes and nothing warns you. Ask once, like most leaderboards do, and it's a coin flip whether this question counts as solved.

What I found

The blue dot is the flattering number: a question counts as solved if the AI got it right even once. The orange dot is what you can actually rely on. The line between them is questions the AI gets right only sometimes. They pass your testing, then fail in production with no warning.

For the main model that gap is 13.9 points. More than a quarter of its apparent ability doesn't hold up across ten tries.

One try hides the problem

Asked once, the two numbers are identical, and that's all a one-try leaderboard can see. Every extra try is another chance to get lucky and another chance to slip. Most of the gap has opened by the third try.

The worst mistakes don't look like mistakes

A crash is easy: your code catches the error. The red bars are the real problem: the query runs fine and the answer is wrong. For the main model, these silent mistakes outnumber crashes by about 2.5 to 1.

I tried the two obvious fixes

Let the AI fix its own mistakes. When a query crashed, the AI saw the error and could try again, up to three times. That's what most "AI agents" do. Both scores went up, but the gap showed no detectable change (13.9 to 14.7, within noise). The retry only fires when something crashes, so it never sees the silent mistakes. Counting every wrong answer, it fixed about 1 in 20.

Use a smaller model. Same family and the same 4-bit compression, less than half the size. It lost 7.5 points on the flattering score but 17.3 points on reliability, more than twice as much. The gap widened from 13.9 to 23.8, 1.7 times as large. And on my machine it wasn't even faster.

What this means if you build with AI

• Test every question more than once. One passing run proves very little.
• Don't rely on error handling. For the main model, most wrong answers never raised one.
• Test on your own database. For the main model the gap ranged from 0 to 23 points depending on the database, and up to 31 for the smaller one.
• Don't pick a smaller model from its benchmark score alone. The score hides how much consistency you're giving up.

The settings, and why

• Randomness ("temperature") at a low 0.2. The study measures how much a sampled model varies, and 0.2 is a low, cautious setting a team would pick for SQL, so the gap isn't caused by turning the randomness up. Next: a temperature-0 run, to measure how identical "identical" is.
• Ten tries per question. The gap keeps growing the more you ask, so fewer tries would hide some of it.
• Qwen2.5-Coder 7B and 3B, free models run locally. The biggest version that fits on an 8 GB graphics card, plus a smaller one from the same family with the same 4-bit compression, so the main difference is size.
• Up to three tries for the retry setup. The third try rescued 12 answers against 128 for the second, so a longer loop would add little.
• Room checked, not assumed. No prompt came near the model's reading limit, and the answer limit cut off just 0.2% of single-try replies.

How the numbers were made trustworthy

• A cleaned-up test set. The popular BIRD benchmark has errors in about half its answer key, so the study uses Arcwise-Plat-SQL, a version corrected by database experts.
• No AI grades the AI. A query counts as correct only if it returns the right rows from the real database.
• Every attempt is published. All 14,880 scored queries, with the AI's raw reply and the verdict, so every number can be recomputed without running a model. Two questions whose reference answer timed out are excluded throughout.
• Every difference is checked against chance. The questions are resampled 10,000 times to see whether a result survives, and the retry's effect on the gap didn't, so it's reported as no detectable change.
• 285 automated tests, 269 of them on every change, and it all ran locally at zero cost.

Where it fits

The "right every time" score comes from τ-bench, a benchmark for customer-service agents, and other researchers have applied the idea to text-to-SQL. This study adds the layer underneath: which failures a program could even notice, how much of the problem a retry loop can reach, and what happens to the gap when the model gets smaller.

The full plain-language write-up, the technical method and every raw result are in the repository.