NL2SQL Reliability Study

The question

AI can turn a plain-English question, like "which customers spent the most last month?", into SQL, the language databases understand. Leaderboards score this by asking each question once. But AI has randomness built in: ask again and you can get a different answer.

A dashboard that refreshes every hour asks the same question 24 times a day. So the number that matters isn't "can it do this?" but "does it do this every time?" I asked every question ten times to find out.

One character, half the time

A real question from the test: "What's the French name of this card?" The database has an English table (T1) and a translations table (T2). Across ten identical tries the AI wrote this five times:

SELECT T2.name ...   -- French name, correct

and this the other five:

SELECT T1.name ...   -- English name, wrong

Same prompt, one character different. The wrong one runs perfectly and returns a reasonable-looking answer. Nothing crashes and nothing warns you. Ask once, like a leaderboard does, and it's a coin flip whether this question counts as solved.

What I found

The blue dot is the flattering number: a question counts as solved if the AI got it right even once. The orange dot is what you can actually rely on. The line between them is questions the AI gets right only sometimes. They pass your testing, then fail in production with no warning.

For the main model that gap is 13.7 points. More than a quarter of its apparent ability doesn't hold up across ten tries.

One try hides the problem

Asked once, the two numbers are identical, and that's all a leaderboard can see. Every extra try is another chance to get lucky and another chance to slip. Most of the gap has opened by the third try.

The worst mistakes don't look like mistakes

A crash is easy: your code catches the error. The red bars are the real problem: the query runs fine and the answer is wrong. For the main model, these silent mistakes outnumber crashes by more than 2 to 1.

I tried the two obvious fixes

Let the AI fix its own mistakes. When a query crashed, the AI saw the error and could try again, up to three times. That's what most "AI agents" do. Both scores went up a little, but the gap didn't shrink. The retry only fires when something crashes, so it never sees the silent mistakes. Counting every wrong answer, it fixed about 1 in 20.

Use a smaller model. Same family, less than half the size. It lost 7.5 points on the flattering score but 17.3 points on reliability, more than twice as much. The gap nearly doubled, from 13.7 to 23.6. And on my machine it wasn't even faster.

What this means if you build with AI

• Test every question more than once. One passing run proves very little.
• Don't rely on error handling. For the main model, most wrong answers never raised one.
• Test on your own database. The gap ranged from 0 to 31 points depending on the database.
• Don't pick a smaller model from its benchmark score alone. The score hides how much consistency you're giving up.

The settings, and why

• Randomness ("temperature") at a low 0.2. At 0 the AI gives the same answer every time, so there'd be nothing to measure. 0.2 is a cautious setting a team would use for SQL, so the gap isn't caused by turning the randomness up.
• Ten tries per question. The gap keeps growing the more you ask, so fewer tries would hide some of it.
• Qwen2.5-Coder 7B and 3B, free models run locally. The biggest version that fits on an 8 GB graphics card, plus a smaller one from the same family so size is the only real difference.
• Up to three tries for the retry setup. The third try rescued 12 answers against 128 for the second, so a longer loop would add little.
• Room checked, not assumed. No prompt came near the model's reading limit, and the answer limit cut off just 0.2% of single-try replies.

How the numbers were made trustworthy

• A cleaned-up test set. The popular BIRD benchmark has errors in about half its answer key, so the study uses Arcwise-Plat-SQL, a version corrected by database experts.
• No AI grades the AI. A query counts as correct only if it returns the right rows from the real database.
• Every attempt is published. All 14,940 queries, with the AI's raw reply and the verdict, so every number can be recomputed without running a model.
• Every difference is checked against chance. The questions are reshuffled 10,000 times to see whether a result survives. One early claim didn't, so I withdrew it before publishing.
• 251 automated tests run on every change, and it all ran locally at zero cost.

Where it fits

The "right every time" score comes from τ-bench, a benchmark for customer-service agents, and other researchers have applied the idea to text-to-SQL. This study adds the layer underneath: which failures a program could even notice, how much of the problem a retry loop can reach, and what happens to the gap when the model gets smaller.

The full plain-language write-up, the technical method and every raw result are in the repository.