Glonce

ESQ-Bench finds NL2SQL accuracy collapses on Oracle schemas

researchThe Glonce editorsarXiv🔥76Glonce rating76Impact68Novelty72Relevance100Surprise58Players72Credibility82

State-of-the-art NL2SQL (natural language to SQL) models report execution accuracy above 89 percent on established academic benchmarks such as Spider and BIRD. Researchers argue those benchmarks rely on simplified schemas and open-source SQL dialects that do not reflect enterprise database complexity, and built ESQ-Bench, an Oracle-first NL2SQL benchmark with systematic complexity tiers and a silent-divergence evaluation, to measure the gap directly.

The benchmark consists of six populated schemas, 465 tables and 164,682 rows in total with zero empty tables, built with identical seed data across Oracle, PostgreSQL, MySQL and SQL Server. On top of the schemas sit 550 gold-validated question-query pairs split into three complexity tiers (95 in Tier-1, 228 in Tier-2, 227 in Tier-3) and a four-metric evaluation harness covering exact match (EM), execution match (EX), and two further metrics abbreviated SR and SD.

On the full 550-question benchmark, GPT-4o with schema-linked prompting shows monotonic execution-match degradation across tiers: 79.8, 60.3 and 57.2 percent EX on executed queries in a June 2026 run. That contrasts with a much rosier 75.6, 80.4 and 95.8 percent the same schema-linked setup scored on an earlier 142-question pilot slice, where accuracy rose rather than fell across the same three tiers; the paper does not explain the reversal.

Claude Sonnet 4.6 with schema-linked prompting reaches 87.4, 74.9 and 68.7 percent EX on executed queries, beating GPT-4o's schema-linked result on every tier. GPT-4o's own zero-shot EX (78.7, 73.5 and 77.8 percent) inverts its schema-linked ranking at Tiers 2 and 3, which the authors attribute to lower execution rates and survivor bias in the zero-shot comparison. Local Llama 3.2 with schema-linked prompting reaches only 13.3 percent bank-wide EX, 73 correct out of 550, underscoring the gap between closed API models and open-weight baselines on enterprise Oracle schemas.

Across every model and tier, exact match (EM) stays below 7 percent. More strikingly, among queries that execute successfully, operational silent divergence, meaning the query runs without error but returns a semantically wrong result, reaches 73 to 99 percent. Failure analysis shows wrong-result semantics dominate at the higher complexity tiers.

Key facts

  • ESQ-Bench introduces six enterprise-scale schemas (465 tables, 164,682 rows, zero empty tables) replicated with identical seed data across Oracle, PostgreSQL, MySQL and SQL Server, paired with 550 gold-validated question-query pairs across three complexity tiers (95, 228, 227).
  • On the full benchmark, GPT-4o schema-linked execution-match accuracy degrades monotonically across tiers: 79.8, 60.3 and 57.2 percent, versus 75.6, 80.4 and 95.8 percent on an earlier, unrepresentative 142-question pilot slice.
  • Claude Sonnet 4.6 with schema-linked prompting outperforms GPT-4o schema-linked on every tier (87.4, 74.9, 68.7 percent EX); local Llama 3.2 schema-linked manages only 13.3 percent bank-wide EX, 73 correct out of 550.
  • Exact match stays below 7 percent tier-wide for every model, while among queries that execute successfully, 73 to 99 percent still silently return a semantically wrong result.
  • GPT-4o's zero-shot EX (78.7, 73.5, 77.8 percent) actually beats its own schema-linked score at Tiers 2 and 3, an inversion the authors attribute to lower execution rates and survivor bias in the zero-shot comparison.

Why it matters

Vendors and researchers routinely cite execution accuracy above 89 percent on academic NL2SQL benchmarks such as Spider and BIRD, numbers that get read as evidence natural-language database access is close to solved. ESQ-Bench argues those benchmarks use simplified schemas and open-source SQL dialects that do not resemble what a real enterprise runs, and builds an Oracle-first alternative, with identical schemas mirrored on PostgreSQL, MySQL and SQL Server, to test whether accuracy survives genuine enterprise complexity. It does not: GPT-4o's schema-linked execution-match score falls from 79.8 to 57.2 percent as schema tier rises.

Who it affects

The benchmark speaks directly to teams evaluating NL2SQL tools for enterprise deployment, and to the model builders whose systems it tests: GPT-4o, Claude Sonnet 4.6 and Llama 3.2 all sit in the results table. Anyone building or buying a natural-language interface to an Oracle, PostgreSQL, MySQL or SQL Server database is the intended audience, since the schemas were built to mirror that exact environment rather than a textbook one.

How to use it

ESQ-Bench ships as six populated schemas (465 tables, 164,682 rows, no empty tables) with identical seed data replicated across four database engines, organised into three complexity tiers, plus 550 gold-validated question-query pairs (95 in Tier-1, 228 in Tier-2, 227 in Tier-3) and a four-metric evaluation harness: exact match (EM), execution match (EX), and two further metrics abbreviated SR and SD. The schemas and question set are described as released, though no repository, host or licence is given for them.

How solid is it

The result pattern comes with a full evaluation harness rather than a single headline number: three tiers, four models, and a monotonic degradation curve for GPT-4o's schema-linked results across the full 550-question set. One inconsistency goes unexplained: an earlier 142-question pilot slice showed the same schema-linked setup improving across the three tiers, 75.6, 80.4 and 95.8 percent, the opposite of the full benchmark's 79.8, 60.3 and 57.2 percent, and no reason for the reversal is given.

Risks and caveats

The sharper finding is not the accuracy drop but what hides behind it: exact match stays below 7 percent tier-wide for every model, and among queries that execute without error, 73 to 99 percent still return the wrong result, a silent semantic divergence that successful execution gives no hint of. Failure analysis attributes this to wrong-result semantics dominating at higher tiers. The paper does not name ESQ-Bench's authors or institution, does not spell out what the SR and SD metrics stand for beyond their initials, and does not explain the pilot-slice discrepancy noted above.