Beyond Benchmark Accuracy: An Enterprise Deployment and Readiness Framework for Large Language Model-Based Text-to-SQL Systems

Abstract: Enterprises piloting LLM-based text-to-SQL keep discovering the same thing: a model that clears 85% execution accuracy on Spider can still be rejected by the security reviewer, the auditor and the risk committee — and rightly so. In the pilots and post-mortems that motivated this work, generated queries ignored row-level access policies, exposed masked columns, broke silently under schema change, ran unbounded scans, and left no evidence trail. Building a prototype to close these gaps taught us that the problem is architectural rather than linguistic: what was missing was not a better model but a system around the model, one in which vocabulary, policies and budgets are machine-readable and enforcement is deterministic. ERF-SQL is that system, together with a way to measure whether it is working. A versioned semantic contract becomes the sole interface between schema and model; a Policy-Aware Query Guard rewrites the abstract syntax tree of every candidate query before execution; uncertain answers are routed to clarification or human review; and behaviour is scored on six readiness dimensions aggregated into a gated Enterprise Readiness Index. On EntSQL-450, an evaluation suite with live role policies and masking rules built for this study, four production LLMs lost 19 to 23 accuracy points relative to Spider under a naive baseline; the implemented pipeline recovered 13 to 16 of those points, blocked 97% of unsafe generations before execution, and raised drift resilience from 0.55 to 0.86 at roughly 0.9 seconds of added latency per query. The demonstration we take from the implementation is simple: deployment readiness can be engineered and measured, not merely hoped for.

Keywords: Text-to-SQL, large language models, enterprise deployment, AI governance, schema linking, data security, readiness assessment, natural language interfaces to databases.