[Submitted on 18 Nov 2025 (v1), last revised 26 Jul 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Natural Language Interfaces for Databases (NLIDBs) let users query data in everyday language instead of SQL, and recent systems translate those questions accurately. Accuracy says little about the work of querying: does an NLIDB remove that work or only reallocate it? We report a mixed-method, between-subjects study comparing SQL-LLM, a GPT-4o-backed NLIDB, with Snowflake, a traditional SQL platform. Twenty SQL-literate professionals and graduate students, ten per interface, each completed 12 tasks drawn from the BIRD benchmark. SQL-LLM cut completion time per query by about 31%, but the speedup did not buy accuracy: graded against the BIRD gold answers, SQL-LLM users were correct on 46% of queries versus 64% for Snowflake. Two analysts independently coded the think-aloud sessions to locate the effort. SQL-LLM users left schema navigation to the model and spent it verifying that the generated SQL matched their intent; Snowflake users explored the schema and built syntax by hand. Per-minute rates of the coded behaviors did not differ between interfaces. The interface reallocated the work of querying rather than removing it: users still had to verify the generated SQL, and an NLIDB that hid it would remove the step that let them trust the answer.

Submission history

From: Panos Ipeirotis [view email]
[v1] Tue, 18 Nov 2025 18:04:24 UTC (630 KB)
[v2] Sun, 19 Jul 2026 04:55:06 UTC (597 KB)
[v3] Sun, 26 Jul 2026 17:21:29 UTC (764 KB)