Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents
Abstract
A comprehensive benchmark evaluates enterprise data agents' ability to integrate and analyze multi-database data through natural language, revealing significant challenges in real-world applications.
Users across enterprises increasingly rely on AI agents to query their data through natural language. However, building reliable data agents remains difficult because real-world data is often fragmented across multiple heterogeneous database systems, with inconsistent references and information buried in unstructured text. Existing benchmarks only tackle individual pieces of this problem -- e.g., translating natural-language questions into SQL queries, answering questions over small tables provided in context -- but do not evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we present the Data Agent Benchmark (DAB), grounded in a formative study of enterprise data agent workloads across six industries. DAB comprises 54 queries across 12 datasets, 9 domains, and 4 database management systems. On DAB, the best frontier model (Gemini-3-Pro) achieves only 38% pass@1 accuracy. We benchmark five frontier LLMs, analyze their failure modes, and distill takeaways for future data agent development. Our benchmark and experiment code are published at github.com/ucbepic/DataAgentBench.
Community
Users across enterprises increasingly rely on AI agents to query
their data through natural language. However, building reliable data
agents remains difficult because real-world data is often fragmented
across multiple heterogeneous database systems, with inconsistent
references and information buried in unstructured text. Existing
benchmarks only tackle individual pieces of this problem—e.g.,
translating natural-language questions into SQL queries, answering questions over small tables provided in context—but do not
evaluate the full pipeline of integrating, transforming, and analyzing data across multiple database systems. To fill this gap, we
present the Data Agent Benchmark (DAB), grounded in a formative
study of enterprise data agent workloads across six industries. DAB
comprises 54 queries across 12 datasets, 9 domains, and 4 database
management systems. On DAB, the best frontier model (Gemini3-Pro) achieves only 38% pass@1 accuracy. We benchmark five
frontier LLMs, analyze their failure modes, and distill takeaways
for future data agent development. Our benchmark and experiment
code are published at github.com/ucbepic/DataAgentBench.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Arming Data Agents with Tribal Knowledge (2026)
- TRUST-SQL: Tool-Integrated Multi-Turn Reinforcement Learning for Text-to-SQL over Unknown Schemas (2026)
- Both Ends Count! Just How Good are LLM Agents at"Text-to-Big SQL"? (2026)
- ReViSQL: Achieving Human-Level Text-to-SQL (2026)
- AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents (2026)
- SQL-Trail: Multi-Turn Reinforcement Learning with Interleaved Feedback for Text-to-SQL (2026)
- GISA: A Benchmark for General Information-Seeking Assistant (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2603.20576 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper