Glonce

DataSpace benchmark puts best data agent accuracy at 66%

researchThe Glonce editorsHugging Face🔥81Glonce rating81Impact70Novelty75Relevance100Surprise35Players40Credibility85
DataSpace benchmark puts best data agent accuracy at 66%

Researchers introduced DataSpace, a benchmark for evaluating data agents: AI systems meant to answer natural-language analytics questions by pulling evidence out of an organization's own scattered files rather than a single clean database. The paper argues that existing benchmarks test structured querying, retrieval, or open-ended analysis in isolation, and that none of them unify heterogeneous evidence discovery, complete tabular answers, and deterministic scoring in one test.

DataSpace contains 410 cross-language tasks built on 7,439 artifacts totaling 15.01 GB, spanning CSV files, JSON, SQLite databases, Markdown, PDF documents and video. For each task an agent gets only a question and the workspace, and has to return the complete tabular result the question asks for, with no other help. The benchmark also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition.

The tasks were produced with a companion tool called DataSpace-Builder, an execution-grounded pipeline that does cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, followed by human review and task repair from 11 domain experts. Grading is handled by a deterministic evaluator that performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison, so results do not depend on a human or an LLM judge reading the output.

Testing six recently released frontier multimodal models paired with five widely used agent harnesses, the authors found the best accuracy reaches 66.34%. Holding the model backbone fixed, the choice of harness alone produced a 15.36-point spread in accuracy. Across all six backbones, integrating multimodal evidence and performing joins across sources consistently reduced accuracy. The authors conclude that DataSpace remains unsaturated, meaning the benchmark still has meaningful headroom before systems max it out.

Key facts

  • DataSpace has 410 cross-language tasks across 7,439 artifacts totaling 15.01 GB, covering CSV, JSON, SQLite, Markdown, PDF and video.
  • It served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition.
  • Across six frontier multimodal models and five agent harnesses, the best accuracy reached was 66.34%.
  • With the model backbone held fixed, harness choice alone created a 15.36-point accuracy spread.
  • Multimodal evidence integration and joins across sources consistently lowered accuracy for all six backbones tested.

Why it matters

Data agents are increasingly pitched as a way to let people ask plain questions of an organization's own scattered records, spreadsheets, PDFs and video, and get back a complete, correct table rather than a rough summary. DataSpace is built to test exactly that end-to-end claim rather than isolated skills like retrieval or SQL generation, and it does so with a deterministic grader instead of a human or LLM judge, so the numbers are reproducible. Its use as the official benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition signals it is meant to become a shared yardstick rather than a one-off academic exercise.

Who it affects

The direct audience is researchers and teams building data agents and agent harnesses, plus the competitors and organizers of the KDD Cup 2026 competition that adopted DataSpace as its evaluation benchmark. More broadly it matters to anyone evaluating whether an AI system can be trusted to answer analytics questions over real company data that mixes structured and unstructured formats, since the benchmark's task design tracks that scenario directly.

How to use it

The paper does not describe pricing, licensing or an access process; DataSpace is presented as a benchmark and an accompanying construction framework, DataSpace-Builder, rather than a product. Teams building or evaluating data agents would use it by running their own model-and-harness combination against the 410 tasks and scoring the tabular output with the paper's deterministic evaluator.

How solid is it

The benchmark's construction includes safeguards meant to keep it credible: cross-language transformation and constraint-aware relational sampling to generate tasks, modality routing and artifact rendering to keep workspaces realistic, and a human review and task-repair pass by 11 domain experts. Scoring itself is deterministic, using header-invariant column alignment, type- and precision-aware normalization and order-aware row comparison, which removes subjective judging from the result. The source text does not name the six models or five harnesses individually, nor does it give a per-system breakdown beyond the best accuracy and the spread, so the headline numbers cannot yet be attributed to specific systems from the abstract alone.

Risks and caveats

Even the best-performing combination of model and harness answered only 66.34% of tasks correctly, and the 15.36-point spread from harness choice alone, with the model held fixed, shows that a weak harness can waste a strong model's capability. The finding that multimodal evidence integration and joins consistently reduce accuracy across all six backbones points to a specific weak spot: agents struggle more as workspaces get more heterogeneous, which is exactly the scenario DataSpace was built to test. The authors state plainly that DataSpace remains unsaturated, meaning there is significant room before it stops being a useful test.