DS-Lighting makes data-science agent harnesses explicit for reproducible testing
Researchers have released DS-Lighting, a toolkit built to fix a specific reproducibility problem in AI agents that automate data-science work. According to the authors, existing data-science agents typically leave their harness, the code that represents tasks, tracks execution state, constrains what output an agent can produce, and scores the result, implicit rather than documented. That makes results across different agents hard to reproduce, hard to compare, and hard to attribute to the right cause. DS-Lighting responds by decomposing the harness into four reusable layers: data, workflow, execution, and evaluation. Within that structure, it represents different agents as executable operator programs, a format that covers both fixed, predefined pipelines and agents that adaptively search for a solution. The toolkit also folds several existing open-source data-science benchmarks into a single MLE-Bench-style task format, giving them a shared task interface, a common sandboxed runtime, and one metric protocol so results can be compared directly instead of task by task. The authors report running experiments across multiple agents, harnesses, models, and ablation settings, and say the results show that making harness design explicit improves reproducibility, comparability, and reliability, while cutting down on avoidable system-level failures during end-to-end data-science runs. The abstract does not give specific benchmark scores or name the benchmarks used; the code is published at github.com/usail-hkust/dslighting.
Key facts
- DS-Lighting decomposes an AI agent's harness into four reusable layers: data, workflow, execution, and evaluation
- Agents are represented as executable operator programs, supporting both predefined pipelines and adaptive search
- Multiple open-source data-science benchmarks are folded into a shared MLE-Bench-style task format with a common task interface, sandboxed runtime, and metric protocol
- The authors report that explicit harness design improved reproducibility, comparability, and reliability and reduced avoidable system-level failures across their experiments
- The toolkit's code is published on GitHub at usail-hkust/dslighting
Why it matters
Benchmarking claims about data-science AI agents are only as trustworthy as the harness underneath them, the code that defines the task, tracks state, and grades the outcome. When that harness is implicit and undocumented, as the authors say it usually is, two papers reporting different numbers for the same task may simply be measuring different setups, not different agents. DS-Lighting's pitch is to make that scaffolding an explicit, shared, four-layer design so comparisons mean what they claim to mean.
Who it affects
The intended audience is researchers and engineers building or evaluating AI agents for data-science automation, plus anyone designing benchmarks for such agents. Making the harness explicit and reusable is aimed at people who need to compare agents, harnesses, or models against each other rather than at end users of a finished data-science product.
How to use it
DS-Lighting is released as code at github.com/usail-hkust/dslighting. It is built around four layers, data, workflow, execution, and evaluation, and represents agents as executable operator programs, so it can express either a fixed predefined pipeline or an agent that adaptively searches for a solution. It also packages multiple existing open-source data-science benchmarks into one MLE-Bench-style task format with a shared task interface, sandboxed runtime, and metric protocol, which is what lets different agents and harnesses be run and scored under the same conditions.
How solid is it
The authors describe experiments across agents, harnesses, models, and ablations, and say these show gains in reproducibility, comparability, and reliability along with fewer avoidable system-level failures. The abstract does not give any of the resulting numbers, so the size of that improvement is not stated here. No publication venue, conference, or timeline is given either.
Risks and caveats
The abstract names no specific benchmarks integrated into the toolkit and no quantitative results, so the claims of improved reproducibility and reliability cannot be checked against figures at this stage. How well a four-layer harness generalizes beyond the settings the authors tested is also not addressed.