PlannerForge automates scenario-based testing for autonomous driving systems

Researchers present PlannerForge, an LLM-agent framework built to cover the full scenario-based testing pipeline for Autonomous Driving Systems (ADS). Scenario-based testing is the standard way to validate self-driving software, but the authors say it has stayed a fragmented pipeline: scenario generation, retrieval, modification, ADS execution and results analysis are each handled by separate tools with little interaction between them. PlannerForge extends every stage of that pipeline, from Scenario Generation through to ADS Assessment, and adds two further LLM-driven stages: ADS Enhancement and ADS Benchmarking. The authors state that no prior work covers the whole scenario-based testing pipeline for ADS with a single, unified LLM-agent framework.

The team evaluated PlannerForge with 10 off-the-shelf large language models across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing and Enhancement) under 5 prompt conditions. Best-per-task scores across these tasks ranged from 0.88 to 1.00. Open-source backends in the 20-35B parameter range matched commercial APIs on most tasks; one such model, Qwen3.6:35B, is reported to match commercial APIs on three of the five tasks. When the pipeline's modules were chained end to end, it retained 83% of seed queries with commercial backends and 78% with open backends.

Against prior tools, PlannerForge outperformed Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-language scenario generation, producing 193 executable scenarios out of 200 versus 144 for the earlier tool, and it realised 92 to 96% of requested city, road and vehicle attributes. On scenario selection, it beat the BM25 retrieval baseline (Robertson and Zaragoza, 2009) at rank-1 accuracy, 92.0% versus 67.5%. On scenario modification, it outperformed the From-Words-to-Collisions method (Gao et al., 2025) on the rate of physically valid edits, at least 94% versus 31%.

In a separate cost-tuning experiment with a sample size of 400, tuning the pipeline lifted planner success from 50.4% to 70.2% and cut the collision rate from 19.0% to 8.4%, without any domain-specific fine-tuning of the underlying models.

Key facts

PlannerForge is an LLM-agent framework that covers the entire scenario-based testing pipeline for autonomous driving systems, adding two new stages: ADS Enhancement and ADS Benchmarking
Evaluated with 10 off-the-shelf LLMs across all tasks under 5 prompt conditions, with best-per-task scores ranging from 0.88 to 1.00
Open-source 20-35B parameter models matched commercial APIs on most tasks; Qwen3.6:35B is reported to match them on three of the five tasks
It beat Scenario Factory 2.0 on executable scenario generation (193 vs. 144 out of 200) and the BM25 baseline on rank-1 selection accuracy (92.0% vs. 67.5%)
At a sample size of 400, cost-tuning raised planner success from 50.4% to 70.2% and cut collisions from 19.0% to 8.4%, without domain-specific fine-tuning

Why it matters

Scenario-based testing is the accepted way to check whether a self-driving car's planning software behaves safely, but the paper argues the process has stayed a fragmented pipeline: separate tools handle scenario generation, retrieval, modification, running the ADS, and analysing results, with little interaction between the stages. PlannerForge's contribution is to unify all of those stages under one LLM-agent framework and add two more, ADS Enhancement and ADS Benchmarking, that were not previously part of the pipeline. The authors state that no earlier work has covered the whole scenario-based testing pipeline for ADS with a single unified LLM-agent framework, which is the gap PlannerForge is built to close.

Who it affects

The direct audience is researchers and engineers who build and validate autonomous driving software, particularly teams responsible for scenario-based testing of motion planners. The comparisons against Scenario Factory 2.0, BM25 retrieval and the From-Words-to-Collisions method place PlannerForge against tools already used in that same testing pipeline, so its most relevant readers are the groups currently relying on those or similar tools.

How to use it

PlannerForge is presented as a research framework rather than a commercial product; the source gives no pricing, licensing or availability details, so none are stated here. As described, it chains LLM agents across the testing pipeline's stages, Generation, Selection, Modification, Module Routing, Planner Testing and Enhancement, and can run on different backend language models. Because open-source backends in the 20-35B parameter range are reported to match commercial APIs on most of these tasks, the framework does not require a commercial API to reach comparable scores on the evaluated tasks.

How solid is it

The evaluation spans 10 off-the-shelf LLMs across all tasks under 5 prompt conditions, with best-per-task scores of 0.88 to 1.00, and is benchmarked against three named prior methods: Scenario Factory 2.0, a BM25 retrieval baseline, and From-Words-to-Collisions, each on a specific metric (executable scenario generation, rank-1 selection accuracy, and physically valid edits respectively). The cost-tuning result is reported from a separate run with a sample size of 400. These are the authors' own reported figures from their paper; the source text does not name the specific commercial APIs used for comparison or explain what the cost-tuning procedure consists of.

Risks and caveats

The source text gives no author names or institutional affiliations, and no submission or publication date. It does not explain what the cost-tuning procedure consists of beyond the label, does not name which commercial APIs Qwen3.6:35B was compared against, and does not define what a seed query is or how the 83%/78% retention figures were measured. It also gives no detail on what the ADS Enhancement and ADS Benchmarking stages actually do beyond naming them. All of the reported figures come from the authors' own evaluation, with no independent replication mentioned in the text.