TU Delft uses GPT-4o-mini to let self-driving cars take driving-style requests
Self-driving cars balance a host of parameters, including speed, acceleration, and how smoothly they turn, to produce a comfortable ride. Normally, engineers tune those parameters before the vehicle ever hits the road, leaving passengers little room to adjust its driving style on the fly, even though preferences vary with how rushed a rider is, whether they feel carsick, or how heavy traffic is. Researchers at Delft University of Technology (TU Delft) in the Netherlands, led by postdoctoral researcher Diego Martinez-Baselga, built a system that lets passengers change that: state a preference in plain language, such as "I am running late, go fast," and have the car adjust its driving style accordingly. The team posted a preprint describing the work on arXiv and are presenting it at the IEEE Intelligent Transportation Systems Conference in September.
The system does not hand a passenger direct control of the car. It only retunes the parameters of a safety-aware motion-planning algorithm that keeps the vehicle's behavior within safe bounds, and it keeps a human in the loop: before acting on a request, it describes in plain language how it plans to change its behavior and waits for the passenger to confirm. Under the hood, the researchers' own model predictive-path integral controller generates several candidate paths through traffic and scores each on criteria such as speed, steering angle, and collision probability, then picks the best-scoring combination. The team paired that controller with OpenAI's GPT-4o-mini, which reads the passenger's request together with a natural-language description of the current driving scenario, handwritten by the researchers for this study but which Martinez-Baselga says could eventually come straight from a car's perception system. GPT-4o-mini does not touch the controller's settings directly; it only rates how important each judging criterion should be right now, nudging every criterion up or down from a safe baseline the researchers fixed in advance. If a passenger says they feel dizzy, for example, the model raises the weight on smooth steering and gentle acceleration so the car favors a more sedate ride. Because the loop stays interactive, a passenger who thinks the car misread a request, or who simply changes their mind, can follow up with further instructions, in Martinez-Baselga's words, "as you would do if you were in a taxi or with a friend that is driving."
The team tested the system in nuPlan, a widely used self-driving simulator, in scenarios where the car had to merge onto a busy highway. Across eight different prompts, the controller's parameters shifted in line with what each prompt asked for: requests for a more comfortable ride increased smoothness, and requests conveying urgency increased speed. The article does not give a quantitative accuracy or success rate for these eight tests, only that the changes matched user intent, and the system has so far been tried only in that simulator, on highway merging, not in a real vehicle or in other driving situations.
Martinez-Baselga says the team deliberately avoided using LLMs or video-language models to control the vehicle's decisions directly, the approach some prior research has taken. Models of that kind respond too slowly for the fast-paced decisions driving demands, he says, and unlike a deterministic motion planner, they cannot offer concrete performance guarantees. Keeping the LLM confined to adjusting the controller's tuning weights, rather than choosing the path itself, sidesteps both problems.
The Delft work is not the first attempt to steer a self-driving motion planner with an LLM. Nicolas Baumann, a PhD student at ETH Zurich in Switzerland, published research last year in which an LLM adjusted the parameters of a model racing-car controller, letting users change driving style and also issue more concrete commands such as "reverse the car" or "maintain a specific speed." Commenting on the new study, Baumann says its strength lies in separating the LLM from the main controller: even if the model hallucinates, it cannot make the car do anything dangerous, because the classical controller's constraints still bound it, though he adds that setting those constraints correctly takes considerable engineering work. Matthias Althoff, a professor of cyberphysical systems at the Technical University of Munich, says provable safety needs to go further still. His own group has built a system in which an LLM suggests driving decisions that are then run through a mathematical check against traffic rules and predictions of other road users' behavior before the car commits to them, verifying their safety in a way the Delft paper does not. "As with any LLM, it is not guaranteed that the result is correct," Althoff says. "For that reason, we safeguard the decisions of the LLM in our works."
Key facts
- TU Delft's Diego Martinez-Baselga and team paired their existing model predictive-path integral controller with OpenAI's GPT-4o-mini, so a passenger's natural-language request, such as "I am running late, go fast," retunes the car's driving style.
- GPT-4o-mini never sets the controller's parameters directly; it rates how heavily the controller should weigh judging criteria such as speed, steering angle, and collision probability, adjusting each up or down from a safe baseline the researchers fixed in advance.
- Before acting on a request, the system describes the planned change in plain language and waits for the passenger to confirm, and the passenger can keep issuing follow-up instructions if the change does not match what they meant.
- Tested in the nuPlan simulator across eight different highway-merging prompts, the system's parameter changes matched user intent qualitatively; no accuracy or success-rate figures, and no real-vehicle testing, are reported.
- Outside researchers Nicolas Baumann (PhD student, ETH Zurich) and Matthias Althoff (professor, Technical University of Munich) say separating the LLM from the controller keeps hallucinations from causing dangerous behavior, but note the Delft system, unlike Althoff's own, does not mathematically verify a chosen path's safety before committing to it.
Why it matters
Motion planners today are locked in at the factory: engineers set the trade-offs between speed, acceleration, and smoothness once, and passengers cannot change them, even though what a rider wants from a ride shifts depending on whether they are late, carsick, or sitting in heavy traffic. This work shows a way to make that trade-off adjustable after the fact, through plain language, without touching the safety guarantees underneath. Rather than asking a large language model to drive, or even to choose a path, the researchers use it purely as a translator: it reads a fuzzy request like "go fast" or a passenger saying they feel dizzy, and turns it into a numerical rating of how much weight the existing, safety-bounded controller should give to criteria such as speed or smooth steering. That separation is the paper's central idea, and it is also what the two outside researchers quoted in the article, working on related problems at ETH Zurich and the Technical University of Munich, focus their comments on.
Who it affects
Directly, this is a research contribution aimed at the autonomous-vehicle research community and at engineers who design motion planners, not a feature riders can use today. TU Delft's own prior controller work, Nicolas Baumann's racing-car controller at ETH Zurich, and Matthias Althoff's safety-verification system at the Technical University of Munich are named as related efforts, suggesting more than one lab is independently working on LLM-mediated tuning of classical vehicle controllers. If the approach eventually reaches production vehicles, it would let passengers adjust ride comfort and pace conversationally, and it would affect the automakers and suppliers who currently hardcode those trade-offs.
How to use it
There is nothing to use yet: this is a preprint plus a nuPlan simulation, not a shipped feature, and the article gives no price, license, or release plan. The interaction the researchers designed works like directing a taxi driver: a passenger states a preference in natural language, the system replies in plain language with the change it plans to make, the passenger confirms or pushes back, and the passenger can keep issuing follow-up requests if the result still is not right. In the current prototype, the natural-language description of the driving scenario that the model reads alongside the passenger's request was handwritten by the researchers themselves; Martinez-Baselga says it could eventually be generated automatically by a car's perception system, but that is not yet built.
How solid is it
This is early-stage academic work: a preprint, not stated to be peer-reviewed, alongside a presentation slated for the IEEE Intelligent Transportation Systems Conference in September, with no year given for either. Testing so far covers eight prompts in the nuPlan simulator, all within highway-merging scenarios, and the article reports only that the parameter changes qualitatively matched what each prompt asked for. No accuracy or success-rate numbers, no baseline or control condition, and no test in a real vehicle or other driving scenario are reported. The idea builds on a controller the same researchers had already developed and sits alongside two other named research efforts, Nicolas Baumann's racing-car work at ETH Zurich and Matthias Althoff's safety-verification system at the Technical University of Munich, which suggests the general approach is being pursued by more than one lab, even though this particular result is a small, narrowly scoped, simulation-only test.
Risks and caveats
The system's safety case rests on keeping the LLM away from the controls: GPT-4o-mini only nudges pre-set weights inside a safety-bounded controller, so Nicolas Baumann says that even if the model hallucinates, it cannot make the car do anything dangerous, though he cautions that setting those bounds correctly still takes considerable engineering work. The design also has to cope with the LLM misjudging what a passenger meant, which is why the system must describe its planned change and wait for confirmation before acting, and why a passenger can keep correcting it afterward. Matthias Althoff argues the human confirmation step is not enough on its own: an LLM's output is never guaranteed correct, he says, which is why his own group instead runs a mathematical check against traffic rules and other road users' predicted behavior before committing to a plan, a verification step the Delft system does not have. No paper title, no co-authors beyond Martinez-Baselga, and no funding source are given in the article.
“The motion-planning problem is not only about reaching a place while avoiding collisions, it's also how you do it”
— Diego Martinez-Baselga, postdoctoral researcher at TU Delft