DroneCATS benchmark: drone AI navigates well but won't stop
Researchers introduce DroneCATS-Agent, an architecture that puts a multimodal large language model (MLLM) directly into a drone's control loop, with the model's entire action space declared solely inside the prompt, no fine-tuning and no function-calling schema required. Alongside it comes DroneCATS, a benchmark that treats the choice of underlying model as the variable being tested, evaluating frontier and open models across four core capabilities: approaching a target that is already visible, tracking a target that is moving, searching for a target outside the drone's initial field of view, and commanding a fleet of multiple drones at once. In this setup the agent is trusted to decide when to yaw and search, when to pause and deliberate if it is unsure, and when to declare that it has arrived.
To find where capability breaks down first, the roster of tested models scales all the way down to 2 billion parameters. The results turn up what the authors call a stark paradox: it is not flying that fails. Small open models often navigate into a target's success radius more reliably than frontier models do. What costs them the episode instead is the action protocol: they declare arrival too early, or never declare it at all. The problem gets worse in multi-drone commanding, where small models fail by blindly copying a single coordinate across views that show different scenes, rather than adapting the coordinate to each drone's own view.
The authors conclude that the models' spatial perception holds up; what breaks is the discipline to sustain a declared protocol and to emit the correct terminating action at the right moment. In their framing, this discipline, not navigation ability, is what actually separates a model fit for onboard deployment from a larger frontier model. The open problem they leave for future work is closing that gap within the compute available onboard a drone: building a model fast enough to run in the control loop while planning persistently and reliably recognizing exactly when the task is done. DroneCATS is presented as the benchmark built to measure progress toward that goal.
Key facts
- DroneCATS-Agent puts an MLLM directly into a drone's control loop, with its entire action space declared only in the prompt, no fine-tuning or function-calling schema.
- The accompanying DroneCATS benchmark tests models on four tasks: approaching a visible target, tracking a moving target, searching outside the initial view, and commanding a multi-drone fleet.
- Tested models scale down to 2 billion parameters to locate where capability first breaks down.
- Small open models often navigate into a target's success radius more reliably than frontier models, but lose the episode by declaring arrival too early or not at all.
- In multi-drone commanding, small models fail by copying one coordinate across different drone views instead of adapting it per view.
Why it matters
The paper's central finding reframes what counts as the hard part of embodied AI. Models are already good enough at perceiving space and flying toward a target, so raw navigation is comparatively solved. What is not solved is the protocol layer: knowing exactly when a task has been completed and saying so correctly. That distinction points integration effort at the layer that actually blocks reliable deployment, rather than at further improving vision or piloting skill.
Who it affects
Anyone building or evaluating agentic control of physical robots, drones specifically, where a language model chooses actions from an open-ended action space described only in a prompt. It is also directly relevant to teams weighing a small, cheap, onboard-capable model against a larger frontier model for embodied tasks, since the paper's finding is that model size does not predict which one completes the mission correctly.
How to use it
DroneCATS-Agent is described as an architecture in which the MLLM is a swappable component, so a team can plug in a candidate model and run it through the four benchmark tasks, approach, track, search, multi-drone command, to see where its action-declaration discipline breaks down before betting on it for onboard use. The text does not state whether the benchmark, its code, or its data have been released publicly, so it is unclear whether the tooling can be run externally yet.
How solid is it
The claims here come from the paper's own abstract. No specific model names, no quantitative success rates, and no author affiliations are given there, so the size of the reported gap between frontier and small models cannot be checked from what is available. The comparison does span a real range of model sizes down to 2 billion parameters, rather than resting on just two data points.
Risks and caveats
All findings described here come from the abstract; the source names no specific frontier or open models, gives no benchmark scores, and does not indicate a release timeline or whether the benchmark and code are public. The "stark paradox" framing is the authors' own characterization of their results, not an independently verified figure, until the full paper or a public benchmark run is available to check it against.
“it is not the flying that fails”
— the authors, DroneCATS paper