Google Research introduces Agile Deliberation for subjective image classifiers

Google Research published a blog post on November 4, 2026, by Otilia Stretcu (Research Scientist) and Leijie Wang (Student Researcher) introducing Agile Deliberation, a framework for building image classifiers for subjective categories.

The problem: machine learning assumes the person labeling data has a clear, stable definition of the category before starting. That holds for a stop sign or a golden retriever. It breaks for categories like "healthy food", "inspiring artwork" or content moderation, where boundaries shift with cultural context, personal philosophies or changing operational policies. People also rarely have a complete specification in their heads; they find the real boundaries and exceptions only when they meet real examples. Traditional workflows ignore this and force rigid labels while the person's mental model is still moving.

Agile Deliberation treats building a subjective classifier as an iterative conversation. Instead of humans acting as static data labelers, the AI becomes a collaborative partner that helps users "deliberate" and sharpen their definitions. The work has two structured stages.

Stage one is concept scoping. The system guides the user to break a broad concept into a hierarchy of sub-concepts and explicit exclusions. For "Healthy Food", a user might add positive sub-categories such as "Salads", "Grain Bowls" and "Lean Proteins", and exclusions such as "Processed Foods" or "Raw, Unprepared Ingredients". This gives an initial structured baseline.

Stage two is concept iteration. It draws on active learning, but where active learning picks examples to minimise machine uncertainty, Agile Deliberation uses vision-language models (VLMs) to mine and surface semantically borderline images that challenge the current text-based definitions. In the post's example, the system shows a fruit smoothie and forces a choice: a healthy beverage because it contains raw fruit, or an excluded processed food because of its sugar content? Facing such cases pushes users to examine hidden assumptions and update their rules. That feedback is routed directly into an automatic prompt optimization loop, so the VLM's classification boundaries move with the user's refined definitions.

Evaluation was a problem of its own. With no static ground truth for subjective concepts, the authors say offline tests on pre-labeled datasets cannot measure how the framework adapts as a person's mental model shifts. So they built a functional prototype and ran 18 live user sessions of 1.5 hours each. Participants used the system and alternative baselines in randomized order to prevent learning bias. The authors chose two highly subjective domains and compared against three baseline paradigms; the post text as provided does not name them.

Results, as reported by the authors: participants who went through structured deliberation showed significantly higher labeling consistency over time, and by resolving boundary ambiguities early they built classifiers that substantially outperformed all standard baselines. The gains were most dramatic on the most challenging subjective images, where traditional classifiers rely on generic default assumptions rather than the user's intent. Expert participants said the system worked as a diagnostic tool, uncovering blind spots in their initial definitions.

The study also found that users felt pronounced cognitive fatigue when trying to predict search terms that might surface edge cases during manual concept deliberation. They also reported persistent anxiety that tweaking a text prompt to fix one mistake might break parts of the classifier they had already fixed. With Agile Deliberation, users look at a curated gallery of borderline images and give natural feedback, and the system updates the model automatically. Participants reported feeling less stressed, less frustrated and less mentally drained, and every participant preferred Agile Deliberation over the manual methods.

The authors frame this as a shift in human-in-the-loop machine learning from a one-way flow of labels to a two-way collaboration, and point to uses such as trust and safety teams building content moderation filters, curators shaping digital collections, and scientists categorizing complex visual data. The code and algorithms are open-sourced on GitHub, there is a paper, and an interactive Colab/Jupyter demo walks through the "Healthy Food" example. The research came from a partnership between Google Research and Google Ads.

Key facts

Agile Deliberation, from Google Research, lets users define subjective image categories in two stages: concept scoping (sub-concepts and exclusions) and concept iteration (feedback on borderline images).
A vision-language model surfaces semantically borderline images, such as a fruit smoothie for "healthy food", and the user's feedback feeds an automatic prompt optimization loop.
The live study ran 18 sessions of 1.5 hours each, with participants using the system and baselines in randomized order.
The authors report significantly higher labeling consistency, classifiers that substantially outperformed all standard baselines, and a preference for Agile Deliberation from every participant.
Code and algorithms are open-sourced on GitHub, with a Colab/Jupyter demo built around the "Healthy Food" example.

Why it matters

Many useful vision tasks have no crisp definition: what counts as healthy food, inspiring art or content that breaks a policy. The post argues that standard labeling assumes people already know their own rules, when they usually discover them by meeting edge cases. Agile Deliberation makes that discovery part of the workflow. The AI picks the confusing examples, the person decides, and the prompt is rewritten automatically. It also takes the prompt engineering burden off the user, which the study found tiring and anxiety-inducing.

Who it affects

The authors name trust and safety teams building content moderation filters, curators shaping digital collections, and scientists categorizing complex visual data. They also suggest subject-matter experts building classifiers in their own field (for example flagging misleading clickbait) and individuals building a classifier around personal taste, such as curating "inspiring artwork". The work came from a partnership between Google Research and Google Ads.

How to use it

The authors have open-sourced the underlying code and algorithms on GitHub and offer an interactive Colab/Jupyter demo that walks through the "Healthy Food" example. From there the post suggests adapting it to any domain where you are a subject-matter expert. A paper covers the active learning techniques and the scoping and clustering algorithms. The post gives no pricing or product availability.

How solid is it

This is a first-party blog post from Google Research, written by two of the researchers, and the claims are the authors' own. The evidence is a live study of 18 sessions of 1.5 hours with randomized ordering, which suits a problem with no fixed ground truth. The post gives qualitative and comparative claims ("significantly", "substantially") but no accuracy figures, percentages or statistical tests. The number of individual participants is not stated, only the 18 sessions.

Risks and caveats

The post does not name the two subjective domains tested or the three baseline paradigms, and gives no numeric performance results, so the size of the improvement cannot be judged from it. The specific VLM or LLM used is not named. The "healthy food" and fruit smoothie examples are illustrations, not study results. Expert participants' views and the fatigue findings are self-reported. The post does not say Google plans to ship this in a product; it only mentions the Google Ads partnership.

“Users don't need to be prompt engineers; they simply look at a curated gallery of borderline images and provide natural feedback.”

— Otilia Stretcu and Leijie Wang, Google Research blog