Google Research unveils AgentHands, giving XR agents synced hand gestures
Google Research has introduced AgentHands, a research prototype described in a blog post by Xun Qian, a Research Scientist on Google XR, and Ruofei Du, the team's Interactive Perception and Graphics Lead. The work, published at CHI 2026, was primarily carried out by Ziyi Liu during his time as a Student Researcher at Google. AgentHands adds synchronized, expressive hand gestures to LLM-powered conversational agents in Extended Reality, so an agent's hands describe shapes, point at objects and mimic actions in sync with its spoken responses rather than relying only on flat visual overlays like the bounding boxes used in Project Astra or Gemini 3.1 Flash Live.
The system starts with a lightweight object registration step: using eye gaze and scene reconstruction, a user tags physical items, such as an orchid or a laptop, and the system builds a spatial registry of 3D bounding boxes the agent can reference. The team built a library of hand gesture behaviors spanning three semantic categories: deictic gestures for referencing objects, iconic gestures for depicting actions or shapes, and expression gestures for conveying social cues and emotion. When a user asks a question, the backend LLM generates a reply containing inline "GestureEvents" tied to specific trigger words, and a local parser on the XR headset uses word-level timestamps to keep the text-to-speech playback and the hand animation in sync.
Google illustrates the approach with three example scenarios: an orchid-care tutor that moves its hands to the plant's base while explaining its air roots, a 3D-printer walkthrough that demonstrates a "turn and click" control sequence, and a wellness-coaching setup where the agent performs a "warning" gesture, including holding the user's hand, to flag an unhealthy behavior. To test the approach, the team ran a within-subjects study with 12 participants comparing AgentHands to a speech-only baseline using identical, researcher-scripted verbal content, so the only variable was the presence of synchronized hand gestures. Participants completed two procedural tasks combining everyday care with technical operation. Google states the results confirmed that pairing XR with co-speech gestures is highly effective for spatially grounded interaction, though the post does not publish the study's scores or effect sizes. The team says it is now exploring how to personalize the gestures further, such as adapting to a user's dominant hand or learning their spatial habits.
Key facts
- Google Research's AgentHands, built primarily by Student Researcher Ziyi Liu and published at CHI 2026, gives XR conversational agents synchronized hand gestures instead of flat 2D overlays.
- Gestures are organized into three categories: deictic (referencing), iconic (depicting actions or shapes), and expression (social and emotional cues).
- The agent's LLM output embeds inline GestureEvents tied to trigger words, and a headset-side parser uses word-level timestamps to sync speech and hand animation.
- A within-subjects study with 12 participants, completing two procedural tasks, compared AgentHands to a speech-only baseline with identical scripted content.
- Google says the study confirmed the gesture-plus-XR combination is highly effective for spatially grounded interaction, but the post does not disclose numeric results.
Why it matters
Multimodal AI assistants are moving from flat screens toward immersive, spatially aware settings, and pointing at a 2D bounding box on a phone camera does not translate to a 3D headset view. AgentHands is Google's attempt to close that gap by giving an agent's hands the same descriptive and referential role human hands play in conversation, building on the company's earlier Human I/O and Sensible Agent research into more embodied AI interaction on Android XR.
Who it affects
The work targets Android XR developers and researchers building spatially grounded assistants, plus Google's own XR product teams; it is a research prototype rather than a shipped feature, so there is no direct end-user impact yet. The blog post credits Xun Qian and Ruofei Du as authors, Ziyi Liu as the primary researcher, David Li, Zhongyi Zhou and David Kim as key contributors, and Adarsh Kowdle, Guru Somadder and Shahram Izadi for strategic guidance and review.
How to use it
AgentHands is a research prototype tied to a CHI 2026 paper, not a released product or SDK; the post gives no release date, pricing or availability for Android XR integration. The workflow it describes, gaze-based object tagging, a three-category gesture library, LLM-generated GestureEvents, and headset-side sync with text-to-speech, illustrates the pipeline rather than a tool developers can install today.
How solid is it
The evidence is a within-subjects user study with 12 participants across two procedural tasks, comparing AgentHands against a speech-only baseline using the same scripted verbal content, which isolates the effect of the gestures themselves. Google states the results confirmed the approach is highly effective, but the post does not publish the underlying scores, percentages or effect sizes, so the claim rests on the company's own summary of a small-sample study rather than disclosed data.
Risks and caveats
The 12-participant sample is small and the post omits the actual measured outcomes, so the effectiveness claim cannot be independently checked from what Google has published. The three showcased scenarios, orchid care, 3D-printer operation and wellness coaching, are presented as illustrative demonstrations; the post does not say which two tasks the study participants actually performed. No academic co-authors, external affiliations, funding sources or a product timeline are disclosed beyond the CHI 2026 venue and the Google-internal team.