Led design work end-to-end, initiated and directed the user research, drove alignment across hardware, platform engineering, and product, and authored the system framework that became the foundation for the shipping roadmap.
Input Fluency across Devices
Gesture interactions as a continuous system, not bounded by devices
Every device has its own gesture language. Interaction begins and ends within screens, apps, and hardware boundaries. But, human intent is continuous — flowing across contexts, space, and time.
As XR hardware proliferates — glasses, phones, watches, controllers — the isolated languages start to conflict.
The result is a fragmented user who must context-switch between interaction models dozens of times a day, paying a cognitive tax no one designed intentionally. This project built a system to explore a different model:
Designers, Prototypers, Engineers
Cross departments, e.g., UX, Engineering, Research, Product Planning, etc. Cross company collaboration: Google
Android platform, Galaxy ecosystem
Shipped for Galaxy displayless glasses P1 feature
Aiming for P0 for display glasses and future releases
Interaction should be treated as a device-agnostic layer, where input, interpretation, and output dynamically shift across contexts.
Problem: Glasses input doesn't scale
As wearable and spatial interfaces grow in complexity, shifting to a distributed input system across devices and making them operate as one coherent system is unavoidable. However, current input systems are not designed to scale across devices. The problem more specifically manifests in four ways:
Conflicting primitives
The same gesture motion carried different meanings across devices. Users had no transferable mental model.
Context-switching cost
Moving between devices required re-learning, not just switching. Every transition added friction and cognitive load.
Hardware asymmetry
Not all gestures were physically available on all devices. Some inputs were architecturally impossible to unify.
Technical ceiling
Latency and detection reliability constrained which gestures could actually be shipped, so design intent and engineering reality diverged.
Approach
A gesture system operates across an ecosystem of devices — watch, ring, and phone — each with distinct capabilities.
In this gesture system, interaction is not executed on devices. It flows through a system: Input is captured across multiple form factors, interpreted through a shared intelligence layer, and dynamically expressed through the most appropriate interface.
Devices become endpoints — not containers of interaction.
System Architecture
A cross-device gesture system composes of three main components, i.e., input, interpretation, and output. Together, these form a continuous interaction system rather than a collection of device-specific interfaces.
Touch, gesture, motion
Signals from glasses, watch, ring, and phone.
Context and space
Spatial, environmental, and situational cues.
Biosignals and EMG
Body-state signals that supplement explicit control.
Signal processing
Normalize raw sensor streams into usable data.
Intent resolution
Translate activity into likely user intent.
Context awareness
Evaluate environment, task state, and availability.
Adaptive mediation
Route execution to the lowest-cost output channel.
Audio / Haptics
Ambient feedback for low-attention moments.
Spatial UI
Glasses-based interfaces when space adds value.
Screen UI
Phone and watch surfaces for precise control.
Further with the help of AI and strengthened contextual awareness, this continuous interaction model ensures seamless communication across devices: signals are captured across devices, interpreted through a shared intelligence layer, then expressed through the interface best suited to the moment.
For how gesture evolves in an context-aware system, how intent is inferred from environmental and physiological signals, and how interaction dynamically shifts across context, visit the related project Context-aware Gesture.
Interaction system: One input model. Four interfaces.
Single input model forms the foundation of the system—each mapped to a consistent semantic meaning across all devices. Under galaxy eco-system, four devices leverage their unique native capabilities.
Gesture Frameworks
To design a unified gesture system, we first needed to establish a consistent gesture language that could operate across devices. This involved defining a set of core principles to guide the design of gestures, ensuring they were intuitive, learnable, and adaptable to different contexts.
Through iterative testing and refinement, a core set of four interaction primitives were identified.
Trade-offs: Where the ideal met its constraints
Principled selection isn't enough on its own. Variation in users, sensing noise, and hardware limitations introduce ambiguity — making some gestures unreliable when deployed. More real-world considerations include:
Availability
Not all gestures are supported on every device today.
Reliability
Higher frequency actions are assigned to more reliable inputs.
Adaptability
Gestures must adapt across different form factors and learning curves.
Ergonomics
High-effort or precision-heavy gestures are deprioritized in frequent interaction paths.
Trade-offs for Shipping Roadmap
The designed system aimed for full cross-device consistency. In reality, intention and implementation diverged. For example, double tap was designed as a core primitive, but failed to meet shipping latency thresholds reliably and was replaced with 2-finger tap to protect the user experience at the cost of expressiveness.
These were not failures, but deliberate trade-offs. The final system protected consistency at the highest-frequency interactions and absorbed compromise at the edges, where cognitive load is lowest. Shipping a coherent 80% is better than shipping an inconsistent 100%.
Device arbitration model
Interaction is not statically assigned to devices. Instead, the system dynamically arbitrates where and how input should be executed. When multiple input devices are active simultaneously—watch, ring, and phone—input conflicts become inevitable. Without a clear arbitration model, cross-device input breaks down. The system resolves control through simple, predictable rules:
Active-screen first
Control follows the waked device, eliminating the need for explicit switching.
User override
Any device can reclaim control through intentional input, without entering modes or settings.
Dynamic arbitration
With help of platform support and contextual aware capability by AI, arbitration is handled at the system layer, so individual applications remain consistent and smartly orchestrated.
System in Practice: Applying the model
The system was validated through three scenarios—each across four devices, glasses, watch, ring, and phone — designed to stress a different dimension of the input model:
Messages — Navigating structure and depth
Cardinal swipes on glasses touchpad, watch and mobile screens, and Thumb cardinal swipe on rings to navigate a notification list. This scenario tests whether a directional input model can scale across multi-level structures without introducing additional gestures.
AI Invocation — Managing session lifecycle
Consistent long tap across different device surfaces or buttons to initiates the AI agent, while single tap/press-based gestures control pause, resume, and exit. This scenario tests whether a minimal gesture set can support a full interaction lifecycle—entry, control, and exit—without relying on voice input.
Media Control — Differentiating parallel actions
Swipes and multi-finger swipes are used separately to control track navigation and volume. This scenario tests whether input can support parallel actions without gesture collision or ambiguity.
Visual feedback exploration— designing the feel of touch
The gesture vocabulary was translated into concrete watch UI and touchpad feedback before any hardware integration. Each expression was designed to be readable at wrist-glance speed, without additional instruction.
Direction 01 — Water
Contact creates a ripple that spreads outward from the touch point, weakens over distance, and fades over time. Pressure encodes amplitude: a heavier touch displaces more particles, producing a stronger disturbance.
This metaphor makes touch duration and intensity readable at a glance — which matters when the watch face is the only feedback channel for what the glasses are doing.
Water-based particle response: ripple expansion on contact, pressure amplitude, fade-over-time on release.
Direction 02 — Cloth
Touch pulls nearby particles inward toward the contact point — creating a local depression — then springs them back on release with elastic rebound. Where water felt expansive and ambient, cloth felt precise and responsive.
The inward pull gives touch a sense of physical grip; the spring-back communicates completion. In testing, users described cloth as feeling more "intentional" — the stronger candidate for production, with the highest legibility across all six gesture states.
Cloth-based particle attraction and elastic rebound — the direction selected for hardware prototyping.
Outcome
This work establishes input as a distributed system rather than a device-bound interaction. A single, consistent input model can extend across glasses, watch, ring, and phone—allowing interaction to remain coherent as it moves between surfaces.
More importantly, it reframes input fluency as a system design problem: as devices evolve, the user's mental model no longer needs to.
The work was elevated to a P0 priority on the shipping roadmap for display glasses. The framework influenced cross-team platform decisions and established a shared gesture vocabulary used as a reference for subsequent device work across the Galaxy ecosystem.
As devices multiply, interaction does not have to be more complex for the user—it should be coordinated by the system.
Next project