Led design work end-to-end, initiated and directed the user research, drove alignment across hardware, platform engineering, and product, and authored the system framework that became the foundation for the product roadmap.
Input Fluency across Devices
Gesture interactions as a continuous system, not bounded by devices
Every device has its own gesture language. Interaction begins and ends within screens, apps, and hardware boundaries. But, human intent is continuous — flowing across contexts, space, and time.
As XR hardware proliferates — glasses, phones, watches, buds, controllers — the isolated languages start to conflict.
The result is a fragmented user who must context-switch between interaction models dozens of times a day, paying a cognitive tax no one designed intentionally. This project built a system to explore a different model:
Free inputs from “the device themselves” to “the user’s device ecosystem”
Designers, Prototypers, Engineers
Cross departments, e.g., UX, Engineering, Research, Product Planning, etc. Cross-company collaboration with a platform partner
Mobile OS platform, cross-device wearable ecosystem
Shipped as a prioritized feature for display-less smart glasses
Targeted for display glasses and future releases
Interaction should be treated as a device-agnostic layer, where input, interpretation, and output dynamically shift across contexts.
Problem: Glasses input doesn't scale
Glasses touchpad size will inevitably face stronger industrial-design constraints, and as the interactive surface becomes smaller, designers are forced to trade off operating comfort against product aesthetics.
Yet product appearance directly influences willingness to purchase and, over time, cannot be compromised.
As wearable and spatial interfaces grow in complexity, shifting to a distributed input system across devices and making them operate as one coherent system is unavoidable.
However, current input systems are not designed to scale across devices. The problem more specifically manifests in four ways:
Conflicting primitives
The same gesture motion carried different meanings across devices. Users had no transferable mental model.
Context-switching cost
Moving between devices required re-learning, not just switching. Every transition added friction and cognitive load.
Hardware asymmetry
Not all gestures were physically available on all devices. Some inputs were architecturally impossible to unify.
Technical ceiling
Latency and detection reliability constrained which gestures could actually be shipped, so design intent and engineering reality diverged.
Approach - Input Fluency
A gesture system operates across an ecosystem of devices — watch, buds, and phone — each with distinct capabilities.
In this gesture system, interaction is not executed on devices. It flows through a system: Input is captured across multiple form factors, interpreted through a shared intelligence layer, and dynamically expressed through the most appropriate interface.
Devices become endpoints — not containers of interaction.
Gesture Framework: One input mental model, scalable interfaces.
Consistent input mental model forms the foundation of the system—each mapped to a consistent semantic meaning across all devices.
While phones, watches, computers, earbuds, voice, and even future sensing devices can all become input endpoints for the glasses, input fluency cannot be reached without the aid of contextual awareness, an ability to choose the most natural input method across devices and according to context.
Shipping Trade-off 01: Gesture selections
When communicating with engineers, variation in hardware limitations and allocation of resources in platform supports introduce ambiguity — making some gestures unreliable when deployed. Main considerations include:
Availability
Not all gestures are supported on the platform and the legacy device ecosystem today.
Reliability
Higher frequency actions are assigned to more reliable inputs.
Latency
Gestures such as double tap have latency considerations across different devices.
Ergonomics
High-effort or precision-heavy gestures are deprioritized in frequent interaction paths.
Shipping Roadmap
The designed system aimed for full cross-device consistency. In reality, intention and implementation diverged. For example, double tap was designed as a core primitive, but failed to meet shipping latency thresholds reliably and was replaced with 2-finger tap to protect the user experience at the cost of expressiveness.
These were not failures, but deliberate trade-offs. The final system protected consistency at the highest-frequency interactions and absorbed compromise at the edges, where cognitive load is lowest. Shipping a coherent 80% is better than shipping an inconsistent 100%.
Shipping Trade-off 02: Device arbitration
The ideal model was dynamic arbitration: the system would infer intent and route input to the right device automatically. However, platform and contextual-awareness limitations made this unreliable for launch.
We therefore shipped a deterministic model built on two predictable rules:
Active-screen first
Control defaults to the active device, eliminating explicit switching.
User override
Intentional input from another device immediately reclaims control.
Dynamic arbitration
As platform-level context awareness matures, arbitration can move from deterministic rules toward intent-aware orchestration across devices.
Shipping Roadmap
The north star was fully context- and intent-aware orchestration. Platform capability and reliability constraints, however, made dynamic arbitration too unpredictable for launch. We therefore shipped a deterministic model: control follows the active device, while explicit user input can always override it. As system intelligence matures, the model can progressively evolve toward dynamic arbitration.
From interaction model to sensory feedback
Working with the visual design team, the interaction principles were translated into visual feedback explorations for touch and pressure. The selected direction used inward attraction and elastic rebound to communicate contact, intensity, and completion before hardware integration.
01 — Water
Pressure encodes amplitude: a heavier touch displaces more particles, producing a stronger disturbance.
02 — Cloth
The inward pull gives touch a sense of physical grip; while elastic rebound communicates completion. Selected for prototyping.
03 — Light
A subtle peripheral light guides attention toward the touch point without demanding focus. Selected for shipping.
Outcome
Impact 1
Shipping
Elevated to top-priority status on the product roadmap.
Impact 2
Cross-team
The framework informed cross-team platform decisions.
Outlook
Ecosystem
Established a shared interaction vocabulary for subsequent device work across the wider device ecosystem.
As devices multiply, interaction does not have to be more complex for the user—it should be coordinated by the system.
Next project