Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations

Abstract

Reality Proxy presents a mixed-reality interaction paradigm that decouples user input from strict physical-object constraints. Instead of interacting directly with crowded, distant, or partially occluded real objects, users interact with abstract proxies. The system enriches these proxies with semantic attributes and hierarchical spatial relationships, enabling more fluid object selection and manipulation.

Problem

Direct interaction with real-world objects in MR breaks down in common situations:

• Objects are far away, dense, or occluded
• Selection becomes physically awkward or imprecise
• Complex multi-object operations are hard to execute efficiently

Core Idea: Proxy-Based Interaction

Reality Proxy shifts interaction targets from physical objects to their abstract proxies during selection. This keeps intent while reducing physical friction.

What the System Enables

• Fast skimming across many real-world objects
• Attribute-based filtering for targeted selection
• Navigation through nested object groups
• Complex multi-object selections without adding new gesture sets or menu-heavy flows

Demonstrated Scenarios

• Office information retrieval
• Large-scale spatial navigation
• Multi-drone control

Key Contributions

• A general interaction framework for proxy-based MR object manipulation.
• AI-supported enrichment of proxies with semantic and structural context.
• Evidence from expert evaluation indicating practical utility and usability.

Why It Matters

This work suggests a broader direction for MR systems: using abstraction as a first-class interaction primitive, rather than forcing all operations to remain tightly bound to physical geometry.

Video / Demo

Demo artifacts and extended resources will be added here.