ABSTRACT
Bridging the physical and digital world through interaction remains a core challenge in augmented reality (AR). Existing systems target single objects, limiting support for planning, comparison, and assembly tasks that depend on relationships among multiple items. We present Semantic Reality, an AR system focused on surfacing inter-object connectivity and making it interactive. Leveraging multimodal reasoning, spatial anchoring, and physical action recognition, Semantic Reality maintains a persistent model of objects around the user and their relationships. Connections are visualized in-situ to highlight compatibility, reveal next steps, and reduce ambiguity during tasks. We contribute a connectivity-centered interaction paradigm and a system architecture that couples anchor tracking, action sensing, and model inference to construct a live connectivity graph. In an exploratory study comparing Semantic Reality to a single-object baseline, participants reported clearer inter-object understanding and higher engagement and satisfaction, without increased workload. A scenario study illustrates where connectivity aids planning, sequencing, and disambiguation.
Overview

Existing current-frame interaction (left) treats the camera view as a single query and overlays a text answer. Our approach (right) adds selection that writes into a scene-anchored semantic graph, making relations between objects visible and actionable. The graph supports connectivity-based interactions such as grouping, comparison, and compatibility checks, while keeping guidance grounded in the physical scene.

Concept overview of Semantic Reality. Left: In a furniture assembly scene, the AR overlay labels detected parts and projects connections in situ. Right: Detected objects become nodes; the system infers typed edges that the AR runtime maps back to the scene.
CITING
@misc{liu2026semanticrealityinteractivecontextaware,
title = {Semantic Reality: Interactive Context-Aware Visualization of Inter-Object Relationships in Augmented Reality},
author = {Liu, Xiaoan and Gonzalez, Eric J and Numan, Nels and Cola{\c{c}}o, Andrea and Abramyan, Lucy and Zhu-Tian, Chen and Suzuki, Ryo and Gonzalez-Franco, Mar},
year = {2026},
eprint = {2604.05265},
archivePrefix = {arXiv},
primaryClass = {cs.HC},
url = {https://arxiv.org/abs/2604.05265},
doi = {10.48550/arXiv.2604.05265}
}