ABSTRACT

Site-specific outdoor AR experiences are typically authored using static 3D models, but are deployed in physical environments that change over time. As a result, virtual content may become misaligned with its intended real-world referents, degrading user experience and compromising contextual interpretation. We present AdjustAR, a system that supports in-situ correction of AR content in dynamic environments using multimodal large language models (MLLMs). Given a composite image comprising the originally authored view and the current live user view from the same perspective, an MLLM detects contextual misalignments and proposes revised 2D placements for affected AR elements. These corrections are backprojected into 3D space to update the scene at runtime. By leveraging MLLMs for visual-semantic reasoning, this approach enables automated runtime corrections to maintain alignment with the authored intent as real-world target environments evolve.

Full-text PDF

Teaser Video

30-second teaser video for AdjustAR. AdjustAR corrects misaligned AR content at runtime: (1) authors place content relative to a georeferenced 3D model of the target site; (2–3) users localize and view the scene in-situ, where misalignments may occur due to environmental changes; (4) the system composites live and authored views; (5) an MLLM detects misalignments and infers corrected 2D anchors; (6) corrections are backprojected into 3D and updated in the scene; (7) the user’s AR view is updated.

Examples

Overview of early examples showcasing AdjustAR. From top left to bottom right: two birds already properly aligned with the fountain edge (AdjustAR does not perform adjustments); an arrow pointing to a first aid kit that has been moved (AdjustAR moves the arrow to the new location); a virtual sign pointing at a missing physical sign (AdjustAR displays a rendering of the environment at authoring time); arrows pointing at books, with some correctly and others incorrectly placed (AdjustAR moves the incorrectly placed ones).

Video

5-minute narrated video overview of AdjustAR, covering the motivation, related work, and pipeline.

CITING

@inproceedings{numanAdjustAR2025,
  title = {AdjustAR: {AI}-{Driven} {In}-{Situ} {Adjustment} of {Site}-{Specific} {Augmented} {Reality} {Content}},
  shorttitle = {{{AdjustAR}}},
  booktitle = {Adjunct Proceedings of the 38th {{Annual ACM Symposium}} on {{User Interface Software}} and {{Technology}}},
  author = {Numan, Nels and Van Brummelen, Jessica and Lu, Ziwen and Steed, Anthony},
  year = {2025},
  month = sept,
  series = {{{UIST}} '25},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  doi = {10.1145/3746058.3758362},
}