We present AdjustAR, a system for runtime AI-driven adjustment of
site-specific augmented reality content.
Model-based authoring is the most widely used approach for creating site-specific
AR experiences. For example, Unity with the Niantic SDK and systems like DistanciAR let
authors anchor virtual content to pre-captured 3D scans, which can later be used to localize
at the target site. These models support remote authoring by providing environmental
context without requiring physical presence, but assume the models remain accurate at deployment.
Prior work has identified that this is often not the case,
and that these models are frequently outdated or incomplete. This leads
to misalignments between virtual content and physical referents,
obscuring author intent and degrading user experience. CoCreatAR addressed this by enabling
live collaboration between a remote author and an on-site user to refine and extend AR experiences.
A different body of work, including system like SemanticAdapt, AUIT,
and ScalAR, aim to reduce dependence on these global models. They use semantic
associations and layout constraints to improve anchoring flexibility and
highlight the potential of strategies that go beyond surface geometry.
More recent work has explored multimodal large language models to enable anyone to create AR
content. For instance, ImaginateAR demonstrated outdoor in-situ authoring using natural language
instructions based on large language models and advanced scene understanding. However,
these systems focus on creation rather than runtime adaptation of existing experiences.
Building on these ideas, we ask: how can we create a site-specific AR system that 1) takes
advantage of existing model-based authoring approaches, and 2) helps ensure AR content
appears as intended despite changes in the target environment, without requiring re-authoring?
The typical workflow begins as follows. An author, working remotely with a computer,
uses 3D editing tools to place content into a pre-captured 3D model of the target site,
either downloaded from the internet or created themselves.
In this example, the author plans to create an AR story experience in a park and places
a virtual sign along a path and positions a character, "Mr. Raccoon," on a trash can.
The author assumes, or at least hopes, that these referents will be present at
deployment and uses the 3D model as a stand-in for the real environment.
Later, when a user tries the experience on-site, inconsistencies may appear. For
example, the trash can has been moved, and the raccoon now floats in mid-air.
To address this, AdjustAR enables runtime correction, triggered manually or automatically.
When activated, it captures the current camera view and outlines each AR element
with a unique color. It also caches the depth map along with camera intrinsics and extrinsics.
In parallel, the system renders a reference image from the same perspective using the
original authored model, with matching object outlines. This pair of images
represents a comparison between authored intent and the current scene as seen by the
user. Both are sent to a multimodal large language model. The model is
prompted to assess whether each AR element appears aligned with its intended referent.
If misalignment is detected, the model is asked
to provide a corrected 2D anchor point based on the provided images.
These corrections are projected into 3D using the cached depth map,
and the AR elements are updated accordingly.
The adjusted content is then shown in the
user’s AR view. In this way, AdjustAR restores semantic and spatial alignment
in correspondence with author intent at runtime, without manual re-authoring.
Here we show several other examples, from top-left to bottom-right:
Objects already aligned with the environment A referent that has moved
A missing referent A mix of correctly and incorrectly placed elements
Looking ahead, we will conduct formal evaluations to assess robustness and usability,
while also focusing on reducing latency and improving accuracy.
We also aim to extend spatial reasoning to address occlusion and off-frame referents.
Finally, we plan to support more flexible anchoring strategies beyond bottom-center
placement, and to incorporate author-defined semantic constraints.