ABSTRACT

While large language models (LLMs) have accelerated traditional software development through intent-driven “vibe coding”, prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic “Reality Model” that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages LLMs to translate high-level prompts (e.g., “create a dandelion that reacts to my hand”) directly into functional, physics-aware mixed-reality applications. To minimize the friction of on-device testing, the workflow introduces a seamless desktop “simulated reality” to headset deployment loop. Finally, we introduce VCXR60, a pilot dataset of 60 XR prompts paired with an automated evaluation pipeline. Our technical evaluation demonstrates high one-shot execution success, enabling practitioners to bypass low-level hurdles and rapidly move from “idea to reality”.

Full-text PDF Publisher page arXiv page XR Blocks Gem Try at Google AI Studio Code


Overview

Example user journey of Vibe Coding XR, an end-to-end workflow for creating immersive AI + XR experiences via vibe coding: (A) User types “create a beautiful dandelion” with XR Blocks Gem (xrblocks.github.io/gem) on a Galaxy XR headset in Chrome. (B) Gemini translates the input into an interactive XR application within a minute, while the user reviews its reasoning and coding process. (C) User selects the “Enter XR” button and sees an animated dandelion dispersing upon pinch.

Example user journey of Vibe Coding XR, an end-to-end workflow for creating immersive AI + XR experiences via vibe coding: (A) User types “create a beautiful dandelion” with XR Blocks Gem (xrblocks.github.io/gem) on a Galaxy XR headset in Chrome. (B) Gemini translates the input into an interactive XR application within a minute, while the user reviews its reasoning and coding process. (C) User selects the “Enter XR” button and sees an animated dandelion dispersing upon pinch.


CITING

@inproceedings{duVibeCodingXR2026,
  title = {Vibe Coding {XR}: Accelerating {AI} + {XR} Prototyping with {XR} {Blocks} and {Gemini}},
  booktitle = {Proceedings of the 32nd {ACM} {Symposium} on {Virtual Reality Software and Technology}},
  author = {Du, Ruofei and Hersh, Benjamin and Li, David and Numan, Nels and Qian, Xun and Chen, Yanhe and Zhou, Zhongyi and Ren, Jiahao and Chen, Xingyue and Bettridge, Robert Timothy and Chen, Xiang 'Anthony' and Toh, Steve and Faruqi, Faraz and Kim, David},
  year = {2026},
  month = nov,
  series = {VRST '26},
  pages = {1--11},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  isbn = {979-8-4007-2811-2/2026/11},
  doi = {10.1145/3822517.3848668}
}