Insights from the Edge of Immersive AI

Field notes from H3M’s exploration of mixed reality, neural media, generative AI and embodied computation: deep dives on Gaussian splatting, NeRFs, AI world generation and spatial interface design.

Latest research

• Persistent AR: Stateful Spatial Worlds, Built on a Phone

  What would it take for anyone to build a spatial world on a phone and find it exactly where they left it? Inside Portals' approach to persistent augmented reality: creation without 3D skills, scenes stored as intent rather than geometry, and real-time Gaussian splats on edge devices.

• Text-to-World Generation: Designed Worlds from a Sentence

  Can a sentence compose a 3D world to an art director's standard? How H3M's worldgen system turns a prompt into a lit, finished space: looks stored as data, a catalogue of 1,576 reusable components, and an AI critic whose feedback became 45 art-direction rules.

• Semantic Memory: AI That Cites Its Source

  Language models rebuild meaning from raw text on every query, and their answers drift from their sources. CSFN, an H3M research project, compiles knowledge once into typed objects tied to exact evidence. In a controlled test world, a 5,152-parameter reasoner matched a frozen LLM baseline on 36 of 37 held-out tasks.

• Latent Compute: Devices That Learn What They Can Sense

  Phones and laptops carry sensors and telemetry that operating systems routinely average away. Latent Compute is an H3M research direction asking whether a small on-device language model performs better with context from the device it runs on. The test is designed, with seven gates set before building, and has not yet been run.

• The Immediacy of Imagination: AI Forges Living Worlds

  The accelerating pace of AI image generation is collapsing the distance between a spark of an idea and a fully realized spatial experience. This shift unlocks a future where imagination manifests instantly, transforming our everyday environments into dynamic, living worlds through Portals.

• The Living Canvas: AI, Wearables, and the Emergence of a Spatial Economy

  A new economic paradigm is unfolding, driven by the convergence of AI's generative power and spatial computing's ability to merge digital with physical. Wearable tech will be the gateway, and Portals the canvas where these AI-born worlds inhabit reality, democratizing creation and transforming our shared spaces into an infinitely rich, interactive medium.

• The Emergence of Living Worlds: Where AI Imagination Steps into Reality

  As AI models advance from static image generation to dynamic, temporal video, they're not just creating content to be watched. They are fabricating the very essence of living entities and evolving scenarios. Portals provides the canvas for these AI-generated imaginations to breathe, move, and co-inhabit our physical spaces, democratizing worldbuilding and igniting an entirely new creative economy.

• The Illumination of Imagination: How AI Lights the Path for Worlds in Reality

  We stand at the precipice of a new era where the imagined and the real seamlessly merge. The subtle yet profound understanding of light, once a barrier to believable spatial experiences, is now being unlocked by intelligent systems. This convergence, powered by AI's ability to interpret and generate light, transforms how worlds emerge into our physical spaces through Portals, making every shared reality feel intrinsically alive and perfectly integrated.

• The Dawn of Instant Reality and the Future of Spatial Creation

  Tencent's Hunyuan3D-2 represents a significant leap in text-to-3D generation, offering rapid, high-fidelity asset creation. At H3M, we see this as a pivotal technology for dynamically populating our Portals, enabling unprecedented levels of personalized and emergent immersive experiences.

• Sharpening Perception for Truly Immersive Spatial Computing

  We delve into BiRefNet, a state-of-the-art salient object detection model. Its capability to precisely identify and delineate critical objects within complex visual scenes is a game-changer for enhancing the realism and interactivity of H3M's immersive Portals, driving us closer to seamless neural media experiences.

• Unlocking High-Fidelity 3D Human Mesh Recovery with Diffusion Models

  This article explores ScoreHMR, a cutting-edge implementation of 3D Human Mesh Recovery (HMR) utilizing score-based diffusion models to infer 3D human shape and pose from 2D images. By framing HMR as an iterative denoising process, ScoreHMR offers unprecedented fidelity and diversity in mesh generation, pushing beyond traditional deterministic approaches. This advancement holds significant implications for spatial computing, AR/XR, and the creation of highly realistic digital avatars.

• The Dawn of Scalable Zero-Shot Image-to-Video Synthesis

  MimicMotion represents a significant breakthrough in AI-driven video generation, leveraging essential diffusion models to achieve scalable, zero-shot image-to-video synthesis. This framework empowers users to animate static reference images into dynamic video sequences using various motion guidance inputs. Its accessibility and robust performance promise to revolutionize content creation across diverse industries, from entertainment to spatial computing.

• Pioneering Dynamic Human Reconstruction with Neural Radiance Fields for Immersive Futures

  GVHMR (Generative Volumetric Human Mesh Reconstruction) represents a significant leap in capturing and recreating dynamic human forms. By leveraging advanced 'Dynamic Ray Radiance Fields,' this framework can reconstruct highly realistic 3D human models, complete with intricate movements and nuanced appearances, from potentially limited input. Its implications are profound, spanning enhanced AR/XR experiences, realistic digital twins, and the next generation of spatial computing interfaces.

• Democratizing High-Fidelity 3D Avatar Generation with AI

  ExAvatar-jupyter represents a significant leap in accessible 3D avatar creation, leveraging advanced AI models to generate realistic digital humans from simple text prompts or single 2D images. This open-source project streamlines the complex process of 3D asset generation, offering a user-friendly framework for developers and content creators alike. Its integration within a Jupyter Notebook environment promises to accelerate innovation across spatial computing, gaming, and the burgeoning metaverse.

• Democratizing Dynamic Narratives: Unpacking StableAnimator-Jupyter's Role in AI-Driven Animation

  StableAnimator-Jupyter represents a significant leap in accessible AI video generation, transforming complex Stable Diffusion models into a user-friendly animation platform via Google Colab. This innovation democratizes the creation of dynamic visual content, enabling artists and researchers to rapidly prototype and generate high-quality animations with unprecedented ease. By integrating advanced control mechanisms and specialized models, it paves the way for new creative workflows and immersive experiences in the rapidly evolving landscape of neural media.

• Real-Time Depth Reconstruction from Gaussian Splats: Advancing Spatial Computing and Immersive Experiences

  Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, but its inherent structure also presents a potent, albeit less explored, pathway for real-time depth map extraction. This article delves into the technical mechanisms and recent advancements enabling the generation of high-fidelity depth information from 3DGS representations, critically impacting applications in spatial computing and immersive AR/XR environments. We explore the methodology, challenges, and transformative potential of leveraging 3DGS for dynamic scene understanding.

• Trellis Tiled Diffusion: Unlocking High-Resolution Generative AI Through Spatial Optimization

  Trellis Tiled Diffusion, as demonstrated by the `trellis-tost` project, introduces a sophisticated method for generating ultra-high-resolution images with Stable Diffusion XL by intelligently breaking down and reassembling latent space tiles. This innovative approach overcomes traditional VRAM limitations and significantly enhances global coherence across large-format outputs. By optimizing the diffusion process, it enables creators to produce visually stunning, detailed imagery previously unattainable on conventional hardware.

• HunyuanWorld-Voyager: Pioneering AI-Driven 3D World Generation for the Spatial Web

  Tencent Hunyuan's 'HunyuanWorld-Voyager' represents a significant breakthrough in AI-powered 3D content creation, enabling the rapid generation of complex, immersive virtual environments from simple text or image prompts. This sophisticated system leverages advanced neural rendering techniques to construct highly detailed and multi-view consistent worlds, poised to revolutionize content development across the metaverse, gaming, and extended reality domains. By dramatically lowering the barrier to entry for 3D content, Voyager accelerates the realization of truly dynamic and responsive spatial computing experiences.