Explore

System and Method for Real-Time 3D Scene Reconstruction and Navigation Using 360 camera and 3D Model - LIVE 3D

ABSTRACT

A system and method are disclosed for creating immersive, real-time, interactive 3D virtual environments by dynamically integrating live video feeds into a 3D model. The technology involves a specialised camera and advanced software that processes multiple video streams, applying deep learning techniques for depth and motion estimation to produce a spatially accurate 3D scene, allowing users to navigate and interact with the environment as if they were physically present.

We have developed a special-purpose camera (

POLYTRON.AI⁠

) for digital twin to operate in REAL-TIME (LIVE 3D™), creating immersive, interactive real-time digital twin.

We merge various video feeds from

Polytron.AI⁠

cameras using an advanced fusion technique to create a unified 3D scene reconstruction, thus facilitating the creation of a real-time, immersive LIVE 3D experience.

FIELD OF INVENTION

The present invention relates generally to the field of digital imaging and 3D modeling, and more specifically, to a system and method for integrating live video feeds into 3D environments for real-time navigation and interaction. The invention pertains to the fields of digital imaging, computer vision, spatial computing, and 3D modeling, particularly to systems and methods for enhancing real-time interaction within virtual environments through the integration of live video data.

BACKGROUND OF INVENTION

Traditional virtual tours and 3D navigations are often limited by pre-scanned, static 2D imagery, which does not reflect real-time changes in the environment and lacks interactivity. There is a need for a more dynamic and interactive system that can provide real-time (LIVE), accurate 3D representation of an environment for various applications, including virtual reality, security surveillance, and remote touring.

SUMMARY OF INVENTION

In today's rapidly advancing technological landscape, the concept of digital exploration is being revolutionised by the introduction of LIVE 3D virtual tours. Accessible directly through a web browser, this innovative technology brings an unprecedented level of realism to virtual experiences. Unlike traditional virtual tours that rely on static images or pre-rendered visuals, LIVE 3D incorporates real-time video within a fully interactive 3D environment. As users navigate through these meticulously crafted scenes, they witness a dynamic and responsive world where everything unfolds in real time—from the bustling activities of people to the natural movements of trees and shifting patterns of light and shadows. This immersive experience not only blurs the lines between virtual and physical realms but also enhances user engagement, making one feel genuinely present in the explored location. LIVE 3D represents a groundbreaking shift in how we interact with and experience spaces, offering a unique, immersive way to explore the world without ever leaving the comfort of your device.

The invention provides a system and method for real-time integration of LIVE video feeds into a 3D environment, using a special-purpose camera (www.polytron.ai) and a suite of software services. The system includes a deep learning architecture that processes video feeds to estimate depth and motion, creating a dynamic 3D scene that users can navigate in real time. The solution enhances user engagement and presence, offering a more realistic and immersive experience.

Stitching video textures into 3D virtual tours involves the process of integrating video content onto 3D models to create immersive experiences. The use of 3D models in virtual tours allows for a more realistic and compelling presentation of properties, products, or environments, enhancing the overall user experience, and conveying a more visceral sense of “being there” while navigating in 3D Tour.This represents a groundbreaking advancement, unmatched by any current virtual tour technology available.

POLYTRON suite of LIVE 3D Software and Services is enabling conventional web-based virtual tour with interactive real-time scene, making you feel like you’re standing right where you took them without physically being there. Immersive, spatial, 3D, real-time, and interactive, visualised across a multitude of devices (phones, tablets, PC).

DETAILED DESCRIPTION OF THE INVENTION

System Architecture:

360 Camera Technology: Description of the special-purpose camera designed for real-time video capture and integration into 3D models. We have developed a special-purpose camera (PATENT-PROTECTED) for digital twin to operate in REAL-TIME (LIVE 3D™), creating immersive, interactive real-time digital twin. This is accomplished by merging various video feeds from

Polytron.AI⁠

cameras using an advanced fusion technique to create a unified 3D scene reconstruction, thus facilitating the creation of a real-time, immersive LIVE 3D experience.

⁠

LIVE 3D™ Video Fusion: Techniques used to merge multiple video streams into a coherent 3D scene. In a nutshell, LIVE 3D™ is the "Fusing Dynamic 3D Geometry and Video Textures for REAL-TIME Navigation" - an end-to-end deep learning architecture for predicting depth from video/imagery and creating LIVE 3D spatial geometry.

⁠

This innovative method combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. Our algorithm interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. With this patented method, we are able to create street-level, outdoor & indoor depth data, geometry, point clouds

LIVE 3D™ Video - Stitching video textures into 3D virtual tours involves the process of integrating video content onto 3D models to create immersive experiences. The use of 3D point clouds, geometry and panoramic videos in virtual tours allows for a more realistic and compelling presentation of properties, products, or environments, enhancing the overall user experience, and conveying a more visceral sense of “being there” while navigating in 3D Tour. This represents a groundbreaking advancement, unmatched by any current virtual tour technology available.

⁠

With the Polytron LIVE 3D™ Camera, the era of limiting virtual tours to pre-scanned, static 2D imagery is over. We present to you a dynamic canvas of the world as it unfolds, enabling you to immerse yourself and navigate in real-time within an accurately modeled 3D environment. Our groundbreaking technology intricately stitches live video feeds directly onto the surfaces of a geometrically precise 3D mesh model, transforming them into video textures that animate the scene with real-world dynamics. This innovation isn't merely a new lens through which to view the world — it's a completely new way to experience and interact with it. Step into the vanguard of virtual exploration with us, where each tour becomes a LIVE, interactive event, and every moment unfolds a fresh discovery, brought to life within the spatial fidelity of a meticulously crafted 3D scene.

Deep Learning Framework: Details on the neural network design, including modules for depth prediction and motion estimation.

Method of Operation:

Data Collection: The process involves capturing video data using a Polytron 4K 360-degree camera, which is then streamed in real-time into a 3D scene. This innovative approach merges LIVE video footage with 3D digital twin, allowing for dynamic and immersive visualisations. As the 360-degree camera captures a complete panoramic view, it provides a comprehensive perspective of the surroundings. This footage is seamlessly integrated into the 3D scene at 60 FPS, ensuring that viewers can experience a REAL-TIME, realistic and interactive depiction of the environment.

This technology not only enhances the user experience by offering a more engaging and interactive way to explore a location but also proves crucial in applications such as virtual tours, real-time surveillance, and augmented reality systems. The real-time streaming capability means that changes in the physical environment are immediately reflected in the 3D scene, providing up-to-date information that is vital for decision-making in various professional fields, including urban planning, security, and entertainment.

Data Processing: The algorithms and methods used to transform raw video feeds from a 360-degree camera into a structured 3D scene involve dynamically updating the textures with video frames. This sophisticated process, referred to as "3D Video Fusion - High-Performance Adaptive Video Texture Streaming and Rendering of Large 3D Scenes", seamlessly integrates video content into a 3D mesh. Key components of this technique include depth estimation, advanced spatial analysis such as segmentation, and motion tracking.

Depth Estimation: This process involves determining the distance between the camera and objects in the video by calculating depth and detecting shapes. This is achieved by comparing differences between views from two cameras, a technique known as stereo vision. By analyzing these differences, or disparities, the system can construct a depth map that represents the distance of objects from the camera.

Segmentation: Additionally, segmentation is performed on the depth map to distinguish and isolate various shapes and objects within the scene. Segmentation algorithms classify different regions of the depth map, enabling the identification of distinct objects and their boundaries. This comprehensive approach not only enhances the accuracy of depth perception but also improves the recognition and understanding of complex scenes by dividing the video into segments that represent different objects or regions. Advanced machine learning models, such as convolutional neural networks (CNNs), are employed to recognise and categorise different areas of the image according to their characteristics.

Motion Tracking: This process involves monitoring the movement of objects over a sequence of video frames. It is accomplished using advanced algorithms designed to detect and track various points or features as they traverse through both space and time.

These algorithms identify distinctive elements within the video frames, such as edges, corners, or textures, and then follow these features across successive frames. By continuously analyzing the positions of these features, the system can accurately determine the trajectory and motion of objects within the scene.

Optical flow algorithms estimate the motion between two frames by observing the changes in pixel intensity, while feature-based tracking methods, such as the Kanade-Lucas-Tomasi (KLT) tracker, identify and follow key points through a video sequence. These techniques are essential for LIVE 3D applications, where understanding the dynamics of moving objects is crucial to “super-impose” the dynamically generated animated 3D scene.

By leveraging these AI tracking algorithms on video streams culled from

Polytron.AI⁠

cameras, the system can generate a comprehensive map of object movements, meshing these animated objects into the 3D scene, and also enhancing its ability to predict future positions and interactions within the environment. This capability is vital for developing REAL-TIME LIVE 3D systems and applications.

LIVE 3D Scene Reconstruction: Steps involved in mapping video textures onto a 3D mesh and point cloud to create interactive, real-time environments.

Use Cases:

Application scenarios including virtual tours, security monitoring, and augmented reality experiences.

CLAIMS

A method for real-time video feed processing to create a 3D navigable environment, comprising:

Steps for capturing, processing, and integrating live video into a 3D model.

A system comprising:

A specialised camera, processing hardware, and software for executing the method described.

The method according to claim 1, wherein the deep learning architecture is used for motion and depth estimation from the video feeds.

CONCLUSION

This invention provides a significant advancement in the field of spatial computing by enabling live, interactive 3D navigation and visualisation of environments, thus overcoming limitations of existing technologies.

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.