ChicMR: Immersive Mixed Reality System using Video-See-Thru HMD and 3D LiDAR scanner

Abstract

In this paper, we propose a mixed-reality system that provides an immersive interaction space for both real and virtual objects. Using a custom-made video-see-thru (VST) HMD, the system streams high-resolution stereo images of what the user is currently looking at. Also, the proposed system includes a rotating 2D LiDAR that configures a geometry of the user's surrounding space. Based on the triangle mesh approach, the proposed system creates an interaction space from the point cloud data. With the constructed interaction space, the user experiences various interactions using both real and virtual objects in the mixed-reality environment. We demonstrate the feasibility and effectiveness of our system with a real-time application.