Segmentation For Holographic Images

Patent I authored while working at Microsoft. Describes an image segmentation technique for producing photorealistic human holograms.

Introduction:

My 2019 U.S. Patent, “Segmentation for Holographic Images” describes a method of 2D image segmentation that - when used as a precursor to 3D reconstruction algorithms - greatly improves resulting hologram photorealism relative to previously established prior art such as this landmark 2015 paper by Collet et al. The invention enhances augmented reality applications such as using human avatars for communication or preservation purposes. This technology was used by Microsoft’s Mixed Reality Capture Studios.

The following image shows the improvement in human avatar quality achieved with the described approach:

Summary:

I summarize the patent here by reviewing the ideas illustrated in the figures:

Figure 1 describes the relationship between volumetric 3D reconstruction and the 2D images: a 3D volume can be calculated from many 2D silhouettes. Thus, to accurately reconstruct a 3D volume observed in 2D images, many accurate volume silhouettes must be first obtained in 2D.

Because humans are extremely good at identifying other humans, producing human holograms with even slight volumetric errors is immersion-breaking for a consumer of augmented reality content. Figure 2 communicates that previous techniques used to obtain silhouettes were not sufficiently accurate for volumetric hologram applications at the time of this patent’s writing. In the absence of extremely well-labeled and diverse training data, even neural networks did not produce silhouettes accurate enough for human hologram applications. Traditional segmentation techniques like background subtraction are also defeated when the RGB intensities of background pixels are very similar to the RGB intensities of the target volume.

Figure 3 describes the approach proposed in this patent: combining (1) foreground-background subtraction, (2) neural network based semantic segmentation, and (3) statistical learning based postprocessing to produce highly refined silouettes for volumetric reconstruction.

Figure 5 outlines the algorithm pipeline proposed in this patent in text form:

Figure 6 shows the final payoff of the proposed approach in terms of hologram quality. The top image is an input to the hologram reconstruction algorithm described by Collet et al. The lower left image is a view of the hologram that is produced with traditional silhouette calculation approaches. The lower right image is a view of the hologram that is produced using the approach described in this patent. The improvement in reconstruction quality is drastic.