REAL-TIME MOBILE DEVICE CAPTURE AND GENERATION OF AR/VR CONTENT

• US 20170148223A1
• Filed: 02/08/2017
• Published: 05/25/2017
• Est. Priority Date: 10/31/2014
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: HOLZER, Stefan Johannes Josef, Ren, Yuheng, Kar, Abhishek, Chande, Krunal Ketan, Saelzle, Martin Josef Nikolaus, Rusu, Radu Bogdan, Trevor, Alexander Jay Bruen
• Granted Patent: US 10,719,939 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/738 Presentation of query results
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Various embodiments describe systems and processes for generating AR/VR content. In one aspect, a method for generating a three-dimensional (3D) projection of an object is provided. A sequence of images along a camera translation may be obtained using a single lens camera. Each image contains at least a portion of overlapping subject matter, which includes the object. The object is semantically segmented from the sequence of images using a trained neural network to form a sequence of segmented object images, which are then refined using fine-grained segmentation. On-the-fly interpolation parameters are computed and stereoscopic pairs are generated for points along the camera translation from the refined sequence of segmented object images for displaying the object as a 3D projection in a virtual reality or augmented reality environment. Segmented image indices are then mapped to a rotation range for display in the virtual reality or augmented reality environment.

Claims

1. A method for generating a three-dimensional (3D) projection of an object in a virtual reality or augmented reality environment, the method including:

obtaining a sequence of images using a single lens camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;

semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images;

refining the sequence of segmented object images using fine-grained segmentation;

computing on-the-fly interpolation parameters;

generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in the virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and

mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.

2. The method of claim 1, wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.

3. The method of claim 1, further comprising stabilizing the sequence of images using camera rotation and focal length values before semantically segmenting the object from the sequence of images.

4. The method of claim 1, wherein on-the-fly interpolation parameters are used to generate interpolated images along any point in the camera translation in real-time.

5. The method of claim 1, wherein a stereoscopic pair includes an interpolated virtual image comprising two selected frames.

6. The method of claim 5, wherein the selected frames are modified by rotating an image of a selected frame such that the image corresponds to a view of the object angled directly toward the object.

7. The method of claim 1, wherein the sequence of segmented object images are fused to generate a projection of the object, the projection depicting a 3D view of the object without polygon generation.

8. The method of claim 1, wherein refining the sequence of segmented object images includes using a temporal conditional random field.

9. The method of claim 8, wherein using the temporal conditional random field includes utilizing a graph of neighboring images for each image to be refined.

10. The method of claim 1, wherein mapping segmented image indices includes mapping physical viewing locations to a frame index.

11. A system for generating a three-dimensional (3D) projection of an object in a virtual reality or augmented reality environment, the system comprising:

a single lens camera for obtaining a sequence of images, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;

a display module;

a processor, and

memory storing one or more programs configured for execution by the processor, the one or more programs comprising instructions for;

semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images;

refining the sequence of segmented object images using fine-grained segmentation;

computing on-the-fly interpolation parameters;

generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in the virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and

mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.

12. The system of claim 11, wherein each image in the sequence of images is captured at a predetermined distance along the camera translation.

13. The system of claim 11, further comprising stabilizing the sequence of images using camera rotation and focal length values before semantically segmenting the object from the sequence of images.

14. The system of claim 11, wherein on-the-fly interpolation parameters are used to generate interpolated images along any point in the camera translation in real-time.

15. The system of claim 11, wherein a stereoscopic pair includes an interpolated virtual image comprising two selected frames.

16. The system of claim 15, wherein the selected frames are modified by rotating an image of a selected frame such that the image corresponds to a view of the object angled directly toward the object.

17. The system of claim 11, wherein the sequence of segmented object images are fused to generate a projection of the object, the projection depicting a 3D view of the object without polygon generation.

18. The system of claim 11, wherein refining the sequence of segmented object images includes using a temporal conditional random field.

19. The system of claim 18, wherein using the temporal conditional random field includes utilizing a graph of neighboring images for each image to be refined.

20. A non-transitory computer readable medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining a sequence of images using a single lens camera, the sequence of images being captured along a camera translation, wherein each image in the sequence of images contains at least a portion of overlapping subject matter, the subject matter including the object;

semantically segmenting the object from the sequence of images using a trained neural network to form a sequence of segmented object images;

refining the sequence of segmented object images using fine-grained segmentation;

computing on-the-fly interpolation parameters;

generating stereoscopic pairs from the refined sequence of segmented object images for displaying the object as a 3D projection in a virtual reality or augmented reality environment, the stereoscopic pairs being generated for one or more points along the camera translation; and

mapping segmented image indices to a rotation range for display in the virtual reality or augmented reality environment.