Systems and methods for generating 360 degree mixed reality environments

• US 9,704,298 B2
• Filed: 06/21/2016
• Issued: 07/11/2017
• Est. Priority Date: 06/23/2015
• Status: Active Grant

• Current Assignee: BowFlex, Inc.
• Original Assignee: Paofit Holdings Pte. Ltd. (BowFlex, Inc.)
• Inventors: Espeset, Tonny, Frank, Andrei Richard, Hardy, Marc Scott
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US15/188,356
• Publication Number:
  US 20160379415A1
• Time in Patent Office: 385 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals

Abstract

Systems and methods for generating a 360 degree mixed virtual reality environment that provides a 360 degree view of an environment in accordance with embodiments of the invention are described. In a number of embodiments, the 360 degree mixed virtual reality environment is obtained by (1) combining one or more real world videos that capture images of an environment with (2) a virtual world environment that includes various synthetic objects that may be placed within the real world clips. Furthermore, the virtual objects embedded within the 360 degree mixed reality environment interact with the real world objects depicted in the real world environment to provide a realistic mixed reality experience.

Claims

1. A system for visualizing controllable virtual 3D objects within a mixed reality application using real-world video captured from a plurality of cameras, comprising:

a processor;

memory including a mixed reality application; and

wherein the mixed reality application directs the processor to;

obtain a plurality of real-world videos captured by one or more cameras, each real-world video capturing a different portion of a surrounding real-world environment as the one or more cameras move through the environment;

for each real-world video, extract information comprising camera movement coordinates information, path coordinates information, and point cloud coordinates information, including a depth of objects shown in the real-world video and translating the information into three dimensional (3D) coordinates;

for each real-world video, generate a 3D mixed reality environment comprising a plurality of separate, synched layers that includes (1) the real-world video as a background layer of the 3D mixed reality environment, (2) an occlusion layer that includes one or more transparent 3D objects that replicate real-world objects, including movement and rotation of the real-world objects, within the real-world video, and (3) one or more virtual synthetic objects, wherein the virtual synthetic objects interact with the transparent 3D objects based on 3D space locations of the objects and wherein the occlusion layer is used as a guide for the virtual synthetic objects to appear to move within the same environment as the real-world objects and to hide any virtual synthetic object that appears behind a transparent 3D object based on the depth information extracted from the real-world video; and

combine at least one 3D mixed reality environment generated for a real-world video with a different 3D mixed reality environment generated for a different real-world video to provide a 3D mixed reality environment that replicates a larger portion of the surrounding real-world environment for use by the mixed reality application, wherein frames of the background layer of each real-world video are combined using the camera movement coordinates information of each real-world video.

2. The system of claim 1, wherein combining the at least one 3D mixed reality environment generated for the real-world video with the different 3D mixed reality environment generated for the different real-world video comprises stitching together frames from the different real-world video clips and combining the background and occlusion layers of each 3D mixed reality environment generated for each real-world video.

3. The system of claim 1, wherein the instructions when executed further configure the one or more processors to combine a plurality of 3D mixed reality environments to provide a complete 360 degree view of the surrounding real-world environment.

4. The system of claim 1, wherein the instructions when executed further configure the one or more processors to detect a user changing a view beyond a threshold and rendering a different 3D mixed reality environment generated for a real-world video corresponding to the current view.

5. The system of claim 1, wherein the instructions when executed further configure the one or more processors to store depth information of real-world objects within a video frame within a subset of pixels of the particular frame.

6. The system of claim 1, wherein the instructions when executed further configure the one or more processors to store motion vectors that describe movement of pixels between frames of video within a subset of pixels within a frame.

7. The system of claim 1, wherein a background layer of a 3D environment is one of a plurality of layers synched in the 3D environment.

8. The system of claim 1, wherein the instructions when executed further configure the one or more processors to render a different real-world video clip within the background layer when a user changes a direction of a path in the real-world video.

9. A non-transitory computer-readable medium containing computer-executable instructions that, when executed by a hardware processor, cause the hardware processor to perform a method for rendering a mixed reality environment, the method comprising:

obtaining a plurality of real-world videos captured by one or more cameras, each real-world video capturing a different portion of a surrounding real-world environment as the one or more cameras move through the environment;

for each real-world video, extracting information comprising camera movement coordinates information, path coordinates information, and point cloud coordinates information, including a depth of objects shown in the real-world video and translating the information into three dimensional (3D) coordinates;

for each real-world video, generating a 3D mixed reality environment comprising a plurality of separate, synched layers that includes (1) the real-world video as a background layer of the 3D mixed reality environment, (2) an occlusion layer that includes one or more transparent 3D objects that replicate real-world objects, including movement and rotation of the real-world objects, within the real-world video, and (3) one or more virtual synthetic objects, wherein the virtual synthetic objects interact with the transparent 3D objects based on 3D space locations of the objects, and wherein the occlusion layer is used as a guide for the virtual synthetic objects to appear to move within the same environment as the real-world objects and to hide any virtual synthetic object that appears behind a transparent 3D object based on the depth information extracted from the real-world video; and

combining at least one 3D mixed reality environment generated for a real-world video with a different 3D mixed reality environment generated for a different real-world video to provide a 3D mixed reality environment that replicates a larger portion of the surrounding real-world environment for use by the mixed reality application, wherein frames of the background layer of each real-world video are combined using the camera movement coordinates information of each real-world video.

10. The non-transitory computer-readable medium of claim 9, wherein combining the at least one 3D mixed reality environment generated for the real-world video with the different 3D mixed reality environment generated for the different real-world video comprises stitching together frames from the different real-world video clips and combining the background and occlusion layers of each 3D mixed reality environment generated for each real-world video.

11. The non-transitory computer-readable medium of claim 9, wherein the method further comprises combining a plurality of 3D mixed reality environments to provide a complete 360 degree view of the surrounding real-world environment.

12. The non-transitory computer-readable medium of claim 9, wherein the method further comprises detecting a user changing a view beyond a threshold and rendering a different 3D mixed reality environment generated for a real-world video corresponding to the current view.

13. The non-transitory computer-readable medium of claim 9, wherein the method further comprises storing depth information of real-world objects within a video frame within a subset of pixels of the particular frame.

14. The non-transitory computer-readable medium of claim 9, wherein the method further comprises storing motion vectors that describe movement of pixels between frames of video within a subset of pixels within a frame.

15. The non-transitory computer-readable medium of claim 9, wherein a background layer of a 3D environment is one of a plurality of layers synched in the 3D environment.

16. The non-transitory computer-readable medium of claim 9, wherein the method further comprises rendering a different real-world video clip within the background layer when a user changes a direction of a path in the real-world video.

17. The non-transitory computer readable medium of claim 9, wherein the virtual synthetic objects interact with the transparent 3D objects based on 3D space locations of the objects such that virtual synthetic objects that are located behind transparent 3D objects based on the 3D space locations of the objects are occluded by the background layer.

18. The non-transitory computer readable medium of claim 9, wherein the virtual synthetic object is an avatar of a user, wherein the avatar moves based on activities of the user.