Integrating real world conditions into virtual imagery

• US 10,019,831 B2
• Filed: 10/20/2016
• Issued: 07/10/2018
• Est. Priority Date: 10/20/2016
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Champion, Clifford S., Tu, Jerome C.
• Primary Examiner(s): Merouan, Abderrahim
• Application Number:
  US15/298,956
• Publication Number:
  US 20180114353A1
• Time in Patent Office: 628 Days
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Systems and methods for incorporating real world conditions into a three-dimensional (3D) graphics object are described herein. In some embodiments, images of a physical location of a user of a three-dimensional (3D) display system may be received from at least one camera and a data imagery map of the physical location may be determined based at least in part on the received images. The data imagery map may capture real world conditions associated with the physical location of the user. Instructions to render a 3D graphics object may be generated and the data imagery map may be incorporated into a virtual 3D scene comprising the 3D graphics object, thereby incorporating the real world conditions into virtual world imagery. In some embodiments, the data imagery may include a light map, a sparse light field, and/or a depth map of the physical location.

Claims

1. A non-transitory computer readable memory medium storing programming instructions executable by a processor to:

receive, from at least one camera, one or more images of a physical environment of a user of a three-dimensional (3D) display system;

determine a data imagery map of the physical environment based at least in part on the one or more images, wherein the one or more images of the physical environment of the user includes the user;

receive instructions to render a virtual 3D scene on a display of the 3D display system; and

incorporate the data imagery map of the physical environment, including a rendering of a reflection of the user on a 3D object within the 3D scene, into the virtual 3D scene, thereby incorporating physical world conditions into virtual world imagery.

2. The non-transitory computer readable memory medium of claim 1,

wherein the data imagery map comprises a light map of the physical environment.

3. The non-transitory computer readable memory medium of claim 1,

wherein the data imagery map comprises a sparse light field of the physical environment.

4. The non-transitory computer readable memory medium of claim 1,

wherein the at least one camera comprises one or more video cameras.

5. The non-transitory computer readable memory medium of claim 1,

wherein the at least one camera comprises an array of cameras.

6. The non-transitory computer readable memory medium of claim 1,

wherein the program instructions are further executable to:

receive user position information from one or more tracking sensors;

determine a point of view of the user based in part of the position information; and

wherein to incorporate the data imagery map, the program instructions are further executable to incorporate the data imagery map based at least in part on the point of view of the user.

7. The non-transitory computer readable memory medium of claim 1,

wherein the one or more images comprises one or more sequences of images.

8. The non-transitory computer readable memory medium of claim 1,

wherein the one or more images comprises a stream of video images.

9. The non-transitory computer readable memory medium of claim 1,

wherein the rendering of the reflection of the user comprises rendering a distorted reflected image of the user.

10. The non-transitory computer readable memory medium of claim 1,

wherein the one or more images provide a 360 degree view of the physical environment of the user, and wherein, to incorporate the data imagery map of the physical environment into the virtual 3D scene, the program instructions are further executable to incorporate the 360 degree view of the physical environment of the user into the virtual 3D scene.

11. The non-transitory computer readable memory medium of claim 1,

wherein the real world condition associated with the physical environment of the user includes textures of physical objects, and wherein, to incorporate the data imagery map of the physical environment into the virtual 3D scene, the program instructions are further executable to render textures of the physical objects into the virtual 3D scene.

12. A three-dimensional (3D) display system comprising:

a memory;

at least one display; and

at least one processor in communication with the memory and at least one display, wherein the at least one processor is configured to;

receive, from at least one camera in communication with the processor, images of a physical environment of a user of the 3D display system, wherein the one or more images of the physical environment of the user includes the user;

determine, based at least in part on the received images, a data imagery map of the physical environment, wherein the data imagery map captures physical world conditions associated with the physical environment of the user;

generate instructions to render a virtual 3D scene on the at least one display; and

incorporate the data imagery map, including a rendering of a reflection of the user on a 3D object within the 3D scene, into the virtual 3D scene, thereby incorporating the physical world conditions into virtual world imagery.

13. The 3D display system of claim 12,

wherein the data imagery map comprises one of:

a light map of the physical environment;

a sparse light field of the physical environment;

or

a depth map of the physical environment.

14. The 3D display system of claim 12,

wherein the at least one camera comprises one of:

one or more video cameras;

or

an array of cameras.

15. The 3D display system of claim 12,

wherein the processor is further configured to:

receive user position information from one or more tracking sensors;

determine a point of view of the user based in part of the position information; and

wherein to incorporate the data imagery map, the processor is further configured to incorporate the data imagery map based at least in part on the point of view of the user.

16. The 3D display system of claim 15,

wherein the system further comprises the one or more tracking sensors, wherein the one or more tracking sensors are in communication with the processor.

17. The 3D display system of claim 12,

wherein the one or more images comprises:

one or more sequences of images;

or

a stream of video images.

18. The 3D display system of claim 12,

wherein the rendering of the reflection of the user comprises rendering a distorted reflected image of the user.

19. A method for incorporating real world conditions into a three-dimensional (3D) graphics object, the method comprising:

receiving, from at least one camera, images of a physical environment of a user of a three-dimensional (3D) display system, wherein the images of the physical environment of the user includes the user;

determining, based at least in part on the received images, a data imagery map of the physical environment, wherein the data imagery map captures real world conditions associated with the physical environment of the user;

generating instructions to render a 3D graphics object; and

incorporating the data imagery map, including a rendering of a reflection of the user on a 3D object within the 3D scene, into a virtual 3D scene comprising the 3D graphics object, thereby incorporating the real world conditions into virtual world imagery.

20. The method of claim 19,

wherein the data imagery map comprises one of:

a light map of the physical environment;

a sparse light field of the physical environment;

or

a depth map of the physical environment.