System and method for real-time depth modification of stereo images of a virtual reality environment

• US 9,288,476 B2
• Filed: 08/17/2015
• Issued: 03/15/2016
• Est. Priority Date: 02/17/2011
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Jared, Baldridge, Tony, Lopez, Anthony, McFarland, Jacqueline, Jones, Scott, Schad, Thomas
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/827,422
• Publication Number:
  US 20150358612A1
• Time in Patent Office: 211 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Enables real-time depth modifications to stereo images of a 3D virtual reality environment locally and for example without iterative workflow involving region designers or depth artists that includes re-rendering these images from the original 3D model. Embodiments generate a spherical translation map from the 3D model of the virtual environment; this spherical translation map is a function of the pixel shifts between left and right stereo images for each point of the sphere surrounding the viewer of the virtual environment. Modifications may be made directly to the spherical translation map, and applied directly to the stereo images, without requiring re-rendering of the scene from the complete 3D model. This process enables depth modifications to be viewed in real-time, greatly improving the efficiency of the 3D model creation, review, and update cycle.

Claims

1. A method for real-time depth modification of stereo images of a virtual reality environment comprising

obtaining or creating a three-dimensional model of a virtual reality environment;

generating a spherical translation map from said three-dimensional model, wherein

said spherical translation map assigns translation values to points on a spherical surface in said three-dimensional model with a user located at a center of said spherical surface;

each of said translation values is a function of a distance of a closest object in said three-dimensional model from said user along a ray between said user and each of said points on said spherical surface;

obtaining a pose of said user relative to said virtual reality environment;

rendering a left viewpoint image of said three-dimensional model from said pose of said user;

rendering a right viewpoint image of said three-dimensional model from said pose of said user;

applying one or more translation modifications to said spherical translation map to adjust a depth of one or more objects in said three-dimensional model relative to said user;

modifying said left viewpoint image and said right viewpoint image to reflect said translation modifications without re-rendering said left viewpoint image and said right viewpoint image from said three-dimensional model.

2. The method of claim 1 wherein said spherical translation map is a parallax map that comprises horizontal pixel offset values between a left and right stereoscopic pair of images.

3. The method of claim 1 wherein said spherical translation map is a depth map.

4. The method of claim 1 wherein said spherical translation map is a U map or a UV map.

5. The method of claim 1 wherein both said left viewpoint image and said right viewpoint image are panoramic images.

6. The method of claim 5 wherein said panoramic images each comprise a 360 degree horizontal field of view.

7. The method of claim 6 wherein said panoramic images each comprise a 180 degree vertical field of view.

8. The method of claim 1 wherein said obtaining or creating said three-dimensional model of said virtual reality environment comprises

obtaining a plurality of 2D images of said virtual reality environment from at least one camera;

stitching together said plurality of 2D images into one or more integrated 2D images of said virtual reality environment;

projecting said one or more integrated 2D images onto said spherical surface, yielding a spherical surface image;

unwrapping said spherical surface image onto an unwrapped plane image;

dividing said unwrapped plane image into a plurality of regions;

assigning depth information to the points of each of said plurality of regions to create said three-dimensional model.

9. The method of claim 7 wherein said assigning depth information to the points of each of said plurality of regions comprises:

defining a flat or curved surface for one or more of said plurality of regions;

rotating and translating said flat or curved surfaces in three-dimensional space;

obtaining said depth information from the three-dimensional position of the points on said flat or curved surfaces.

10. The method of claim 7, further comprising

inserting one or more objects into said unwrapped plane image.

11. The method of claim 7, further comprising

removing one or more objects from said unwrapped plane image.

12. The method of claim 7, further comprising

extending one or more objects appearing in said unwrapped plane image.

13. The method of claim 8, wherein said dividing said unwrapped plane image into a plurality of regions further comprises accepting mask region inputs to define objects in said plurality of 2D images.

14. The method of claim 7, further comprising accepting external depth information and applying said external depth information to said plurality of regions.

15. The method of claim 7, further comprising adjusting said plurality of regions to eliminate depth noise.

16. The method of claim 7 wherein said obtaining or creating said three-dimensional model of said virtual reality environment further comprises

adding one or more computer-generated elements to said three-dimensional model;

assigning depth information to each of said one or more computer-generated elements.

17. The method of claim 1 wherein said obtaining or creating said three-dimensional model of said virtual reality environment comprises

obtaining depth information for one or more of said one or more objects from a second user.

18. The method of claim 1 wherein said obtaining or creating said three-dimensional model of said virtual reality environment comprises

accepting external depth information and applying said external depth information to one or more of said one or more objects.

19. The method of claim 8 wherein said assigning depth information to the points of each of said plurality of regions to create said three-dimensional model comprises

assigning said depth information based on underlying image data of one or more of

said plurality of 2D images,

said one or more integrated 2D images,

said spherical surface image, and,

said unwrapped plane image.

20. A system configured to perform real-time depth modification of stereo images of a virtual reality environment comprising at least one computer configured to:

obtain or create a three-dimensional model of a virtual reality environment;

generate a spherical translation map from said three-dimensional model, wherein

said spherical translation map assigns translation values to points on a spherical surface in said three-dimensional model with a user located at a center of said spherical surface;

each of said translation values is a function of a distance of a closest object in said three-dimensional model from said user along a ray between said user and each of said points on said spherical surface;

obtain a pose of said user relative to said virtual reality environment;

render a left viewpoint image of said three-dimensional model from said pose of said user;

render a right viewpoint image of said three-dimensional model from said pose of said user;

apply one or more translation modifications to said spherical translation map to adjust a depth of one or more objects in said three-dimensional model relative to said user;

modify said left viewpoint image and said right viewpoint image to reflect said translation modifications without re-rendering said left viewpoint image and said right viewpoint image from said three-dimensional model.