Method for creating 3D virtual reality from 2D images

• US 9,407,904 B2
• Filed: 05/11/2015
• Issued: 08/02/2016
• Est. Priority Date: 05/01/2013
• Status: Expired due to Fees

• Current Assignee: Legend3D, Inc.
• Original Assignee: Legend3D, Inc.
• Inventors: Sandrew, Jared, Baldridge, Tony, McFarland, Jacqueline, Jones, Scott, Schad, Thomas
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/709,327
• Publication Number:
  US 20150249815A1
• Time in Patent Office: 449 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method that enables creation of a 3D virtual reality environment from a series of 2D images of a scene. Embodiments map 2D images onto a sphere to create a composite spherical image, divide the composite image into regions, and add depth information to the regions. Depth information may be generated by mapping regions onto flat or curved surfaces, and positioning these surfaces in 3D space. Some embodiments enable inserting, removing, or extending objects in the scene, adding or modifying depth information as needed. The final composite image and depth information are projected back onto one or more spheres, and then projected onto left and right eye image planes to form a 3D stereoscopic image for a viewer of the virtual reality environment. Embodiments enable 3D images to be generated dynamically for the viewer in response to changes in the viewer'"'"'s position and orientation in the virtual reality environment.

Claims

1. A method for creating 3D virtual reality from 2D images comprising:

obtaining a plurality of 2D images of an environment from at least one camera;

stitching together said plurality of 2D images into one or more integrated 2D images of said environment;

projecting said one or more integrated 2D images onto a spherical surface, yielding a spherical surface image;

unwrapping said spherical surface image onto an unwrapped plane image;

dividing said unwrapped plane image into a plurality of regions;

assigning depth information to points of each of said plurality of regions; and

generating stereo images for a viewer at a viewer position and orientation in a virtual reality environment using said depth information and said unwrapped plane image;

wherein said assigning depth information to the points of each of said plurality of regions comprises

defining a flat or curved surface for one or more of said plurality of regions;

rotating and translating said flat or curved surface for one or more of said plurality of regions in three-dimensional space; and

,

obtaining said depth information from the three-dimensional space of the points on said flat or curved surface for one or more of said plurality of regions.

2. The method of claim 1, wherein said generating stereo images for said viewer further comprises the steps:

projecting said unwrapped plane image and said depth information onto said spherical surface, yielding a modified spherical surface image and spherical depth information;

generating a left eye image and a right eye image for said viewer using said spherical depth information and using the location and orientation of said viewer in said virtual reality environment.

3. The method of claim 1, wherein said generating stereo images for said viewer further comprises the steps:

generating a left eye unwrapped plane image and a right eye unwrapped plane image using said depth information and said unwrapped plane image;

projecting said left eye unwrapped plane image onto a left eye spherical surface, yielding a left eye spherical surface image;

projecting said right eye unwrapped plane image onto a right eye spherical surface, yielding a right eye spherical surface image;

generating a left eye image for said viewer using said left eye spherical surface image and using a location and an orientation of said viewer in said virtual reality environment;

generating a right eye image for said viewer using said right eye spherical surface image and using the location and the orientation said viewer in said virtual reality environment.

4. The method of claim 1, further comprising

inserting one or more objects into said unwrapped plane image.

5. The method of claim 1, further comprising

removing one or more objects from said unwrapped plane image.

6. The method of claim 1, further comprising

extending one or more objects appearing in said unwrapped plane image.

7. The method of claim 1, wherein said dividing said unwrapped plane image into a plurality of regions further comprises accepting mask region inputs to define objects in said plurality of 2D images.

8. The method of claim 1, further comprising accepting external depth information and applying said external depth information to said plurality of regions.

9. The method of claim 8, further comprising adjusting said plurality of regions to eliminate depth noise.

10. A method for creating 3D virtual reality from 2D images comprising:

obtaining a plurality of 2D images of an environment from at least one camera;

stitching together said plurality of 2D images into one or more integrated 2D images of said environment;

projecting said one or more integrated 2D images onto a spherical surface, yielding a spherical surface image;

unwrapping said spherical surface image onto an unwrapped plane image;

dividing said unwrapped plane image into a plurality of regions,

wherein said dividing said unwrapped plane image into a plurality of regions further comprises accepting mask region inputs to define objects in said plurality of 2D images;

accepting external depth information and applying said external depth information to said plurality of regions;

obtaining at least one mask within each of said plurality of regions;

assigning depth information to points of each of said plurality of regions;

calculating a best fit for a plane using a computer based on depth associated with each of the at least one mask;

applying depth associated with the plane having the best fit to each of said plurality of regions;

generating stereo images for a viewer at a viewer position and orientation in a virtual reality environment using said depth information and said unwrapped plane image; and

,

altering automatically using said computer, any combination of position, orientation, shape, depth or curve of the plane in order to fit edges or corners of the plane with another plane.

11. The method of claim 10, wherein said generating stereo images for said viewer further comprises the steps:

projecting said unwrapped plane image and said depth information onto said spherical surface, yielding a modified spherical surface image and spherical depth information;

generating a left eye image and a right eye image for said viewer using said spherical depth information and using the location and orientation of said viewer in said virtual reality environment.

12. The method of claim 10, wherein said generating stereo images for said viewer further comprises the steps:

generating a left eye unwrapped plane image and a right eye unwrapped plane image using said depth information and said unwrapped plane image;

projecting said left eye unwrapped plane image onto a left eye spherical surface, yielding a left eye spherical surface image;

projecting said right eye unwrapped plane image onto a right eye spherical surface, yielding a right eye spherical surface image;

generating a left eye image for said viewer using said left eye spherical surface image and using a location and an orientation of said viewer in said virtual reality environment;

generating a right eye image for said viewer using said right eye spherical surface image and using the location and the orientation said viewer in said virtual reality environment.

13. The method of claim 10, wherein said assigning depth information to the points of each of said plurality of regions comprises:

defining a flat or curved surface for one or more of said plurality of regions;

rotating and translating said flat or curved surface for one or more of said plurality of regions in three-dimensional space;

obtaining said depth information from the three-dimensional space of the points on said flat or curved surface for one or more of said plurality of regions.

14. The method of claim 10, further comprising

inserting one or more objects into said unwrapped plane image.

15. The method of claim 10, further comprising

removing one or more objects from said unwrapped plane image.

16. The method of claim 10, further comprising

extending one or more objects appearing in said unwrapped plane image.

17. The method of claim 10, further comprising adjusting said plurality of regions to eliminate depth noise.