Depth-map generation for an input image using an example approximate depth-map associated with an example similar image

• US 9,471,988 B2
• Filed: 05/23/2012
• Issued: 10/18/2016
• Est. Priority Date: 11/02/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mukherjee, Debargha, Wu, Chen, Wang, Meng
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Le, Sarah
• Application Number:
  US13/478,449
• Publication Number:
  US 20130106837A1
• Time in Patent Office: 1,609 Days
• Field of Search: 345/419
• US Class Current: 1/1
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

An image converter receives a two-dimensional image to be converted to a first three-dimensional image. The image converter computes a feature-to-depth mapping function based on an approximate depth map associated with a second three-dimensional image. The image converter applies the feature-to-depth mapping function to a plurality of pixels of the two-dimensional image to determine a depth value for each of the plurality of pixels and generates the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

Claims

1. A method comprising:

receiving a two-dimensional image to be converted to a first three-dimensional image;

identifying, in a data store, a second three-dimensional image that corresponds to a stereoscopic pair of images and is visually similar with the two-dimensional image to be converted, wherein the second three-dimensional image was previously created from a conversion of another two-dimensional image or was initially captured as a three-dimensional image;

computing, by a processing device, a feature-to-depth mapping function for the first three-dimensional image by averaging depth values of at least a portion of a plurality of pixels from an approximate depth map associated with the previously created second three-dimensional image that corresponds to the stereoscopic pair of images and is visually similar with the two-dimensional image to be converted;

applying the feature-to-depth mapping function to a plurality of pixels of the two-dimensional image to determine a depth value for each of the plurality of pixels; and

generating the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

2. The method of claim 1, wherein the feature-to-depth mapping function associates a plurality of pixels in the second three-dimensional image having a first color value with a depth value.

3. The method of claim 1, further comprising:

computing a plurality of feature-to-depth mapping functions associated with the second three-dimensional image, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the second three-dimensional image.

4. The method of claim 1, wherein computing the feature-to-depth mapping function comprises:

identifying the second three-dimensional image;

identifying a plurality of bins within a feature space;

associating a plurality of pixels of the second three-dimensional image with the plurality of bins;

determining a depth value for each of the plurality of bins; and

determining a weight value for each of the plurality of bins.

5. The method of claim 1, wherein applying the feature-to-depth mapping function comprises:

identifying a feature of each of the plurality of pixels of the two-dimensional image;

calculating a depth value of each of the plurality of pixels of the two-dimensional image; and

computing an n-linear interpolation of the depth values for the plurality of pixels of the two-dimensional image.

6. The method of claim 1, wherein the feature-to-depth mapping function is based on a color space.

7. The method of claim 6, wherein the color space is Y-Cb-Cr.

8. A system comprising:

a processing device; and

a memory coupled to the processing device; and

an image converter, executed by the processing device from the memory, to;

receive a two-dimensional image to be converted to a first three-dimensional image;

identify, in a data store, a second three-dimensional image that corresponds to a stereoscopic pair of images and is visually similar with the two-dimensional image to be converted, wherein the second three-dimensional image was previously created from a conversion of another two-dimensional image or was initially captured as a three-dimensional image;

compute a feature-to-depth mapping function for the first three-dimensional image by averaging depth values of at least a portion of a plurality of pixels from an approximate depth map associated with the previously created second three-dimensional image that corresponds to the stereoscopic pair of images and is visually similar with the two-dimensional image to be converted;

apply the feature-to-depth mapping function to a plurality of pixels of the two-dimensional image to determine a depth value for each of the plurality of pixels; and

generate the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

9. The system of claim 8, wherein the feature-to-depth mapping function associates a plurality of pixels in the second three-dimensional image having a first color value with a depth value.

10. The system of claim 8, the image converter further to:

compute a plurality of feature-to-depth mapping functions associated with the second three-dimensional image, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the second three-dimensional image.

11. The system of claim 8, wherein when computing the feature-to-depth mapping function, the image converter to:

identify the second three-dimensional image;

identify a plurality of bins within a feature space;

associate a plurality of pixels of the second three-dimensional image with the plurality of bins;

determine a depth value for each of the plurality of bins; and

determine a weight value for each of the plurality of bins.

12. The system of claim 8, wherein when applying the feature-to-depth mapping function, the image converter to:

identify a feature of each of the plurality of pixels of the two-dimensional image;

calculate a depth value of each of the plurality of pixels of the two-dimensional image; and

compute an n-linear interpolation of the depth values for the plurality of pixels of the two-dimensional image.

13. The system of claim 8, wherein the feature-to-depth mapping function is based on a color space.

14. The system of claim 13, wherein the color space is Y-Cb-Cr.

15. A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:

receiving a two-dimensional image to be converted to a first three-dimensional image;

identifying, in a data store, a second three-dimensional image that corresponds to a stereoscopic pair of images and is visually similar with the two-dimensional image to be converted, wherein the second three-dimensional image was previously created from a conversion of another two-dimensional image or was initially captured as a three-dimensional image;

computing, by the processing device, a feature-to-depth mapping function for the first three-dimensional image by averaging depth values of at least a portion of a plurality of pixels from an approximate depth map associated with the previously created second three-dimensional image that corresponds to the stereoscopic pair of images and is visually similar with the two-dimensional image to be converted;

applying the feature-to-depth mapping function to a plurality of pixels of the two-dimensional image to determine a depth value for each of the plurality of pixels; and

generating the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

16. The non-transitory machine-readable storage medium of claim 15, wherein the feature-to-depth mapping function associates a plurality of pixels in the second three-dimensional image having a first color value with a depth value.

17. The non-transitory machine-readable storage medium of claim 15, the operations further comprising:

computing a plurality of feature-to-depth mapping functions associated with the second three-dimensional image, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the second three-dimensional image.

18. The non-transitory machine-readable storage medium of claim 15, wherein computing the feature-to-depth mapping function comprises:

identifying the second three-dimensional image;

identifying a plurality of bins within a feature space;

associating a plurality of pixels of the second three-dimensional image with the plurality of bins;

determining a depth value for each of the plurality of bins; and

determining a weight value for each of the plurality of bins.

19. The non-transitory machine-readable storage medium of claim 15, wherein applying the feature-to-depth mapping function comprises:

identifying a feature of each of the plurality of pixels of the two-dimensional image;

calculating a depth value of each of the plurality of pixels of the two-dimensional image; and

computing an n-linear interpolation of the depth values for the plurality of pixels of the two-dimensional image.

20. The non-transitory machine-readable storage medium of claim 15, wherein the feature-to-depth mapping function is based on a color space.