Depth-map generation for an input image using an example approximate depth-map associated with an example similar image

• US 10,194,137 B1
• Filed: 10/17/2016
• Issued: 01/29/2019
• Est. Priority Date: 11/02/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Mukherjee, Debargha, Wu, Chen, Wang, Meng
• Primary Examiner(s): Lee, Sarah
• Application Number:
  US15/295,944
• Publication Number:
  US 20190037197A1
• Time in Patent Office: 834 Days
• Field of Search: None
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A two-dimensional image to be converted to a first three-dimensional image may be received. A second three-dimensional image that is visually similar to the two-dimensional image that is to be converted may be identified. A feature-to-depth mapping function may be computed for the first three-dimensional image by using an approximate depth map of the second three-dimensional image that is visually similar to the two-dimensional image that is to be converted. The feature-to-depth mapping function may be applied to a plurality of pixels of the two-dimensional image to determine a depth value for the plurality of pixels of the two-dimensional image. The first three-dimensional image may be generated based on the depth values for the plurality of pixels of the two-dimensional image.

Claims

1. A method comprising:

receiving a two-dimensional image to be converted to a first three-dimensional image;

identifying a second three-dimensional image that is visually similar to the two-dimensional image that is to be converted, the second three-dimensional image sharing a threshold number of color features with the two-dimensional image;

identifying an approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image;

identifying a distribution of spatial variation of the second three-dimensional image;

computing, by a processing device, a plurality of feature-to-depth mapping functions for the first three-dimensional image by using the approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image, wherein a number of the plurality of feature-to-depth mapping functions that are computed is based on the distribution of the spatial variation of the second three-dimensional image;

applying the plurality of feature-to-depth mapping functions to a plurality of pixels of the two-dimensional image to determine depth values for the plurality of pixels of the two-dimensional image, wherein the plurality of feature-to-depth mapping functions determine the depth values for the plurality of pixels of the two-dimensional image based on a color value of each of the plurality of pixels; and

generating the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

2. The method of claim 1, further comprising:

dividing the second three-dimensional image into a plurality of regions, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the plurality of regions of the second three-dimensional image.

3. The method of claim 2, wherein the determination of the depth values is based on an interpolation of depth values associated with the plurality of feature-to-depth mapping functions for each of the different regions of the plurality of regions of the second three-dimensional image.

4. The method of claim 1, wherein applying the plurality of feature-to-depth mappings function comprises:

identifying a feature of each of the plurality of pixels of the two-dimensional image; and

calculating a corresponding depth value for each of the plurality of pixels of the two-dimensional image, wherein calculating the depth value for each of the plurality of pixels comprises computing a linear interpolation based on the calculated depth values for each of the plurality of pixels of the two-dimensional image.

5. The method of claim 1, wherein the plurality of feature-to-depth mapping functions are based on a color space.

6. The method of claim 5, wherein the color space is Y-Cb-Cr.

7. The method of claim 1, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the first three-dimensional image.

8. A system comprising:

a memory; and

a processing device, operatively coupled with the memory, to;

receive a two-dimensional image to be converted to a first three-dimensional image;

identify a second three-dimensional image that is visually similar to the two-dimensional image that is to be converted, the second three-dimensional image sharing a threshold number of color features with the two-dimensional image;

identify an approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image;

identify a distribution of spatial variation of the second three-dimensional image;

compute a plurality of feature-to-depth mapping functions for the first three-dimensional image by using the approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image, wherein a number of the plurality of feature-to-depth mapping functions that are computed is based on the distribution of the spatial variation of the second three-dimensional image;

apply plurality of feature-to-depth mapping functions to a plurality of pixels of the two-dimensional image to determine depth values for the plurality of pixels of the two-dimensional image, wherein plurality of feature-to-depth mapping functions determine the depth values for the plurality of pixels of the two-dimensional image based on a color value of each of the plurality of pixels; and

generate the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

9. The system of claim 8, wherein the processing device is further to:

divide the second three-dimensional image into a plurality of regions, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the plurality of regions of the second three-dimensional image.

10. The system of claim 9, wherein the determination of the depth values is based on an interpolation of depth values associated with the plurality of feature-to-depth mapping functions for each of the different regions of the plurality of regions of the second three-dimensional image.

11. The system of claim 8, wherein to apply the plurality of feature-to-depth mappings, the processing device is further to:

identify a feature of each of the plurality of pixels of the two-dimensional image; and

calculate a corresponding depth value for each of the plurality of pixels of the two-dimensional image, wherein calculating the depth value for each of the plurality of pixels comprises computing a linear interpolation based on the calculated depth values for each of the plurality of pixels of the two-dimensional image.

12. The system of claim 8, wherein the plurality of feature-to-depth mappings are based on a color space.

13. The system of claim 12, wherein the color space is Y-Cb-Cr.

14. A non-transitory machine-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform operations comprising:

receiving a two-dimensional image to be converted to a first three-dimensional image;

identifying a second three-dimensional image that is visually similar to the two-dimensional image that is to be converted, the second three-dimensional image sharing a threshold number of color features with the two-dimensional image;

identifying an approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image;

identifying a distribution of spatial variation of the second three-dimensional image;

computing a plurality of feature-to-depth mapping functions for the first three-dimensional image by using the approximate depth map of the second three-dimensional image that shares the threshold number of color features with the two-dimensional image, wherein a number of the plurality of feature-to-depth mapping functions that are computed is based on the distribution of the spatial variation of the second three-dimensional image;

applying the plurality of feature-to-depth mapping functions to a plurality of pixels of the two-dimensional image to determine depth values for the plurality of pixels of the two-dimensional image, wherein the plurality of feature-to-depth mapping functions determine the depth values for the plurality of pixels of the two-dimensional image based on a color value of each of the plurality of pixels; and

generating the first three-dimensional image based on the depth values for the plurality of pixels of the two-dimensional image.

15. The non-transitory machine-readable storage medium of claim 14, the operations further comprising:

dividing the second three-dimensional image into a plurality of regions, wherein each of the plurality of feature-to-depth mapping functions is associated with a different region of the plurality of regions of the second three-dimensional image.

16. The non-transitory machine-readable storage medium of claim 15, wherein the determination of the depth values is based on an interpolation of depth values associated with the plurality of feature-to-depth mapping functions for each of the different regions of the plurality of regions of the second three-dimensional image.

17. The non-transitory machine-readable storage medium of claim 14, wherein to apply the plurality of feature-to-depth mappings, the operations further comprise:

identifying a feature of each of the plurality of pixels of the two-dimensional image; and

calculating a corresponding depth value for each of the plurality of pixels of the two-dimensional image, wherein calculating the depth value for each of the plurality of pixels comprises computing a linear interpolation based on the calculated depth values for each of the plurality of pixels of the two-dimensional image.

18. The non-transitory machine-readable storage medium of claim 14, wherein the plurality of feature-to-depth mappings are based on a color space.