Depth map generation from a monoscopic image based on combined depth cues

• US 9,098,911 B2
• Filed: 11/01/2012
• Issued: 08/04/2015
• Est. Priority Date: 11/01/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wu, Chen, Mukherjee, Debargha, Wang, Meng
• Primary Examiner(s): PRINCE, JESSICA MARIE
• Application Number:
  US13/666,566
• Publication Number:
  US 20140118494A1
• Time in Patent Office: 1,006 Days
• Field of Search: 348/44
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

A combined depth map is generated for a monoscopic image based on a weighted combination of a color depth map, a spatial depth map, and a motion depth map for the image, each describing the depth of pixels in the image relative to the image plane. The color depth map determines the depth of pixels according to their color, the spatial depth map determines the depth of pixels according to their location in the image, and the motion depth map determines the depth of pixels according to their motion in the image. Each of the depth maps is associated with a weight that is used to generate the weighted combination. The weights are adaptive to account for variation between different monoscopic images.

Claims

1. A method for generating a depth map of an image, the method comprising:

accessing the image, the image comprising a plurality of pixels, wherein each pixel has a color and a location in the image;

determining a color depth map for the image based upon the colors of the pixels in the image;

determining a spatial depth map for the image based upon the locations of pixels and historical depth information for pixels in similar locations in a plurality of other images;

determining a motion depth map for the image based upon pixel motion in the image;

determining a color depth map weight, a spatial depth map weight, and a motion depth map weight, the color depth map weight determined by;

analyzing a distribution of the colors of the pixels in the image, and

determining the color depth map weight based on the distribution of the colors; and

generating a combined depth map for the image from a combination of the color depth map weighted by the color depth map weight, the spatial depth map weighted by the spatial depth map weight, and the motion depth map weighted by the motion depth map weight.

2. The method of claim 1, wherein determining the spatial depth map weight comprises:

determining historical depth variance information for the pixels in similar locations in a plurality of other images, the historical depth variance information describing a variance of the historical depth information; and

determining the spatial depth map weight based on the historical depth variance information.

3. The method of claim 1, wherein determining the motion depth map weight comprises determining a percentage of pixels in the image having local motion, the motion depth map weight based on the percentage of pixels having local motion.

4. The method of claim 1, wherein determining the color depth map weight based on the distribution of the colors described by the histogram comprises:

determining an entropy associated with the histogram, the entropy based on the distribution of colors;

determining a ratio of the entropy to a maximum entropy associated with the image, the ratio describing a relative distribution of the colors; and

determining the color depth map weight based on the ratio, the color depth map weight directly proportional to the ratio.

5. The method of claim 2, wherein determining the spatial depth map weight based on the historical depth variance information comprises:

retrieving historical depth variance information associated with a location in the image;

determining a first multiplier, the first multiplier having a value greater than one;

determining a second multiplier, the second multiplier having a value less than one;

comparing the historical depth variance information associated with the location to a variance threshold;

wherein determining the spatial depth map weight comprises;

responsive to a determination that the historical depth variance information associated with the location is above the variance threshold, multiplying the spatial depth map weight by the first multiplier; and

responsive to a determination that the historical depth variance information associated with the location is below the variance threshold, multiplying the spatial depth map weight by the second multiplier.

6. The method of claim 3, further comprising:

determining a motion depth map weight for a second image based on a percentage of pixels in the second image having local motion, the second image prior to the first image in a video sequence;

determining a first multiplier, the first multiplier having a value greater than one;

determining a second multiplier, the second multiplier having a value less than one; and

comparing the percentage of pixels in the first image having local motion to a motion threshold;

wherein determining the motion depth map weight for the first image comprises;

responsive to a determination that the percentage of pixels in the first image having local motion is above the motion threshold, multiplying the motion depth map weight for the second image by the first multiplier; and

responsive to a determination that the percentage of pixels in the first image having local motion is below the motion threshold, multiplying the motion depth map weight for the second image by the second multiplier.

7. A non-transitory computer-readable storage medium storing computer program instructions for generating a depth map of an image, the computer program instructions executable to perform steps comprising:

accessing the image, the image comprising a plurality of pixels, wherein each pixel has a color and a location in the image;

determining a color depth map for the image based upon the colors of the pixels in the image;

determining a spatial depth map for the image based upon the locations of pixels and historical depth information for pixels in similar locations in a plurality of other images;

determining a motion depth map for the image based upon pixel motion in the image;

determining a color depth map weight, a spatial depth map weight, and a motion depth map weight, the color depth map weight determined by;

analyzing a distribution of the colors of the pixels in the image, and

determining the color depth map weight based on the distribution of the colors; and

generating a combined depth map for the image from a combination of the color depth map weighted by the color depth map weight, the spatial depth map weighted by the spatial depth map weight, and the motion depth map weighted by the motion depth map weight.

8. The non-transitory computer-readable storage medium of claim 7, wherein determining the spatial depth map weight comprises:

determining historical depth variance information for the pixels in similar locations in a plurality of other images, the historical depth variance information describing a variance of the historical depth information; and

determining the spatial depth map weight based on the historical depth variance information.

9. The non-transitory computer-readable storage medium of claim 7, wherein determining the motion depth map weight comprises determining a percentage of pixels in the image having local motion, the motion depth map weight based on the percentage of pixels having local motion.

10. The non-transitory computer-readable storage medium of claim 7, wherein determining the color depth map weight based on the distribution of the colors described by the histogram comprises:

determining an entropy associated with the histogram, the entropy based on the distribution of colors;

determining a ratio of the entropy to a maximum entropy associated with the image, the ratio describing a relative distribution of the colors; and

determining the color depth map weight based on the ratio, the color depth map weight directly proportional to the ratio.

11. The non-transitory computer-readable storage medium of claim 8, wherein determining the spatial depth map weight based on the historical depth variance information comprises:

retrieving historical depth variance information associated with a location in the image;

determining a first multiplier, the first multiplier having a value greater than one;

determining a second multiplier, the second multiplier having a value less than one;

comparing the historical depth variance information associated with the location to a variance threshold;

wherein determining the spatial depth map weight comprises;

responsive to a determination that the historical depth variance information associated with the location is above the variance threshold, multiplying the spatial depth map weight by the first multiplier; and

responsive to a determination that the historical depth variance information associated with the location is below the variance threshold, multiplying the spatial depth map weight by the second multiplier.

12. The non-transitory computer-readable storage medium of claim 9, the steps further comprising:

determining a motion depth map weight for a second image based on a percentage of pixels in the second image having local motion, the second image prior to the first image in a video sequence;

determining a first multiplier, the first multiplier having a value greater than one;

determining a second multiplier, the second multiplier having a value less than one;

comparing the percentage of pixels in the first image having local motion to a motion threshold; and

wherein determining the motion depth map weight for the first image comprises;

responsive to a determination that the percentage of pixels in the first image having local motion is above the motion threshold, multiplying the motion depth map weight for the second image by the first multiplier; and

responsive to a determination that the percentage of pixels in the first image having local motion is below the motion threshold, multiplying the motion depth map weight for the second image by the second multiplier.

13. A system for generating a depth map of an image, comprising:

a non-transitory computer-readable storage medium storing computer program instructions executable to perform steps comprising;

accessing the image, the image comprising a plurality of pixels, wherein each pixel has a color and a location in the image;

determining a color depth map for the image based upon the colors of the pixels in the image;

determining a spatial depth map for the image based upon the locations of pixels and historical depth information for pixels in similar locations in a plurality of other images;

determining a motion depth map for the image based upon pixel motion in the image;

determining a color depth map weight, a spatial depth map weight, and a motion depth map weight, the color depth map weight determined by;

analyzing a distribution of the colors of the pixels in the image, and

determining the color depth map weight based on the distribution of the colors; and

generating a combined depth map for the image from a combination of the color depth map weighted by the color depth map weight, the spatial depth map weighted by the spatial depth map weight, and the motion depth map weighted by the motion depth map weight; and

a processor for executing the computer program instructions.

14. The system of claim 13, wherein determining the spatial depth map weight comprises:

determining historical depth variance information for the pixels in similar locations in a plurality of other images, the historical depth variance information describing a variance of the historical depth information; and

determining the spatial depth map weight based on the historical depth variance information.

15. The system of claim 13, wherein determining the motion depth map weight comprises determining a percentage of moving pixels in the image, the motion depth map weight based on the percentage of moving pixels.

16. The system of claim 13, wherein determining the color depth map weight based on the distribution of the colors described by the histogram comprises:

determining an entropy associated with the histogram, the entropy based on the distribution of colors;

determining a ratio of the entropy to a maximum entropy associated with the image, the ratio describing a relative distribution of the colors; and

determining the color depth map weight based on the ratio, the color depth map weight directly proportional to the ratio.

17. The system of claim 14, wherein determining the spatial depth map weight based on the historical depth variance information comprises:

retrieving historical depth variance information associated with a location in the image;

determining a first multiplier, the first multiplier having a value greater than one;

determining a second multiplier, the second multiplier having a value less than one;

comparing the historical depth variance information associated with the location to a variance threshold;

wherein determining the spatial depth map weight comprises;

responsive to a determination that the historical depth variance information associated with the location is above the variance threshold, multiplying the spatial depth map weight by the first multiplier; and

responsive to a determination that the historical depth variance information associated with the location is below the variance threshold, multiplying the spatial depth map weight by the second multiplier.