Depth map generation based on soft classification

• US 8,897,542 B2
• Filed: 12/15/2011
• Issued: 11/25/2014
• Est. Priority Date: 12/15/2011
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Wei, Jianing
• Primary Examiner(s): DRENNAN, BARRY T
• Application Number:
  US13/327,733
• Publication Number:
  US 20130156294A1
• Time in Patent Office: 1,076 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/36 Indoor scenes
• CPC Class Codes: G06V 20/38 Outdoor scenes

Abstract

A method for generating a depth map for a 2D image and video includes receiving the 2D image and video; defining a plurality of object classes; analyzing content of the received 2D image and video; calculating probabilities that the received 2D image belongs to the object classes; and determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.

Claims

1. A computer-implemented method for generating a depth map for a 2D image, the method comprising:

receiving the 2D image;

defining a plurality of object classes;

analyzing content of the received 2D image by computing preliminary depth maps for the plurality of object classes based on features in the received 2D image and corresponding object class properties;

calculating probabilities that the received 2D image belongs to the object classes; and

determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.

2. The method of claim 1, wherein analyzing content comprises:

acquiring a set of visual features from the received 2D image; and

comparing the received 2D image with the object classes based on the acquired set of visual features.

3. The method of claim 1, wherein determining a final depth map comprises:

computing the final depth map as a weighted sum of the preliminary depth maps and the calculated probabilities for the object classes.

4. The method of claim 3, wherein computing the final depth map comprises:

calculating the products of the calculated probabilities and the preliminary depth maps for each of the object classes; and

summing the products for each of the object classes.

5. The method of claim 1, further comprising:

if there are exactly two object classes in the plurality of object classes;

performing a single stage two-class classification of the received 2D image; and

calculating the probabilities for the two object classes in a single stage.

6. The method of claim 1, further comprising:

if there are more than two object classes in the plurality of object classes;

performing a multi-stage two-class classification of the received 2D image;

calculating the probabilities for a first two object classes in a first stage;

calculating the probabilities for other object classes calculated in subsequent stages; and

combining the calculated probabilities to determine the probabilities that the received 2D image belongs to each of a plurality of combinations of object classes from the different stages.

7. The method of claim 6, wherein performing a multi-stage two-class classification of the received 2D image comprises:

performing a two-stage two-class classification by;

executing a first stage classification by;

calculating the probability that the received 2D image belongs to a first object class; and

calculating the probability that the received 2D image belongs to a second object class;

executing a second stage classification by;

calculating the probability that the received 2D image belongs to a third object class; and

calculating the probability that the received 2D image belongs to a fourth class; and

combining the probabilities calculated in the first and second stage classifications.

8. The method of claim 7, wherein combining the probabilities calculated in the first and second stage classifications comprises:

calculating, in the first and second stage classifications, the probability that the received 2D image belongs to each of a plurality of combinations of object classes.

9. The method of claim 1, comprising receiving a plurality of 2D images.

10. A computer-implemented method for generating a depth map for a 2D image, the method comprising:

receiving the 2D image;

defining a plurality of object classes, wherein the object classes comprise;

a landscape class;

a city class;

an indoor class; and

a portrait class;

analyzing content of the received 2D image;

calculating probabilities that the received 2D image belongs to the object classes; and

determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.

11. A computer-implemented method for generating a depth map for a 2D image, the method comprising:

receiving the 2D image;

defining a plurality of object classes;

analyzing content of the received 2D image;

calculating probabilities that the received 2D image belongs to the object classes;

determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes; and

applying an infinite impulse response filter to smooth the calculated probabilities for the object classes if the received 2D image is a single image of a fast action scene.

12. A computer-implemented method for generating a depth map for a 2D image, the method comprising:

receiving the 2D image;

defining a plurality of object classes;

analyzing content of the received 2D image;

calculating probabilities that the received 2D image belongs to the object classes;

determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes; and

calculating the probability that the received 2D image belongs to each of a plurality of combinations of the object classes by determining the product of a calculated probability for each object class of the object classes in each of a plurality of combinations of the object classes.

13. A system for generating a depth map for a received 2D image, the system comprising:

a user computing device; and

a non-transitory computer-readable medium operatively coupled to the user computing device via one or more networks and storing an image converter, the image converter comprising;

an image content analyzer to analyze content of the received 2D image;

a class probability calculator to calculate the probability that the received 2D image belongs to each of a plurality of object classes;

a depth map generator to compute preliminary depth maps for the object classes using configuration information stored in an image database and

a final depth map generator to determine a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.

14. The system of claim 13, wherein the image converter further comprises:

an image database to store configuration information pertaining to the object classes for analyzing content of the received 2D image.

15. The system of claim 14, wherein the image converter further comprises:

an image database to store configuration information pertaining to properties corresponding to the object classes.

16. A system for generating a depth map for a received 2D image, the system comprising:

a user computing device; and

a non-transitory computer-readable medium operatively coupled to the user computing device via one or more networks and storing an image converter, the image converter comprising;

an image content analyzer to analyze content of the received 2D image;

a class probability calculator to calculate the probability that the received 2D image belongs to each of a plurality of object classes;

a final depth map generator to determine a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes; and

an infinite impulse response filter to smooth the calculated probabilities for the object classes if the received 2D image is a single image of a fast action scene.

17. A non-transitory computer-readable medium storing instructions that, when executed, causes a computer to perform a method for generating a depth map for a 2D image, the method comprising:

receiving the 2D image;

defining a plurality of object classes;

analyzing content of the received 2D image by computing preliminary depth maps for the plurality of object classes based on features in the received 2D image and corresponding object class properties;

calculating probabilities that the received 2D image belongs to the object classes; and

determining a final depth map based on a result of the analyzed content and the calculated probabilities for the object classes.