Estimating depth from a single image

• US 9,594,774 B2
• Filed: 01/13/2016
• Issued: 03/14/2017
• Est. Priority Date: 09/05/2013
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: Bhardwaj, Anurag, Baig, Mohammad Haris, Piramuthu, Robinson, Jagadeesh, Vignesh, Di, Wei
• Primary Examiner(s): Park, Edward
• Application Number:
  US14/994,459
• Publication Number:
  US 20160124995A1
• Time in Patent Office: 426 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

During a training phase, a machine accesses reference images with corresponding depth information. The machine calculates visual descriptors and corresponding depth descriptors from this information. The machine then generates a mapping that correlates these visual descriptors with their corresponding depth descriptors. After the training phase, the machine may perform depth estimation based on a single query image devoid of depth information. The machine may calculate one or more visual descriptors from the single query image and obtain a corresponding depth descriptor for each visual descriptor from the generated mapping. Based on obtained depth descriptors, the machine creates depth information that corresponds to the submitted single query image.

Claims

1. A method comprising:

accessing reference images and corresponding reference depth maps from a reference database, a first reference image corresponding to a first reference depth map and including a color pixel defined by at least three color values, the first reference depth map including a depth value that corresponds to the color pixel in the first reference image;

calculating visual descriptors and corresponding depth descriptors from the accessed reference images and their corresponding reference depth maps;

generating a matrix that correlates the calculated visual descriptors with their calculated corresponding depth descriptors, the generating of the matrix being performed by a processor of a machine;

receiving a query image that depicts a surface of a physical item;

calculating a visual descriptor from the received query image;

obtaining a depth descriptor that corresponds to the calculated visual descriptor from the generated matrix;

creating a depth map that corresponds to the query image based on the obtained depth descriptor that corresponds to the visual descriptor calculated from the query image;

subdividing the query image that depicts the surface into superpixels; and

modifying the created depth map of the query image by modifying an orientation of a plane represented by a superpixel among the superpixels based on a summation of depth values of color pixels in the superpixel and based on a cardinality of the depth values of the color pixels in the superpixel.

2. The method of claim 1, wherein:

the receiving of the query image receives the query image without any corresponding depth map.

3. The method of claim 1, wherein:

the reference images and the query image are red-green-blue (RGB) images that contain only RGB values and are devoid of depth values.

4. The method of claim 1, wherein:

the reference images are reference RGB images; and

the accessing of the reference images and corresponding depth maps includes accessing reference red-green-blue-depth (RGB-D) images from the reference database, each reference RGB-D image including one of the reference RGB images and its corresponding reference depth map.

5. The method of claim 1, wherein:

the receiving of the query image receives the query image as part of a request to estimate depth information solely from the query image; and

the creating of the depth map that corresponds to the query image is in response to the request to estimate the depth information.

6. The method of claim 1, wherein:

the modifying of the created depth map includes assigning a constant depth value to each pixel within a superpixel in the query image.

7. The method of claim 1, wherein:

the modifying of the created depth map includes modifying the orientation of the plane represented by the superpixel in the query image in accordance with a random sample consensus (RANSAC) algorithm.

8. The method of claim 1, wherein:

the first reference depth map that corresponds to the first reference image is a first reference depth image that includes a depth pixel that is defined by the depth value and corresponds to the color pixel in the first reference image.

9. The method of claim 1, wherein:

the query image includes camera information;

the created depth map includes a three-dimensional representation of the surface of the physical item whose surface is depicted in the query image; and

the method further comprises generating a three-dimensional model of the surface of the physical item based on the camera information included in the query image and based on the created depth map that corresponds to the query image that depicts the physical item.

10. The method of claim 9 further comprising:

providing the generated three-dimensional model to a three-dimensional rendering engine to create a three-dimensional visualization of the surface of the physical item.

11. The method of claim 9, wherein:

the generated three-dimensional model is a three-dimensional cloud of points among which are points that represent the surface of the physical item; and

the method further comprises calculating a length of the surface of the physical item based on the generated three-dimensional cloud of points.

12. The method of claim 11, wherein:

the physical item depicted in the query image is a shippable item; and

the method further comprises providing the calculated length of the surface of the shippable item to a shipping application.

13. A system comprising:

one or more processors;

a database trainer module that configures at least one processor among the one or more processors to;

access reference images and corresponding reference depth maps from a reference database, a first reference image corresponding to a first reference depth map and including a color pixel defined by at least three color values, the first reference depth map including a depth value that corresponds to the color pixel in the first reference image;

calculate visual descriptors and corresponding depth descriptors from the accessed reference images and their corresponding reference depth maps; and

generate a matrix that correlates the calculated visual descriptors with their calculated corresponding depth descriptors; and

a depth map module that configures at least one processor among the one or more processors to;

receive a query image that depicts a surface of a physical item;

calculate a visual descriptor from the received query image;

obtain a depth descriptor that corresponds to the calculated visual descriptor from the matrix generated by the trainer module;

create a depth map that corresponds to the query image based on the obtained depth descriptor that corresponds to the visual descriptor calculated from the query image;

subdivide the query image that depicts the surface into superpixels; and

modify the created depth map of the query image by modifying an orientation of a plane represented by a superpixel among the superpixels based on a summation of depth values of color pixels within the superpixel and based on a cardinality of the depth values of the color pixels in the superpixel.

14. The system of claim 13, wherein the depth map module further configures the at least one processor to:

receive the query image as part of a request to estimate depth information solely from the query image; and

create the depth map that corresponds to the query image in response to the request to estimate the depth information.

15. The system of claim 13, wherein the depth map module further configures the at least one processor to:

partition the query image into superpixels; and

modify the created depth map that corresponds to the query image based on the superpixels in the query image.

16. The system of claim 13, wherein:

the query image includes camera information;

the created depth map includes a three-dimensional representation of the surface of the physical item whose surface is depicted in the query image; and

the system further comprises a visualization module configured to generate a three-dimensional model of the surface of the physical item based on the camera information included in the query image and based on the created depth map that corresponds to the query image that depicts the physical item.

17. The system of claim 16, wherein:

the physical item depicted in the query image is a shippable item;

the generated three-dimensional model is a three-dimensional cloud of points among which are points that represent the surface of the shippable item; and

the system further comprises a shipping module configured to;

calculate a length of the surface of the shippable item based on the generated three-dimensional cloud of points; and

provide the calculated length of the surface of the shippable item to a shipping application.

18. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing reference images and corresponding reference depth maps from a reference database, a first reference image corresponding to a first reference depth map and including a color pixel defined by at least three color values, the first reference depth map including a depth value that corresponds to the color pixel in the first reference image;

calculating visual descriptors and corresponding depth descriptors from the accessed reference images and their corresponding reference depth maps;

generating a matrix that correlates the calculated visual descriptors with their calculated corresponding depth descriptors;

receiving a query image that depicts a surface of a physical item;

calculating a visual descriptor from the received query image;

obtaining a depth descriptor that corresponds to the calculated visual descriptor from the generated matrix;

creating a depth map that corresponds to the query image based on the obtained depth descriptor that corresponds to the visual descriptor calculated from the query image;

subdividing the query image that depicts the surface into superpixels; and

modifying the created depth map of the query image by modifying an orientation of a plane represented by a superpixel among the superpixels based on a summation of depth values of color pixels within the superpixel and based on a cardinality of the depth values of the color pixels in the superpixel.

19. The non-transitory machine-readable storage medium of claim 18, wherein:

the query image includes camera information;

the created depth map includes a three-dimensional representation of the surface of the physical item whose surface is depicted in the query image; and

the operations further comprise generating a three-dimensional model of the surface of the physical item based on the camera information included in the query image and based on the created depth map that corresponds to the query image that depicts the physical item.