SENSOR FUSION FOR DEPTH ESTIMATION

• US 20140368615A1
• Filed: 06/12/2013
• Published: 12/18/2014
• Est. Priority Date: 06/12/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: van Baar, Jeroen, Beardsley, Paul A., Pollefeys, Marc, Gross, Markus
• Granted Patent: US 9,424,650 B2
• Time in Patent Office: Days
• US Class Current: 348/48
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

To generate a pixel-accurate depth map, data from a range-estimation sensor (e.g., a time-of flight sensor) is combined with data from multiple cameras to produce a high-quality depth measurement for pixels in an image. To do so, a depth measurement system may use a plurality of cameras mounted on a support structure to perform a depth hypothesis technique to generate a first depth-support value. Furthermore, the apparatus may include a range-estimation sensor which generates a second depth-support value. In addition, the system may project a 3D point onto the auxiliary cameras and compare the color of the associated pixel in the auxiliary camera with the color of the pixel in reference camera to generate a third depth-support value. The system may combine these support values for each pixel in an image to determine respective depth values. Using these values, the system may generate a depth map for the image.

Claims

1. A method for calculating a depth value for a pixel in a reference image, the method comprising:

receiving the reference image captured by a reference camera and at least one auxiliary image captured by an auxiliary camera;

receiving a first support value indicating whether the pixel in the reference image is at a particular depth, relative to the reference camera, based on comparing a region of the auxiliary image captured by the auxiliary camera with a region of the reference image captured by the reference camera;

providing a depth estimate of the pixel from a range-estimation camera;

receiving a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing the depth estimate from the range-estimation camera to the particular depth;

receiving a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fusing, by operation of one or more computer processors, the first, second, and third support values to generate a total support value for the pixel at the particular depth.

2. The method of claim 1, further comprising:

calculating respective first, second, and third support values for the pixel at each of a plurality of discrete depths that divide a depth volume; and

fusing the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

selecting the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

3. The method of claim 2, further comprising:

selecting respective depth values for a plurality of pixels in the reference image based on calculating first, second, and third support values at each of the plurality of descrete depths; and

generating a depth map associated with the reference image using the respective depth values.

4. The method of claim 1, wherein calculating the first support value comprises:

projecting the auxiliary image, at the particular depth, onto the reference image; and

determining the first support value based on comparing a feature of a plurality of pixels in the reference image to a feature of a corresponding plurality of pixels in the reference image.

5. The method of claim 1, wherein calculating the third support value comprises:

determining the third support value by comparing the color of the pixel in the reference image with a corresponding pixel in the auxiliary image.

6. The method of claim 1, further comprising:

generating a depth map by selecting respective depth values for a plurality of pixels in the image;

forming a super pixel comprising a plurality of pixels in the reference image based on features extracted from the reference image and a corresponding thermal image;

determining a depth-plane of the super pixel based on the depth values of the plurality of pixels; and

adjusting at least one of the depth values based on the depth-plane.

7. The method of claim 1, wherein the reference camera, auxiliary camera, and ToF camera are mounted on a common support structure.

8. A computer program product for calculating a depth value for a pixel in a reference image, the computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to;

receive the reference image captured by a reference camera and at least one auxiliary image captured by an auxiliary camera;

receive a first support value indicating whether the pixel in the reference image is at a particular depth, relative to the reference camera, based on comparing a region of the auxiliary image with a region of the reference image;

provide a depth estimate of the pixel from a range-estimation camera;

receive a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing the depth estimate from the range-estimation camera to the particular depth;

receive a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fuse the first, second, and third support values to generate a total support value for the pixel at the particular depth.

9. The computer program product of claim 8, wherein the computer-readable program code is further configured to:

calculate respective first, second, and third support values for the pixel for each of a plurality of discrete depths that divide a depth volume; and

fuse the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

select the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

10. The computer program product of claim 9, wherein the computer-readable program code is further configured to:

select respective depth values for a plurality of pixels in the reference image based on calculating first, second, and third support values at each of the plurality of descrete depths; and

generate a depth map associated with the reference image using the respective depth values.

11. The computer program product of claim 8, wherein calculating the first support value comprises computer-readable program code further configured to:

project the auxiliary image, at the particular depth, onto the reference image; and

determining the first support value based on comparing a feature of a plurality of pixels in the reference image to a feature of a corresponding plurality of pixels in the reference image.

12. The computer program product of claim 8, wherein calculating the third support value comprises computer-readable program code further configured to:

determining the third support value by comparing the color of the pixel in the reference image with a corresponding pixel in the auxiliary image.

13. The computer program product of claim 8, wherein the computer-readable program code is further configured to:

generate a depth map by selecting respective depth values for a plurality of pixels in the image;

forming a super pixel comprising a plurality of pixels in the reference image based on features extracted from the reference image and a corresponding thermal image;

determining a depth-plane of the super pixel based on the depth values of the plurality of pixels; and

adjusting at least one of the depth values based on the depth-plane.

14. A system, comprising:

a common support structure comprising;

a reference camera,

at least one auxiliary camera, and

a range-estimation camera; and

a computing device communicatively coupled to the reference, auxiliary, and ToF cameras, the computing device configured to;

calculate a first support value indicating whether a pixel in a reference image captured by the reference camera is at a particular depth, relative to the reference camera, based on comparing a region of the reference image with a region of an auxiliary image captured by the auxiliary camera;

calculate a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing a depth estimate from the range-estimation camera to the particular depth;

calculate a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fuse the first, second, and third support values to generate a total support value for the pixel at the particular depth.

15. The system of claim 14, wherein the computing device is further configured to:

calculate respective first, second, and third support values for the pixel for each of a plurality of discrete depths that divide a depth volume; and

fuse the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

select the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

16. The system of claim 15, wherein the computing device is further configured to:

select respective depth values for a plurality of pixels in the image based on calculating first, second, and third support values at each of the plurality of descrete depths; and

generate a depth map associated with the image using the respective depth values.

17. The system of claim 14, wherein the support structure includes at least two auxiliary cameras, wherein a first one of the auxiliary cameras is positioned on a first side of the reference camera and a second one of the auxiliary cameras is positioned on a second side of the reference camera opposite of the first side.

18. The system of claim 14, wherein the support structure includes a thermal camera, wherein the thermal camera is configured to provide thermal data associated with the image to the computing device.

19. A method for adjusting a depth value for a pixel in an image, the method comprising:

providing a depth map associated with the image, wherein the depth map comprises respective depth values for a plurality of pixels in the image;

grouping a subset of the plurality of pixels into a super pixel based on comparing respective thermal values associated with neighboring pixels; and

estimating a depth plane for the super pixel based on the respective depth values of the subset of the plurality of pixels; and

adjusting at least one of the respective depth values by comparing the respective depth values of the subset of the plurality of pixels to the estimated depth plane.

20. The method of claim 19, wherein grouping the subset of the plurality of pixels into the super pixel further comprises:

determining a first similarity score between the plurality of pixels by comparing color values associated with each of the plurality of pixels;

determining a second similarity score between the plurality of pixels by comparing thermal values associated with each of the plurality of pixels; and

upon determining that the first and second similarity scores satisfy a threshold, grouping the subset of the plurality of pixels into the super pixel.