Depth map stereo correspondence techniques

• US 9,135,710 B2
• Filed: 11/30/2012
• Issued: 09/15/2015
• Est. Priority Date: 11/30/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Cohen, Scott D., Price, Brian L., Somanath, Gowri
• Primary Examiner(s): Yentrapati, Avinash
• Application Number:
  US13/690,755
• Publication Number:
  US 20140153816A1
• Time in Patent Office: 1,019 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Depth map stereo correspondence techniques are described. In one or more implementations, a depth map generated through use of a depth sensor is leveraged as part of processing of stereo images to assist in identifying which parts of stereo images correspond to each other. For example, the depth map may be utilized to describe depth of an image scene which may be used as part of a stereo correspondence calculation. The depth map may also be utilized as part of a determination of a search range to be employed as part of the stereo correspondence calculation.

Claims

1. A method implemented by one or more computing devices, the method comprising:

determining a search range for calculating stereo correspondence for images of an image scene based at least in part on;

a depth map, generated through the use of a depth sensor, of the image scene;

examining confidence values, generated by the depth sensor, included in the depth map; and

selecting between a search range calculated based at least in part on uniqueness of pixels in relation to each other and a search range calculated at least in part based on the confidence values generated by the depth sensor; and

calculating stereo correspondence of image data of stereoscopic images of the image scene, the calculating performed based at least in part using the determined search range.

2. A method as described in claim 1, wherein the examining includes calculating a density map from the depth map, the density map describing density of values of depth described by the depth map, the density usable to define the confidence of the values.

3. A method as described in claim 2, wherein the determining further comprises calculating a search map from the density map, the search map defining a search area that is to be used based on the density described by the density map.

4. A method as described in claim 1, wherein the determining is based at least in part on generating a stereo search range based on uniqueness of pixels in relation to other pixels in a respective one of the stereoscopic images.

5. A method as described in claim 4, wherein the determining includes examining uniqueness of patches of pixels to other patches of pixels within a neighborhood defined in the stereoscopic images.

6. A method as described in claim 1, wherein the calculating includes calculating a data cost based on a set of known disparities around a neighborhood of a pixel, the set of known disparities indicated by the depth map.

7. A method as described in claim 1, wherein the calculating includes calculating a smoothness cost based at least in part on guided filtering performed using of a respective one of the stereoscopic images and the depth map.

8. A method as described in claim 1, further comprising up-sampling the depth map such that the depth map has a resolution that substantially corresponds to a resolution of the stereoscopic images.

16. A method as described in claim 1, wherein the stereoscopic images are not captured using the depth sensor that was used to generate the depth map.

9. A system comprising:

at least one module implemented at least partially in hardware, the at least one module configured to determine a search range based at least in part on;

a depth map generated through use of a depth sensor of an image scene; and

selecting between a search range calculated at least in part based on uniqueness of pixels in relation to each other and a search range calculated at least in part based on confidence of values of the depth sensor; and

one or more modules implemented at least partially in hardware, the one or more modules configured to calculate stereo correspondence of the image data of stereoscopic images of an image scene, the calculation configured to be performed using the search range and the depth map.

10. A system as described in claim 9, wherein the one or more modules are configured to perform the calculation by calculating a data cost based on a set of known disparities around a fixed neighborhood of a pixel, the set of known disparities indicated by the depth map.

11. A system as described in claim 9, wherein the one or more modules are configured to perform the calculation by calculating a smoothness cost based at least in part on guided filtering based on image edges identified in a respective one of the stereoscopic images and the depth map.

17. A system as described in claim 9, wherein the depth map is not generated based on stereo correspondence of image data of stereoscopic images.

20. A system as described in claim 9, wherein the one or more modules are further configured to up-sample the depth map such that the depth map has a resolution that substantially corresponds to a resolution of the stereoscopic images.

12. One or more computer-readable storage media comprising instructions that are stored thereon that, responsive to execution by a computing device, causes the computing device to perform operations comprising:

receiving a depth map from one or more depth sensors of an image scene, the depth map describing depth of the image scene;

determining a search range for calculating stereo correspondence of stereoscopic images of the image scene, the determining based at least in part on selecting between a search range calculated based at least in part on uniqueness of pixels in relation to each other and a search range calculated based at least in part on confidence values that are included in the depth map; and

calculating stereo correspondence of image data of stereoscopic images of the image scene, the calculating performed based at least in part using the depth map.

13. One or more computer-readable storage media as described in claim 12, wherein the calculating includes calculating a data cost based on a set of known disparities around a fixed neighborhood of a pixel, the set of known disparities indicated by the depth map.

14. One or more computer-readable storage media as described in claim 12, wherein the calculating includes calculating a smoothness cost based at least in part on guided filtering performed using image edges identified in a respective one of the stereoscopic images and the depth map.

15. One or more computer-readable storage media as described in claim 12, wherein the determining of the search range is further based at least in part on generating a stereo search range based on uniqueness of pixels in relation to other pixels in a respective one of the stereoscopic images.

18. One or more computer-readable storage media as described in claim 12, wherein the depth map is not generated based on stereo correspondence of image data.

19. One or more computer-readable storage media as described in claim 12, wherein the examining includes calculating a density map from the depth map, the density map describing density of values of depth described by the depth map, the density of values being usable to define the confidence of the values.