Stereo correspondence and depth sensors

• US 10,455,219 B2
• Filed: 11/30/2012
• Issued: 10/22/2019
• Est. Priority Date: 11/30/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Cohen, Scott D., Price, Brian L., Somanath, Gowri
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Hasan, Syed Y
• Application Number:
  US13/690,724
• Publication Number:
  US 20140152776A1
• Time in Patent Office: 2,517 Days
• Field of Search: 348 47, 348336, 348348, 382104, 382154, 382224, 382103, 705 1469, 345158
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Stereo correspondence and depth sensor techniques are described. In one or more implementations, a depth map generated by a depth sensor is leveraged as part of processing of stereo images to assist in identifying which parts of stereo images correspond to each other. The depth map, for instance, may be utilized to assist in identifying depth discontinuities in the stereo images. Additionally, techniques may be employed to align the depth discontinuities identified from the depth map to image edges identified from the stereo images. Techniques may also be employed to suppress image edges that do not correspond to the depth discontinuities of the depth map in comparison with image edges that do correspond to the depth discontinuities as part of the identification.

Claims

1. A system comprising:

first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;

one or more depth sensors, separate from the first and second image capture devices, configured to detect depth values at locations of the image scene; and

a computing device configured to;

generate a depth map identifying depth discontinuities, from the detected depth values;

identify image edges in the stereoscopic images based on characteristics of pixels from the stereoscopic images;

determine image edge correspondence by comparing image edges identified from the stereoscopic images with depth discontinuities identified using the depth map;

suppress image edges identified from the stereoscopic images that do not correspond to depth discontinuities identified using the depth map, by removing image edges that do not correspond to depth discontinuities from a stereo correspondence calculation;

calculate the stereo correspondence by using the depth map generated from the detected depth values, detected by the one or more separate depth sensors, in conjunction with the stereoscopic images captured by the first and second image capture devices; and

generate, using the calculated stereo correspondence, a version of the image scene that includes the image edges that do correspond to depth discontinuities and are not suppressed.

2. A system as described in claim 1, wherein the first and second images capture devices are part of a stereoscopic camera.

3. A system as described in claim 1, wherein the characteristics of pixels from the stereoscopic images are texture and color.

4. A system as described in claim 1, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.

5. A system comprising:

first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;

one or more depth sensors, separate from the first and second image capture devices, configured to detect depths at locations of the image scene without using the captured stereoscopic images; and

a computing device configured to;

generate a depth map identifying depth discontinuities, from the detected depth values;

identify image edges in the stereoscopic images based on characteristics of pixels from the stereoscopic images;

determine image edge correspondence by projecting the depth map onto the stereoscopic images and comparing image edges identified from the stereoscopic images with depth discontinuities identified using the depth map;

classify image edges as corresponding to the depth discontinuities or as not corresponding to the depth discontinuities;

suppress image edges that do not correspond to the depth discontinuities by removing them from a stereo correspondence calculation;

calculate the stereo correspondence using the image edges that do correspond to the depth discontinuities; and

generate, using the calculated stereo correspondence, a version of the image scene that includes the image edges that do correspond to depth discontinuities and are not suppressed.

6. A system as described in claim 5, wherein the first and second images capture devices are part of a stereoscopic camera.

7. A system as described in claim 5, wherein the characteristics of pixels from the stereoscopic images are texture and color.

8. A system as described in claim 5, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.

9. A system comprising:

first and second image capture devices having respective first and second image sensors configured, at least partially in hardware, operative to capture stereoscopic images of an image scene;

one or more depth sensors, separate from the first and second image capture devices, configured in addition to the first and second image capture devices to detect depths at locations of the image scene; and

a computing device configured to use the detected depths, detected by the one or more separate depth sensors, to generate a depth map identifying depth discontinuities, and to use the depth map in conjunction with the stereoscopic images captured by the first and second image capture devices identifying image edges based on characteristics of pixels, to identify depth discontinuities in the stereoscopic images, to generate a version of the image scene that does not suppress image edges that do correspond to depth discontinuities and that does suppresses the image edges that do not correspond to depth discontinuities by removing them from a stereo correspondence calculation used to generate the version of the image scene.

10. A system as described in claim 9, wherein the first and second images capture devices are part of a stereoscopic camera.

11. A system as described in claim 9, wherein the depth map is usable to determine a search range used in conjunction with the stereoscopic images to calculate stereo correspondence.

12. A system as described in claim 11, wherein the depth map is usable to determine correspondence of image edges identified from the stereoscopic images with the identified depth discontinuities in the stereoscopic images.

13. A system as described in claim 12, wherein the characteristics of pixels from the stereoscopic images are texture and color.

14. A system as described in claim 12, wherein the depth sensor is disposed between first and second image capture devices used to capture stereoscopic images.