Real-time depth extraction using stereo correspondence

• US 8,823,777 B2
• Filed: 03/30/2011
• Issued: 09/02/2014
• Est. Priority Date: 03/30/2011
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Aviv, Ziv, Govrin, Omri
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Brown, Jr., Howard D
• Application Number:
  US13/075,428
• Publication Number:
  US 20120249747A1
• Time in Patent Office: 1,252 Days
• Field of Search: 348/47
• US Class Current: 348/47
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods may provide for determining a one-dimensional (1D) disparity between a plurality of rectified images, and extracting depth information from the plurality of rectified images based at least in part on the 1D disparity. In one example, the 1D disparity is in the horizontal direction and the images are rectified with respect to one another in the vertical direction.

Claims

1. A system comprising:

a first camera to capture a first image;

a second camera to capture a second image;

a processor;

and a computer readable storage medium including a set of instructions which, if executed by the processor, cause the system to, rectify the first and second images to obtain a plurality of rectified images, determine a one-dimensional disparity between the plurality of rectified images, extract depth information from the plurality of rectified images based at least in part on the one-dimensional disparity, and construct a three-dimensional image based at least in part on the depth information;

wherein the instructions, if executed, cause the system to;

identify a pixel neighborhood for the plurality of rectified images, detect an edge in the pixel neighborhood, and conduct a block search to refine the one-dimensional disparity in response to detecting the edge.

2. The system of claim 1, wherein the one-dimensional disparity is to be in a horizontal direction and the first and second images are to be rectified with respect to one another in a vertical direction.

3. The system of claim 2, wherein the instructions, if executed, cause the system to shift a quadratic polynomial of a local approximation formula in the horizontal direction.

4. The system of claim 1, wherein the plurality of rectified images is to include a first rectified image and a second rectified image, and the block search is to:

wrap the second rectified image with the one-dimensional disparity to obtain a wrapped image, calculate a second horizontal derivative and a second vertical derivative of the wrapped image, apply a sum of absolute differences formula to the second horizontal derivative, the second vertical derivative, and the first rectified image, and determine a minimum value of the sum of absolute differences formula in the block to obtain a refined disparity.

5. The system of claim 1, wherein the instructions, if executed, cause the system to:

identify a first pixel neighborhood for the plurality of rectified images,

detect that the first pixel neighborhood contains insufficient detail for disparity estimation, and

select a second pixel neighborhood for the plurality of rectified images in response to detecting that the first pixel neighborhood contains insufficient detail, wherein the second pixel neighborhood is to have a greater size than the first pixel neighborhood.

6. A computer implemented method comprising:

determining a one-dimensional disparity between a plurality of rectified images;

and extracting depth information from the plurality of rectified images based at least in part on the one-dimensional disparity;

wherein the instructions, if executed, cause the system to;

identify a pixel neighborhood for the plurality of rectified images, detect an edge in the pixel neighborhood, and conduct a block search to refine the one-dimensional disparity in response to detecting the edge.

7. The method of claim 6, wherein the one-dimensional disparity is in a horizontal direction and the images are rectified with respect to one another in a vertical direction.

8. The method of claim 7, wherein determining the one-dimensional disparity includes shifting a quadratic polynomial of a local approximation formula in the horizontal direction.

9. The method of claim 6, wherein determining the one-dimensional disparity includes:

identifying a pixel neighborhood for the plurality of rectified images;

detecting an edge in the pixel neighborhood; and

conducting a block search to refine the one-dimensional disparity in response to detecting the edge.

10. The method of claim 9, wherein the plurality of rectified images includes a first rectified image and a second rectified image, and the block search includes:

wrapping the second rectified image with the one-dimensional disparity to obtain a wrapped image;

calculating a second horizontal derivative and a second vertical derivative of the wrapped image;

applying a sum of absolute differences formula to the second horizontal derivative, the second vertical derivative, and the first rectified image; and

determining a minimum value of the sum of absolute differences formula in the block to obtain a refined disparity.

11. The method of claim 6, wherein determining the one-dimensional disparity includes:

identifying a first pixel neighborhood for the plurality of rectified images;

detecting that the first pixel neighborhood contains insufficient detail for disparity estimation; and

selecting a second pixel neighborhood for the plurality of rectified images in response to detecting that the first pixel neighborhood contains insufficient detail, wherein the second pixel neighborhood has a greater size than the first pixel neighborhood.

12. The method of claim 6, further including constructing a three-dimensional image based at least in part on the depth information.

13. A non-transitory computer readable storage medium including a set of instructions which, if executed by a processor, cause a computer to:

determine a one-dimensional disparity between a plurality of rectified images;

and extract depth information from the plurality of rectified images based at least in part on the one-dimensional disparity;

wherein the instructions, if executed, cause the system to;

identify a pixel neighborhood for the plurality of rectified images, detect an edge in the pixel neighborhood, and conduct a block search to refine the one-dimensional disparity in response to detecting the edge.

14. The medium of claim 13, wherein the one-dimensional disparity is to be in a horizontal direction and the images are to be rectified with respect to one another in a vertical direction.

15. The medium of claim 14, wherein the instructions, if executed, cause a computer to shift a quadratic polynomial of a local approximation formula in the horizontal direction.

16. The medium of claim 13, wherein the instructions, if executed, cause a computer to:

identify a pixel neighborhood for the plurality of rectified images;

detect an edge in the pixel neighborhood; and

conduct a block search to refine the one-dimensional disparity in response to detecting the edge.

17. The medium of claim 16, wherein the plurality of rectified images is to include a first rectified image and a second rectified image, and the block search is to:

wrap the second rectified image with the one-dimensional disparity to obtain a wrapped image;

calculate a second horizontal derivative and a second vertical derivative of the wrapped image;

applying a sum of absolute differences formula to the second horizontal derivative, the second vertical derivative, and the first rectified image; and

determine a minimum value of the sum of absolute differences formula in the block to obtain a refined disparity.

18. The medium of claim 13, wherein the instructions, if executed, cause a computer to:

identify a first pixel neighborhood for the plurality of rectified images;

detect that the first pixel neighborhood contains insufficient detail for disparity estimation; and

select a second pixel neighborhood for the plurality of rectified images in response to detecting that the first pixel neighborhood contains insufficient detail, wherein the second pixel neighborhood is to have a greater size than the first pixel neighborhood.

19. The medium of claim 13, wherein the instructions, if executed, cause a computer to construct a three-dimensional image based at least in part on the depth information.