System and method for progressive stereo matching of digital images

• US 7,106,899 B2
• Filed: 03/17/2005
• Issued: 09/12/2006
• Est. Priority Date: 05/04/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Shan, Ying
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US11/082,444
• Publication Number:
  US 20050163366A1
• Time in Patent Office: 544 Days
• Field of Search: 382/154, 345/422, 348/47, 348/49, 348/50, 356/375, 356/376, 250/559.23, 250/559.31, 702/152, 702/172
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method and a system for progressive stereo matching of digital images representing a scene. In general, the present invention uses a progressive iterative technique that includes a disparity gradient limit principle and a least commitment strategy. Generally unambiguous pixel matches are found by beginning with a few reliable pixel matches and finding progressively more unambiguous pixel matches. Unambiguous pixel matches are used to define the search ranges for each pixel to guide matching in the current iteration. Unambiguous pixel matches then are found using a novel correlation technique and based on a correlation score associated with a pixel match. The search range is capable of being rotated, and is part of a novel correlation technique that provides a more robust estimate of pixel match reliability. Potential pixel matches found in the search ranges are tested for ambiguity and unambiguous matches are selected and added to the reliable pixel matches.

Claims

1. A computer-readable medium having computer-executable instructions for stereo matching digital images representing a scene, comprising:

determining possible pixel matches between a target pixel from a first digital image and pixels from a second digital image;

evaluating a reliability of each of the possible pixel matches; and

classifying a certain pixel match based on the reliability of the possible pixel matches;

wherein classifying a certain pixel match further comprises;

generating a list of candidate pixels from the pixels of the second image;

ordering the list of candidate pixels on an epipolar line;

determining a correlation score for each pixel on the list of candidate pixels;

computing a correlation curve from the correlation scores;

computing a threshold intensity; and

determining a number of peaks on the correlation curve that approximately exceed the threshold intensity.

2. The computer-readable medium of claim 1, wherein a certain pixel match is classified as no match if the number equals zero.

3. The computer-readable medium of claim 1, wherein a certain pixel match is classified as ambiguous if the number is greater than one.

4. The computer-readable medium of claim 1, wherein a certain pixel match is classified as a match if the number is equal to one.

5. The computer-readable medium of claim 1, wherein the pixels from the second image satisfy an epipolar constraint.

6. The computer-readable medium of claim 5, wherein the pixels from the second image satisfy a disparity gradient limit constraint.

7. The computer-readable medium of claim 1, wherein the reliability is evaluated using the least commitment strategy.

8. The computer-readable medium of claim 1, wherein evaluating the reliability comprises:

predicting a disparity between the target pixel and pixels from the second image; and

estimating an uncertainty of the predicted pixel disparity.

9. A method for stereo matching digital images representing a scene, comprising:

a determination step for determining possible pixel matches between a target pixel from a first digital image and pixels from a second digital image;

an evaluation step for evaluating a reliability of each of the possible pixel matches; and

a classification step for classifying a certain pixel match based on the reliability of the possible pixel matches;

wherein the classification step for classifying a certain pixel match further comprises;

a generation step for generating a list of candidate pixels from the pixels of the second image;

an ordering step for ordering the list of candidate pixels on an epipolar line;

a correlation step for determining a correlation score for each pixel on the list of candidate pixels;

a computation step for computing a correlation curve from the correlation scores;

a threshold step for computing a threshold intensity; and

a comparison step for determining a number of peaks on the correlation curve that approximately exceed the threshold intensity.

10. The method of claim 9, wherein a certain pixel match is classified as no match if the number equals zero.

11. The method of claim 9, wherein a certain pixel match is classified as ambiguous if the number is greater than one.

12. The method of claim 9, wherein a certain pixel match is classified as a match if the number is equal to one.

13. The method of claim 9, wherein the pixels from the second image satisfy an epipolar constraint.

14. The method of claim 13, wherein the pixels from the second image satisfy a disparity gradient limit constraint.

15. The method of claim 9, wherein the reliability is evaluated using the least commitment strategy.

16. The method of claim 9, wherein the evaluation step further comprises:

a prediction step for predicting a disparity between the target pixel and pixels from the second image; and

an estimation step for estimating an uncertainty of the predicted pixel disparity.

17. A digital image stereo matching system for stereo matching of digital images representing a scene, comprising:

an initialization module that determines possible pixel matches between a target pixel from a first digital image and pixels from a second digital image, evaluates a reliability of each of the possible pixel matches, and generates a list of candidate pixels from the pixels of the second image;

a correlation module that orders the list of candidate pixels on an epipolar line, determines a correlation score for each pixel on the list of candidate pixels, and computes a correlation curve from the correlation scores; and

a classification module that classifies a certain pixel match based on the reliability of the possible pixel matches, computes a threshold intensity, and determines a number of peaks on the correlation curve that approximately exceed the threshold intensity.

18. The digital image stereo matching system of claim 17, wherein a certain pixel match is classified as no match by the classification module if the number equals zero.

19. The digital image stereo matching system of claim 17, wherein a certain pixel match is classified as ambiguous by the classification module if the number is greater than one.

20. The digital image stereo matching system of claim 17, wherein a certain pixel match is classified as a match by the classification module if the number is equal to one.