System and method of processing stereo images

• US 8,385,630 B2
• Filed: 12/29/2010
• Issued: 02/26/2013
• Est. Priority Date: 01/05/2010
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Kuthirummal, Sujit, Kumar, Rakesh, Samarasekera, Supun, Sawhney, Harpreet Singh, Sizintsev, Mikhail
• Primary Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US12/980,590
• Publication Number:
  US 20110176722A1
• Time in Patent Office: 790 Days
• Field of Search: 382/154, 382/195, 382/304, 382/303, 382/260, 382/205, 382/299, 382/209, 382/199, 382/107, 382/106, 382/278, 382/285, 348/43, 348/47, 348/51, 348/44, 348/42, 348/E13.014, 348/E13.001, 348/E13.016, 348/E13.02, 348/E13.025, 348/E13.071, 348/E13.073, 348/E13.062, 348/E1.026, 348/E13.074, 348/E13.008, 348/E13.065, 345/419, 356/12
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

The present invention is a system and a method for processing stereo images utilizing a real time, robust, and accurate stereo matching system and method based on a coarse-to-fine architecture. At each image pyramid level, non-centered windows for matching and adaptive upsampling of coarse-level disparities are performed to generate estimated disparity maps using the ACTF approach. In order to minimize propagation of disparity errors from coarser to finer levels, the present invention performs an iterative optimization, at each level, that minimizes a cost function to generate smooth disparity maps with crisp occlusion boundaries.

Claims

1. A computer implemented method for processing stereo images comprising the steps of:

(a) preprocessing, using a stereo image pre-processor module, left and right stereo images;

(b) constructing, using a pyramid generator module, left image pyramids of the left stereo image at a plurality of levels;

(c) constructing, using the pyramid generator module, right image pyramids of the right stereo image at the plurality of levels;

(d) generating, using an adaptive disparity estimation module, estimated left and right disparity maps at the coarsest level of the plurality of the levels of the left and the right image pyramids, said estimated left and right disparity maps having estimated disparity values at each pixel;

(e) determining, using an iterative local disparity optimization module, disparity values of all neighboring pixels of each said pixel in the estimated left and right disparity maps at the coarsest level;

(f) computing, using the iterative local disparity optimization module, a first iteration of TotalCost values for each of the estimated disparity value at each said pixel and the first iteration of the TotalCost values for the disparity values of said neighboring pixels in the estimated left and right disparity maps at the coarsest level;

wherein a second iteration of TotalCost values is computed using result of the first iteration of the Total Cost values;

(g) selecting, using the iterative local disparity optimization module, the disparity at each said pixel having a minimum value of the first and the second iterations of the TotalCost values to generate optimized left and right disparity maps at the coarsest level.

2. The method of claim 1 further comprising repeating step (d) using the optimized left and the right disparity maps at a coarser level to generate estimated left and right disparity maps at another level of said plurality of levels.

3. The method of claim 2 further comprising repeating steps (e) through (g) to generate optimized left and right disparity maps at said another level.

4. The method of claim 3 further comprising repeating steps (d) through (g) to generate optimized left and right disparity maps at all of said plurality of levels.

5. The method of claim 1 wherein said computing Totalcost values are generated as

TotalCost(d_i)=CC(d_i)+λ

SC(d_i)

where CC(d_i) is a data term, which is defined as 1−

C, C being the highest correlation score for disparity di obtained over all centered and non-centered windows that include pixel, p, SC is the smoothness term that measures the compatibility of di and λ

is a user defined parameter to control the smoothness of the estimated left and right disparity maps.

6. The method of claim 5 wherein said smoothness term is generated as

SC(di)=Σ

W(q)×

min(|d_i−

D(q)|,α

)

where q is a pixel in p'"'"'s neighborhood and D(q) denotes its current disparity estimate, W(q) is one of a constant value or a value inversely proportional to the difference in intensities of p and q, and α

is a constant value.

7. The method of claim 5 wherein said correlation (C) is generated as

8. The method of claim 1 further comprising detecting, using occlusion detection and substitution module, occluded pixels in the left and the right optimized disparity maps at the coarsest pyramid level and substituting the occluded pixels in the optimized left and right disparity maps at that level to generate enhanced left and right disparity maps at the coarsest level.

9. The method of claim 8 wherein said substituting comprising identifying the neighboring non-occluded pixel with a smallest disparity value and substituting the occluded pixel'"'"'s disparity with said smallest disparity value.

10. The method of claim S further comprising repeating step (d) using the enhanced left and the right disparity maps at the coarsest level to generate estimated left and right disparity maps at another level of said plurality of levels.

11. The method of claim 10 further comprising repeating steps (e) through (g) to generate optimized left and right disparity maps at said another level.

12. The method of claim 11 comprising detecting occluded pixels in the left and the right optimized disparity maps at said another level and substituting the occluded pixels in the optimized left and right disparity maps at said another level to generate enhanced left and right disparity maps at said another level.

13. The method of claim 11 further comprising repeating steps (d) through (g) to generate optimized left and right disparity maps at all of said plurality of levels.

14. The method of claim 13 further comprising detecting occluded pixels in the left and the right optimized disparity maps at all of said plurality of levels and substituting the occluded pixels at each of the optimized left and right disparity maps to generate enhanced left and right disparity maps at all of said plurality of levels.

15. A system for processing stereo images comprising:

a stereo image preprocessor module for capturing, digitizing and preprocessing stereo imagery of left and right stereo images;

a pyramid generator module for constructing left and right pyramids for the left and right stereo images at a plurality of levels;

an adaptive disparity estimation module for generating estimated left and right disparity maps of the left and the light image pyramids at the coarsest level of the plurality of the levels, said estimated left and right disparity maps having estimated disparity values at each pixel; and

an iterative local disparity optimization module for determining disparity values of all neighboring pixels of each said pixel in the estimated left and right disparity maps at the coarsest level, said iterative local disparity optimization module for computing a first iteration of TotalCost values for each of the estimated disparity value at each said pixel and a first iteration of the TotalCost values for the disparity values of said neighboring pixels in the estimated left and right disparity maps at the coarsest level, said iterative local disparity optimization module for computing a second iteration of TotalCost values using results from the first iteration of the TotalCost values;

said iterative local disparity optimization module for selecting the disparity value at each said pixel having a minimum value of the first and the second iteration of the TotalCost values to generate optimized left and right disparity maps at the coarsest level.

16. The system of claim 15 wherein said adaptive estimation module functions to generate estimated left and right disparity maps at another level of said Plurality of levels using the optimized left and the right disparity maps at the coarsest level, and said iterative local disparity optimization module functions to generate optimized left and right disparity maps at said another level using the estimated left and right disparity maps at said another level.

17. The system of claim 16 wherein said adaptive estimation module functions to generate estimated left and right disparity maps at all of said plurality of levels and said local disparity optimization module functions to generate said optimized left and right disparity maps at said all of the plurality of levels.

18. The system of claim 15 further comprising an occlusion detection and substitution module, to detect occluded pixels in the left and the right optimized disparity maps at the coarsest level and fill in the occluded pixels in the optimized left and right disparity maps to generate enhanced left and right disparity maps at the coarsest level.

19. The system of claim 18 wherein said adaptive estimation module functions to generate estimated left and right disparity maps at another level of said plurality of levels using the enhanced left and the right disparity maps at the coarsest level, said iterative local disparity optimization module functions to generate optimized left and right disparity maps at said another level using the estimated left and right disparity maps at said another level, said occlusion detection and substitution module, to detect occluded pixels in the left and the right optimized disparity maps at said another level and substitute the occluded pixels in the optimized left and right disparity maps to generate enhanced left and right disparity maps at said another level.

20. The system of claim 19 wherein said adaptive estimation module functions to generate estimated left and right disparity maps at all levels of said plurality of levels, said local disparity optimization module functions to generate optimized left and right disparity maps at said all levels, and said occlusion detection and substitution module function to detect occluded pixels in the left and the right optimized disparity maps at said all levels and substitute the occluded pixels at each of the optimized left and right disparity maps to generate enhanced left and right disparity maps at said all levels.