Hardware-Oriented Dynamically Adaptive Disparity Estimation Algorithm and its Real-Time Hardware

• US 20150319419A1
• Filed: 05/01/2014
• Published: 11/05/2015
• Est. Priority Date: 05/01/2014
• Status: Active Grant

• Current Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Original Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Inventors: LEBLEBICI, Yusuf, SCHMID, Alexandre, AKIN, Abdulkadir, BAZ, Ipek, BOYBAT, Irem, ATAKAN, Huseyin Baris
• Granted Patent: US 9,756,312 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A real-time stereo camera disparity estimation device comprises input means arranged to input measured data corresponding to rows of left and right images; a plurality of on-chip memories arranged to buffer the input measured data; a vertical rotator hardware module configured to align the rows of left and right images in a same column; a reconfigurable data allocation hardware module; a reconfigurable computation of metrics hardware module; and an adaptive disparity selection hardware module configured to select disparity values with the minimum matching costs.

Claims

1. A real-time stereo camera disparity estimation device comprising:

input means arranged to input measured data corresponding to rows of left and right images;

a plurality of on-chip memories arranged to buffer the input measured data;

a vertical rotator hardware module configured to align the rows of left and right images in a same column;

a reconfigurable data allocation hardware module;

a reconfigurable computation of metrics hardware module; and

an adaptive disparity selection hardware module configured to select disparity values with the minimum matching costs.

2. The device of claim 1, further comprising an iterative disparity refinement hardware module configured to iteratively refine the disparity values.

3. The device of claim 1, wherein the reconfigurable data allocation hardware module is configured to create variable window sizes to adapt the window size to the local texture on the image.

4. The device of claim 1, wherein the reconfigurable computation of metrics hardware module comprises plurality of processing elements for multiple processed pixels in a two dimensional block to compute their stereo matching costs for the candidate disparities in parallel.

5. The device of claim 1, wherein each of the plurality of on-chip memories comprises:

dual-ports configured to write and read concurrently;

a connection of read address ports to the same read address request of the processing elements to allow processing elements to read multiple rows and the same column of the image in parallel; and

YCbCr or RGB data for the pixels.

6. The device of claim 5, wherein pixels of different rows are stored in separate block RAMs to be able to access multiple pixels in the same column in parallel.

7. The device of claim 5, wherein the data in the block RAMs are overwritten by the new rows of the image after they are processed.

8. The device of claim 1, wherein the vertical rotator is further configured

to rotate either Y, Cb or Cr, either R, G or B to make disparity estimation in any of the selected pixel data channel;

and

to rotate and align either left image pixels or right image pixels.

9. The device of claim 3, wherein the reconfigurable data allocation module to create variable window sizes comprises,

a flip-flop array configured to store and shift aligned outputs of the vertical rotator;

wires connected to the flip-flops array arranged to sample the pixels while pixels are flowing inside the flip-flops array;

a plurality of first sampling schemes to provide the variable window sizes;

a plurality of second sampling schemes to provide constant number of contributing pixels in the variable window sizes to provide constant computational load for the variable window sizes;

a plurality of multiplexers configured to select the windows to be used in disparity estimation process of multiple pixels in a block according to the selected window size.

10. The device of claim 9, wherein the selection of window size is determined depending on the variance of the neighboring pixels for variable window sizes.

11. The device of claim 9, wherein a same selected window size is applied to the multiple searched pixels in a block.

12. The device of claim 9, wherein for every searched block of pixels, window size is dynamically re-determined.

13. The device of claim 4, wherein the plurality of processing elements hardware are configured for a computation of metrics and comprises,

a plurality of census, Hamming, SAD and BW-SAD cost computation modules for the concurrent and independent disparity search of the multiple pixels in the two dimensional block,

selection means configured for a configurability through selection either of SAD or BW-SAD cost computations.

14. The device of claim 13, wherein the plurality of processing elements hardware comprises:

SAD and BW-SAD computations for the sampled pixels in the searched block to reduce the overall computational complexity;

interpolation of SAD and BW-SAD values of the sampled pixels in the block to compute and estimate the SAD and BW-SAD values of all the remaining pixels in the searched block for which SAD and BW-SAD are not computed; and

Hamming computations for all the pixels in the searched block.

15. The device of claim 1, wherein the adaptive disparity selection hardware module for the selection of the disparities with the minimum matching costs comprises:

a multiplier to normalize the hamming cost using adaptive penalties; and

means for performing addition of multiplied hamming value with the SAD result to compute hybrid cost.

16. The device of claim 15, wherein the adaptive penalties are in the order of two to simplify the implementation of multipliers with shifters.

17. The device of claim 15, wherein small penalty values are used for small window size, and big penalty values are used for big window size.

18. The device of claim 2, wherein the disparity refinement hardware module to refine the disparity values comprises:

a flip-flop array to store and shift the disparity results; and

at least a highest frequency selection hardware module configured to determine the most frequent disparity values to replace the processed disparity values with the most frequent ones.

19. The device of claim 18, wherein the highest frequency selection hardware module is configured to determine the most frequent disparity values and refine the disparities using the color similarity of the neighboring pixels.

20. The device of claim 18, wherein multiple rows are refined in parallel using multiple hardware modules to determine the most frequent disparity values.

21. The device of claim 18, wherein the disparity results are iteratively refined.

22. The device of claim 18, wherein the disparity results are iteratively refined by processing multiple consecutive columns using multiple highest frequency selection hardware modules.

23. The device of claim 18, wherein the refined disparity values are written back to the disparity results array to iteratively use refined disparity values for the further refinements.

24. The device of claim 18, wherein the final shifted disparity values at the end of the disparity results array are used as the output of the disparity estimation hardware.

25. An iterative disparity refinement hardware module to refine disparity values which comprises,

a flip-flop array to store and shift the disparity results;

and

a highest frequency selection hardware module configured to determine the most frequent disparity value to replace the processed disparity value with the most frequent one.

26. The hardware module of claim 25, wherein the highest frequency selection hardware module is configured to determine the most frequent disparity values and refine the disparities using the color similarity of the neighboring pixels.

27. The hardware module of claim 25, wherein multiple rows are refined in parallel using multiple hardware modules to determine the most frequent disparity values.

28. The hardware module of claim 25, wherein the disparity results are iteratively refined.

29. The hardware module of claim 25, wherein the disparity results are iteratively refined by processing multiple consecutive columns using multiple highest frequency selection hardware modules.

30. The hardware module of claim 25, wherein the refined disparity values are written back to the disparity results array to iteratively use refined disparity values for the further refinements.

31. The hardware module of claim 25, wherein the final shifted disparity values at the end of the disparity results array are used as the output of the disparity estimation hardware.

32. A reconfigurable data allocation hardware module configured to create variable window sizes to adapt the window size to the local texture on the image comprising,

a flip-flop array configured to store and shift aligned outputs of the vertical rotator;

wires connected to the flip-flops array arranged to sample the pixels while pixels are flowing inside the flip-flops array;

a plurality of first sampling schemes to provide the variable window sizes;

a plurality of second sampling schemes to provide constant number of contributing pixels in the variable window sizes to provide constant computational load for the variable window sizes;

a plurality of multiplexers configured to select the windows to be used in disparity estimation process of multiple pixels in a block according to the selected window size.

33. The hardware module of claim 32, wherein the selection of window size is determined depending on the variance of the neighboring pixels for variable window sizes.

34. The hardware module of claim 32, wherein a same selected window size is applied to the multiple searched pixels in a block.

35. The hardware module of claim 32, wherein for every searched block of pixels, window size is dynamically re-determined.