STEREO IMAGE PROCESSING DEVICE AND STEREO IMAGE PROCESSING METHOD

• US 20150036917A1
• Filed: 06/08/2012
• Published: 02/05/2015
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: Sovereign Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nanri, Takuya, Nishimura, Hirofumi, Ueda, Junko, Arata, Koji
• Granted Patent: US 9,275,463 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G01C 3/085 with electronic parallax measurement
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Provided is a stereo image processing device, with which it is possible to compute disparity with high precision even for an object of a small image region size in a baseline length direction. With this device, an image matching unit (102) acquires a correspondence point of a reference image for a target point of a target image. An image cropping unit (201) extracts first two-dimensional pixel data including the target point from the target image, and extracts second two-dimensional pixel data including the correspondence point from the reference image. An image reconfiguration unit (202) reconfigures the respective first two-dimensional pixel data and second two-dimensional pixel data into first one-dimensional pixel data and second one-dimensional pixel data. A peak position detection unit (104) computes disparity based on the correlation between the first one-dimensional pixel data and the second one-dimensional pixel data.

Claims

1. A stereo image processing apparatus comprising:

an image matching section that performs image matching on a target image and a reference image so as to obtain a corresponding point in the reference image corresponding to a target point in the target image;

an image clipping section that extracts, from the target image, first two-dimensional pixel data having a plurality of pixel rows and a plurality of pixel columns and including the target point, and extracts, from the reference image, second two-dimensional pixel data having a plurality of pixel rows and a plurality of pixel columns and including the corresponding point;

an image reconstructing section that reconstructs, based on a predetermined reconstruction rule, the first two-dimensional pixel data and the second two-dimensional pixel data to first one-dimensional pixel data and second one-dimensional pixel data, respectively; and

a peak position detecting section that calculates a disparity based on a correlation between the first one-dimensional pixel data and the second one-dimensional pixel data.

2. The stereo image processing apparatus according to claim 1, wherein

the predetermined reconstruction rule is a rule for reconstructing two-dimensional pixel data to one-dimensional pixel data such that a ratio of distances between given pixels included in a same pixel row in two-dimensional pixel data before reconstruction to a ratio of the distances after the reconstruction are identical for any of pixels included in the two-dimensional pixel data.

3. The stereo image processing apparatus according to claim 1, wherein

the predetermined reconstruction rule is a rule for generating the one-dimensional pixel data such that a distance between pixels in a direction of the pixel row is greater than or equal to an original distance between the pixels.

4. The stereo image processing apparatus according to claim 1, wherein

the first two-dimensional pixel data and the second two-dimensional pixel data include data of the same number of pixels.

5. The stereo image processing apparatus according to claim 1, further comprising:

an anti-aliasing section that removes a folding noise generated due to the reconstruction from the first one-dimensional pixel data and the second one-dimensional pixel data; and

a filter generating section that calculates an inverted phase filtering coefficient by inverting, in a direction in which pixels are arranged, a position of each pixel value in the first one-dimensional pixel data with the noise removed, and filters the second one-dimensional pixel data with the noise removed using the inverted phase filtering coefficient, wherein

the peak position detecting section calculates a disparity based on a correlation between the first one-dimensional pixel data with the noise removed and the second one-dimensional pixel data with the noise removed.

6. The stereo image processing apparatus according to claim 5, further comprising

a windowing section that windows the first one-dimensional pixel data and the second one-dimensional pixel data with the noise removed, using a window function having a continuous change at each end of a window, wherein:

a length of the window of the window function is set according to a shift amount of a position of the target point on the target image and a position of the corresponding point on the reference image; and

the filter generating section calculates an inverted filtering coefficient by inverting, in the direction in which the pixels are arranged, a position of each pixel value in the windowed first one-dimensional pixel data, and filters, using the inverted filtering coefficient, the windowed second one-dimensional pixel data.

7. The stereo image processing apparatus according to claim 5, further comprising:

a rectification processing section that performs distortion correction and rectification on the target image and the reference image, and outputs the target image and the reference image after the distortion correction and the rectification to the image matching section; and

a coordinate converting section that converts coordinates of the target point and the reference point to coordinates in the target image and the reference image, and outputs, to the peak position detecting section, a value obtained by rounding up or rounding off at the time of the conversion, wherein

the peak position detecting section calculates a disparity by adding a value rounded up or rounded off in the pixel row direction at the time of conversion to a shift amount from the target point on the target image after the conversion by the coordinate conversion section and the corresponding point on the reference image after the conversion by the coordinate conversion section, the target point and the corresponding point being calculated based on a correlation between the target pixel data and the reference pixel data in a result of the filtering.

8. The stereo image processing apparatus according to claim 7, further comprising:

a vertical pixel corrected image generating section that performs distortion correction and rectification on the target image and the reference image only in the vertical direction, and outputs a vertical pixel corrected target image and a vertical pixel corrected reference image to the image clipping section, the vertical pixel corrected target image being the target image corrected by the distortion correction and the rectification only in the vertical direction and the vertical pixel corrected reference image being the reference image corrected by the distortion correction and the rectification only in the vertical direction, wherein;

the coordinate converting section converts coordinates of the target point and the corresponding point obtained from the image matching section to coordinates on the vertical pixel corrected target image and the vertical pixel corrected reference image, respectively;

the image clipping section clips an image range from the vertical pixel corrected target image and the vertical pixel corrected reference image;

the filter generating section calculates an inverted phase filtering coefficient by inverting, in the direction in which the pixels are arranged, a position of each pixel value in target pixel data which is one-dimensional pixel data generated from the vertical pixel corrected target image and with a folding noise removed, and filters reference pixel data which is one-dimensional pixel data generated from the vertical pixel corrected reference image and with a folding noise removed; and

the peak position detecting section calculates a disparity between a target point on the vertical pixel corrected target image and a corresponding point on the vertical pixel corrected reference image calculated which are obtained by the conversion by the coordinate converting section based on a correlation between the target pixel data and the reference pixel data in the result of the filtering.

9. The stereo image processing apparatus according to claim 8, further comprising:

a sub-pixel disparity approximate calculation section that calculates a disparity approximate calculation value on a sub-pixel level; and

a data extracting section that calculates a data extraction starting position based on the disparity approximate calculation value on the sub-pixel level and obtains extracted one-dimensional pixel data by extracting data from the one-dimensional pixel data based on the data extraction starting position, wherein;

the windowing section windows the extracted one-dimensional pixel data; and

the peak position detecting section corrects the calculated disparity based on the data extraction starting position.

10. The stereo image processing apparatus according to claim 1, wherein:

the stereo image is an image in a Bayer pattern;

the image clipping section extracts two-dimensional pixel data composed of one or a plurality of pixel rows and a plurality of pixel columns; and

the reconstruction rule is for generating one-dimensional pixel data by extracting at least one of an R pixel value, a G pixel value, and a B pixel value from an image range.

11. The stereo image processing apparatus according to claim 8, wherein

the stereo image is an image in the Bayer pattern,

the stereo image processing apparatus further comprises

a Bayer pattern adjusting section that adjusts, when arrangements in the Bayer pattern in the vertical pixel corrected target image and the vertical pixel corrected reference image do not correspond to each other, a position of the corresponding point calculated by the matching section to match the arrangements in the Bayer pattern, wherein

the reconstruction rule is for generating one-dimensional pixel data by extracting at least one of an R pixel value, a G pixel value, and a B pixel value from the image range.

12. A stereo image processing method comprising:

performing image matching on a target image and a reference image so as to obtain a corresponding point in the reference image corresponding to a target point in the target image;

extracting, from the target image, first two-dimensional pixel data having a plurality of pixel rows and a plurality of pixel columns and including the target point, and extracting, from the reference image, second two-dimensional pixel data having a plurality of pixel rows and a plurality of pixel columns and including the corresponding point;

reconstructing, based on a predetermined reconstruction rule, the first two-dimensional pixel data and the second two-dimensional pixel data to first one-dimensional pixel data and second one-dimensional pixel data, respectively; and

calculating a disparity based on a correlation between the first one-dimensional pixel data and the second one-dimensional pixel data.