IMAGE PROCESSING APPARATUS, METHOD, PROGRAM, AND RECORDING MEDIUM

• US 20110311130A1
• Filed: 03/15/2011
• Published: 12/22/2011
• Est. Priority Date: 03/19/2010
• Status: Active Grant

• Current Assignee: LAPIS Semiconductor Co., Ltd. (ROHM Co., Ltd.)
• Original Assignee: Oki Semiconductor Company Limited (ROHM Co., Ltd.)
• Inventors: Ichimori, Takashi
• Granted Patent: US 8,917,929 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Extracting information corresponding to a three-dimensional object from an image captured by plural imaging apparatuses is implemented with a simple configuration and a simple processing.

Parallax information representing a parallax amount in the X direction of a pair of images captured by a pair of imaging apparatuses disposed at different horizontal positions is stored in a storage section 44, and a parallax correction control section 42 reads out a parallax amount corresponding to a Y coordinate value of image data for one line input from a pre-image processing unit 14 from the storage section 44 and outputs a selection signal to selectors 38 and 44 such that the output of one image data is delayed by the differential amount. A differential image generation unit 18 calculates an absolute value of the differential of the pair of input image data for each pixel and outputs the result as a differential image for detecting the three-dimensional object.

Claims

1. An image processing apparatus comprising:

a storage unit that stores bias amount information representing a bias amount, which corresponds to a parallax, along a first direction of an image corresponding to a horizontal direction between a first image captured by a first imaging apparatus and a second image captured by a second imaging apparatus which is located at a position different from that of the first imaging apparatus in the horizontal direction, with respect to each position in a second direction of the image corresponding to a vertical direction;

an acquisition unit that acquires the first image captured by the first imaging apparatus and the second image captured by the second imaging apparatus;

a processing unit that performs a parallax correction with respect to each pixel row along the first direction of the image, the parallax correction moving a position of the image in the first direction of a pixel row along the first direction of the image in accordance with the bias amount represented by the bias amount information corresponding to a position of the pixel row in the second direction of the image; and

a generation unit that generates differential information representing a difference between the first image and the second image that have been subjected to the parallax correction by the processing unit.

2. The image processing apparatus of claim 1, wherein:

the generation unit generates a differential image representing a difference between the first image and the second image for each pixel as the differential information; and

the image processing apparatus further includes a removal unit that removes noise from the differential image by performing a contraction processing with respect to a differential pixel that exists in the differential image generated by the generation unit and that indicates a presence of the difference between the first image and the second image.

3. The image processing apparatus of claim 1, wherein the bias amount represented by the bias amount information stored in the storage unit is bias amount D which represents a bias amount by a number of pixels, and the bias amount D is derived in advance by performing operations of Equations 1 to 3 below, where a height of an imaging optical axis from a ground at installation positions of the first imaging apparatus and the second imaging apparatus is defined as h_cam, an inclination angle of the imaging optical axis with respect to a vertical direction is defined as θ

, a focal length of an optical system is defined as f, the number of pixels along the first direction of the image is defined as w, the number of pixels along the second direction is defined as h, an imaging size along the first direction of the image is defined as w_img, an imaging size along the second direction is defined as h_img, a position (the number of pixels) on the image along the second direction is defined as P, and a baseline length as a distance of the imaging optical axis of the first imaging apparatus and the second imaging apparatus is defined as b;

Equation 1

D=b×

f/L_—

cam×

(w/2)/(w_—

img/2)  

 

(1)

wherein L_cam denotes a linear distance between an object which is positioned on the ground and which is imaged at a position (the number of pixels) P on the image along the second direction and the first imaging apparatus or the second imaging apparatus, and L denotes a distance between the object and the first imaging apparatus or the second imaging apparatus;

Equation 2

L_—

cam=√

(h_—

cam²+L²)×

cos(tan<sup>−

1</sup>(L/h_—

cam)−

θ

)  

 

(2)

Equation 3

L=tan(tan<sup>−

1</sup>((P−

(h/2))×

(h_—

img/2)/(h/2)/f)+θ

)×

h_—

cam  

 

(3).

4. The image processing apparatus of claim 1, wherein:

the bias amount information is information which represents a number of pixels at each position on the image along the first direction; and

the processing unit includes a delay unit which delays an output of the first image or an output of the second image in units of pixels, and performs the parallax correction such that the delay unit relatively delays the output of the first image or the output of the second image by the number of pixels represented by the bias amount corresponding to a position on the image along the first direction of a pixel row to be output when the first image and the second image are output to the generation unit in units of pixels in a parallel manner, while switching the number of pixels to be delayed by the delay unit according to a change in the position on the image along the first direction of the pixel row to be output.

5. The image processing apparatus of claim 4, wherein:

the delay unit includes a plurality of serially connected delay sections which delay the output of the first image or the output of the second image by one pixel; and

the processing unit selects data, which has passed through a number of the delay sections corresponding to the number of pixels represented by the bias amount corresponding to the position on the image along the first direction of the pixel row to be output, as data to be output as the first image or the second image, thereby delaying the data to be output as the first image or the second image by the number of pixels.

6. The image processing apparatus of claim 1, wherein the processing unit excludes a pixel row, a position of which on the image along the first direction is in a range corresponding to an upper side in the vertical direction as compared with a position corresponding to a predetermined horizontal line, from an object to be subjected to the parallax correction.

7. The image processing apparatus of claim 1, further comprising a correction unit that corrects at least one of a difference in imaging ranges along the horizontal direction of the first imaging apparatus and the second imaging apparatus, a difference in imaging magnifications, a difference in rotation angles around an imaging optical axis, or a difference in intensities, with respect to the first image and the second image acquired by the acquisition unit, wherein the processing unit performs the parallax correction with respect to the first image and the second image corrected by the correction unit.

8. The image processing apparatus of claim 1, further comprising a three-dimensional object detection unit that detects a three-dimensional object which exists in imaging ranges of the first imaging apparatus and the second imaging apparatus based on the differential information generated by the generation unit.

9. The image processing apparatus of claim 8, further comprising an output unit that outputs a detection result of the three-dimensional object by the three-dimensional object detection unit.

10. The image processing apparatus of claim 8, wherein:

the generation unit generates a differential image which represents the difference between the first image and the second image for each pixel as the differential information; and

the three-dimensional object detection unit extracts an image area corresponding to the three-dimensional object from the differential image based on a distribution on the differential image of differential pixels which exist in the differential image generated by the generation unit and which indicate a presence of the difference between the first image and the second image, thereby detecting a position on an image of the image area corresponding to the three-dimensional object.

11. The image processing apparatus of claim 8, wherein:

the generation unit generates a differential image which represents the difference between the first image and the second image for each pixel as the differential information; and

the three-dimensional object detection unit extracts an image area corresponding to the three-dimensional object from the differential image generated by the generation unit, and detects a distance to the three-dimensional object corresponding to the image area based on a width of a linear differential area which includes differential pixels which exist in the extracted image area and which indicate a presence of the difference between the first image and the second image.

12. The image processing apparatus of claim 8, wherein:

the generation unit generates a differential image which represents the difference between the first image and the second image for each pixel as the differential information; and

the three-dimensional object detection unit extracts an image area corresponding to the three-dimensional object from the differential image generated by the generation unit, repeats a contraction processing with respect to differential pixels which exist in the differential image and which represent a presence of the difference between the first image and the second image while determining whether a linear differential area, which includes the differential pixels existing in the extracted image area, has disappeared, and detects a distance to the three-dimensional object corresponding to the image area based on a number of executions of the contraction processing at a time point at which the linear differential area has disappeared from the image area.

13. The image processing apparatus of claim 9, wherein:

the generation unit generates a differential image which represents the difference between the first image and the second image for each pixel as the differential information;

the three-dimensional object detection unit performs a contraction processing with respect to differential pixels which exist in the differential image and indicate a presence of the difference between the first image and the second image a plurality of times and determines whether a linear differential area which includes the differential pixels has disappeared from the differential image; and

the output unit outputs a three-dimensional object detection signal or switches a type of the three-dimensional object detection signal to be output according to whether a number of executions of the contraction processing is less than a threshold value at a time point at which the linear differential area has disappeared from the differential image, or whether the linear differential area has disappeared from the differential image at a time point at which the contraction processing has been performed a predetermined number of times.

14. The image processing apparatus of claim 1, wherein:

the generation unit generates a differential image which represents the difference between the first image and the second image for each pixel as the differential information; and

the image processing apparatus further includes a pixel number counting unit that counts the number of differential pixels which exist in the differential image generated by the generation unit and which indicate a presence of the difference between the first image and the second image, and outputs a counting result.

15. The image processing apparatus of claim 14, further comprising a designation unit that designates an area of the differential image, wherein the pixel number counting unit counts a number of the differential pixels which exist in the area designated from the differential image.

16. The image processing apparatus according to claim 1, wherein:

the generation unit generates a differential image which represents a differential between the first image and the second image subjected to the parallax correction using the processing unit for each pixel; and

the differential image generated by the generation unit is formed as a differential image which represents a differential between the first image and the second image in a state in which a geometric parallax is corrected by the parallax correction using the processing unit and the images are relatively shifted in the first direction by a predetermined shift amount.

17. The image processing apparatus according to claim 16, wherein:

the bias amount represented by the bias amount information stored in the storage unit is set as a bias amount corresponding to the geometric parallax; and

the processing unit relatively shifts the first image and the second image in the first direction by the predetermined shift amount before or after performing the parallax correction on the first image and the second image.

18. The image processing apparatus according to claim 16, wherein the bias amount represented by the bias amount information stored in the storage unit is set as a bias amount obtained by adding a bias amount corresponding to the predetermined shift amount to a bias amount corresponding to the geometric parallax, or the bias amount represented by the bias amount information stored in the storage unit is set as the bias amount corresponding to the geometric parallax and a bias amount corresponding to the shift amount is added thereto before the bias amount is used for the parallax correction using the processing unit.

19. The image processing apparatus according to claim 16, wherein:

the bias amount represented by the bias amount information stored in the storage unit is set as a bias amount corresponding to the geometric parallax; and

directions of the first imaging apparatus and the second imaging apparatus are set such that a distance between imaging optical axes of the first imaging apparatus and the second imaging apparatus becomes larger further away from the first and second imaging apparatuses, and the directions are adjusted such that the first image and the second image are relatively shifted in the first direction by the predetermined shift amount.

20. The image processing apparatus according to claim 16, further comprising a contraction processing unit that performs a contraction processing in the second direction for a first number of times and in the first direction for a second number of times which is greater than the first number of times, with respect to a differential pixel indicating the presence of a differential between the first image and the second image which exists in the differential image generated by the generation unit.

21. The image processing apparatus according to claim 16, wherein the shift amount is set to one half or less of a maximum value of a bias amount corresponding to the geometric parallax.

22. The image processing apparatus according to claim 1, wherein directions of the first imaging apparatus and the second imaging apparatus are set such that a distance between imaging optical axes of the first imaging apparatus and the second imaging apparatus becomes larger further away from the first and second imaging apparatuses.

23. An image processing method comprising:

a storage unit storing bias amount information representing a bias amount, which corresponds to a parallax, along a first direction of an image corresponding to a horizontal direction between a first image captured by a first imaging apparatus and a second image captured by a second imaging apparatus which is located at a position different from that of the first imaging apparatus in the horizontal direction, with respect to each position in a second direction of the image corresponding to a vertical direction;

an acquisition unit acquiring the first image captured by the first imaging apparatus and the second image captured by the second imaging apparatus;

a processing unit performing a parallax correction with respect to each pixel row along the first direction of the image, the parallax correction moving a position of the image in the first direction of a pixel row along the first direction of the image in accordance with the bias amount represented by the bias amount information corresponding to a position of the pixel row in the second direction of the image; and

a generating unit generating differential information representing a difference between the first image and the second image that have been subjected to the parallax correction by the processing unit.

24. (canceled)

25. A computer readable non-transitory medium storing a program executable by a computer connected to a storage unit, which stores bias amount information representing a bias amount, which corresponds to a parallax, along a first direction of an image corresponding to a horizontal direction between a first image captured by a first imaging apparatus and a second image captured by a second imaging apparatus which is located at a position different from that of the first imaging apparatus in the horizontal direction, with respect to each position in a second direction of the image corresponding to a vertical direction, to function as:

an acquisition unit acquiring the first image captured by the first imaging apparatus and the second image captured by the second imaging apparatus;

a processing unit performing a parallax correction with respect to each pixel row along the first direction of the image, the parallax correction moving a position of the image in the first direction of a pixel row along the first direction of the image in accordance with the bias amount represented by the bias amount information corresponding to a position of the pixel row in the second direction of the image; and

a generating unit generating differential information representing a difference between the first image and the second image that have been subjected to the parallax correction by the processing unit.