Stereo calibration apparatus and stereo image monitoring apparatus using the same

• US 7,313,265 B2
• Filed: 03/12/2004
• Issued: 12/25/2007
• Est. Priority Date: 03/13/2003
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Nakai, Hiroaki, Maruyama, Masayuki, Putthividhya, Duangmanee
• Primary Examiner(s): Ahmed; Samir
• Assistant Examiner(s): Allison; Andrae S
• Application Number:
  US10/798,383
• Publication Number:
  US 20040234124A1
• Time in Patent Office: 1,383 Days
• Field of Search: 382103-107, 382/154, 382/190, 382/199, 382/209, 382/266, 382/269, 348 42- 47, 348/143, 348/148, 348/118
• US Class Current: 382/154
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

The stereo calibration apparatus includes an image input device inputting an image from each of two image pickup devices, a straight-line detector detecting at least four straight lines from each of two input images, a straight-line correspondence detector detecting at least four sets of corresponding lines by using an image feature in a neighboring area of the straight line, and a transformation matrix calculator calculating a transformation matrix between two image pickup devices from the sets of corresponding lines.

Claims

1. A stereo calibration apparatus for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the stereo calibration apparatus comprising:

an image input device to which a pair of images is input from the one pair of image pickup devices;

a straight-line detector detecting at least four straight lines from each image of the pair of images output from the image input device;

a straight-line correspondence detector detecting at least four sets of corresponding lines between the pair of images by using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images; and

a transformation matrix calculator calculating the transformation matrix by using the sets of corresponding lines,

wherein the straight-line correspondence detector is configured to;

obtain data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtain data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtain normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determine one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

2. The stereo calibration apparatus according to claim 1, wherein the transformation matrix calculator calculates the transformation matrix in accordance with a linear expression by using the sets of corresponding lines and a projective transformation expression regarding the straight lines.

3. A stereo calibration method for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the method comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images by using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images;

calculating the transformation matrix between the image pickup devices by using the sets of corresponding lines;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

4. A computer-readable medium storing a computer program product tangibly embodying a program of instructions which is executed by a computer to perform a process for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the process comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images by using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images;

calculating the transformation matrix between the image pickup devices by using the sets of corresponding lines;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

5. A stereo image monitor for detecting an object, the stereo image monitor comprising:

a pair of image pickup devices configured to photograph a pair of images containing a common area;

a straight-line detector detecting at least four straight lines from each image of the pair of images;

a straight-line correspondence detector detecting at least four sets of corresponding lines between the pair of images by using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images;

a transformation matrix calculator calculating the transformation matrix by using the four sets of corresponding lines; and

a monitor transforming one image of the pair of images by using the transformation matrix to detect an object in the common area,

wherein the straight-line correspondence detector is configured to;

obtain data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtain data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtain normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determine one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

6. A stereo image monitoring method for monitoring a common area, which photographed by a pair of image pickup devices, the method comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images;

calculating a transformation matrix between the image pickup devices by using the four sets of corresponding lines;

transforming one image of the pair of images by using the transformation matrix to detect an object in the common area;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

7. A computer-readable medium storing a computer program product tangibly embodying a program of instructions which is executed by a computer to perform a process for monitoring a common area, which photographed by a pair of image pickup devices, the process comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images using an image feature in a neighboring area of each of the four straight lines detected for each image of the pair of images;

calculating a transformation matrix between the image pickup devices by using the four sets of corresponding lines;

transforming one image of the pair of images by using the transformation matrix to detect an object in the common area;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

8. A stereo calibration apparatus for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the stereo calibration apparatus comprising:

an image input device to which a pair of images is input from the one pair of image pickup devices;

a straight-line detector detecting at least four straight lines from each image of the pair of images output from the image input device;

a straight-line correspondence detector detecting at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images; and

a transformation matrix calculator calculating the transformation matrix by using the sets of corresponding lines,

wherein the straight-line correspondence detector is configured to;

obtain data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtain data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtain normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determine one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

9. The stereo calibration apparatus according to claim 8, wherein the straight-line correspondence detector obtains the geometrical relation among each of the four straight lines detected for each image of the pair of images by using an intersection between one of the four straight lines in each image and the other of the four straight lines in each image.

10. The stereo calibration apparatus according to claim 8, wherein the straight-line correspondence detector uses an image feature in a neighboring area of the four straight lines and the geometrical relation to obtain the sets of corresponding lines.

11. A stereo calibration method for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the method comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images;

calculating the transformation matrix between the image pickup devices by using the sets of corresponding lines;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

12. A computer-readable medium storing a computer program product tangibly embodying a program of instructions causing a computer to perform a process for obtaining a transformation matrix, which is an image transformation parameter between a pair of image pickup devices for photographing a preset common area, the process comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

detecting at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images;

calculating the transformation matrix between the image pickup devices by using the sets of corresponding lines;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

13. A stereo image monitor for detecting an object, the stereo image monitor comprising:

a pair of image pickup devices configured to photograph images containing a common area;

an image input device to which a pair of images is input from the image pickup devices;

a straight-line detector detecting at least four straight lines from each image of the pair of images output from the image input device;

a straight-line correspondence detector detecting at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images;

a transformation matrix calculator calculating the transformation matrix by using the sets of corresponding lines; and

a monitor transforming one image of the pair of images by using the transformation matrix to detect an object in the common area,

wherein the straight-line correspondence detector is configured to;

obtain data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtain data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtain normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determine one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

14. A stereo image monitoring method for monitoring a common area, which photographed by a pair of image pickup devices, the method comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

obtaining at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images;

calculating a transformation matrix between the image pickup devices by using the four sets of corresponding lines;

transforming one of the images of the pair of images by using the transformation matrix to detect an object in the common area;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.

15. A computer-readable medium storing a computer program product tangibly embodying a program of instructions which is executed by a computer to perform a process for monitoring a common area, which photographed by a pair of image pickup devices, the process comprising:

inputting a pair of images from the image pickup devices;

detecting at least four straight lines from each image of the pair of images;

obtaining at least four sets of corresponding lines between the pair of images by using a geometrical relation among each of the four straight lines detected for each image of the pair of images;

calculating a transformation matrix between the image pickup devices by using the four sets of corresponding lines;

transforming one of the images of the pair of images by using the transformation matrix to detect an object in the common area;

obtaining a data series regarding the image feature in the neighboring area for each of the four straight lines detected for each image of the pair of images;

obtaining data differences between each of data in the data series of each of the four straight lines detected for a first image of the pair of images and each of data in the data series of each of the four straight lines detected for a second image of the pair of images;

obtaining normalized distances by normalizing a weighted sum of the data differences using a sum of weighting factors; and

determining one of the four straight lines in the first image of the pair of images and the corresponding one of the four straight lines in the second image of the pair of images having a smallest normalized distance as a pair of corresponding lines.