Calibration device, calibration method, and non-transitory computer-readable storage medium for the same

• US 10,546,380 B2
• Filed: 07/15/2016
• Issued: 01/28/2020
• Est. Priority Date: 08/05/2015
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Tanaka, Hitoshi
• Primary Examiner(s): Grant, II, Jerome
• Application Number:
  US15/749,908
• Publication Number:
  US 20180232893A1
• Time in Patent Office: 1,292 Days
• Field of Search: 382103
• CPC Class Codes: B60Q 9/00 Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 , e.g. haptic signalling
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: G01P 1/08 Arrangements of scales, pointers, lamps or acoustic indicators, e.g. in automobile speedometers
• CPC Class Codes: G01P 3/00 Measuring linear or angular speed; Measuring differences of linear or angular speeds G01P5/00 - G01P11/00 take precedence; direction and speed indication G01P13/045; counting mechanisms G06M
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A calibration device acquires an image of an imaging target located ahead of a vehicle. Images of the imaging targets behind the vehicle are acquired, which are different from the acquired images captured in the forward direction. The shape and size of the imaging targets are already known. The calibration device performs projective transformation for the imaging targets shown in overlap regions of the acquired images, and identifies a roll, a pitch and a vertical position of each camera such that the conversion results are in agreement with the shape and size of the targets. The calibration device identifies an x-coordinate, a y-coordinate, and a yaw of each camera such that one link angle in a virtual link structure is minimized.

Claims

1. A calibration device identifying attitude parameters of a plurality of cameras mounted to a vehicle, wherein:

the plurality of cameras are mounted to the vehicle so as to capture images of specified regions different from each other and defined around the vehicle, each specified region having overlap regions where the specified region partially overlap with another specified region;

the plurality of cameras include a first camera capturing an image in a forward direction of the vehicle, a second camera capturing an image in a rightward direction of the vehicle, a third camera capturing an image in a rearward direction of the vehicle, and a fourth camera capturing an image in a leftward direction of the vehicle;

the calibration device comprises;

an image acquisition unit acquiring images captured by the plurality of cameras;

a first identification unit acting on imaging targets with given shape and size to perform projective transformation for the imaging targets shown in the overlap regions of each image acquired by the image acquisition unit and identify a roll, a pitch and a vertical position among the attitude parameters of each of the plurality of cameras so that conversion results are consistent with the shape and size of the imaging target, the roll being a rotation angle of each camera about a lens center axis of the camera, the pitch being a rotation angle of each camera about a horizontal axis of the camera, the vertical position being a coordinate in a height direction of the vehicle; and

a second identification unit acting on a virtual link structure to identify an x-coordinate, a y-coordinate and a yaw among the attitude parameters of each of the plurality of cameras so that one link angle of the virtual link structure is minimized, the x-coordinate being a coordinate in a longitudinal direction of the vehicle, the y-coordinate being a coordinate in a lateral direction of the vehicle, the yaw being a rotation angle of the vehicle about an axis perpendicular to a horizontal plane, the virtual structure being formed of line segments serving as links and virtually connecting representative points of the imaging targets shown in the images acquired by the image acquisition unit, where the representative points of the imaging targets located at the same position and shown in the overlap regions of different images are permitted to coincide with each other;

the image acquisition unit comprises;

a first acquisition unit acquiring images of the imaging targets ahead of the vehicle captured by the first camera, the second camera, and the fourth camera; and

a second acquisition unit acquiring images of the imaging targets behind the vehicle captured by the second camera, the third camera, and the fourth camera, these images being different from the images acquired by the first acquisition unit;

and

the first identification unit and the second identification unit use images acquired by the first acquisition unit and images acquired by the second acquisition unit, as images acquired by the image acquisition unit to identify attitude parameters of each of the plurality of cameras.

2. The calibration device according to claim 1, wherein the imaging target is disposed in a space larger than the vehicle width of the vehicle:

the first acquisition unit acquires images captured by the first camera, the second camera, and the fourth camera, before the vehicle passes between the imaging targets;

the second acquisition unit acquires images captured by the second camera, the third camera, and the fourth camera, after the vehicle has passed between the imaging targets.

3. The image acquisition unit according to claim 1, wherein the image acquisition unit comprises:

a first determination unit determining whether the imaging target is shown in each image captured by the plurality of cameras; and

the first notification unit notifying that the imaging target is not shown when the first determination unit determines that the imaging target is not shown.

4. The calibration device according to claim 3, wherein:

the calibration device comprises a range determination unit determining a search region that is a target region in which a determination is made as to whether the imaging target is shown in each image captured by the plurality of cameras; and

the first determination unit determines whether the imaging target is not shown in the search region determined by the range setting unit.

5. The calibration device according to claim 3, wherein:

the calibration device comprises;

a speed acquisition unit acquiring a traveling speed of the vehicle; and

a second determination unit determining whether a traveling speed acquired by the speed acquisition unit is not more than a predetermined speed threshold;

and

the first determination unit determines whether the imaging target is shown in each image captured by the plurality of cameras, when the second determination unit determines that the traveling speed is not more than the speed threshold.

6. The calibration device according to claim 5, wherein:

the calibration device comprises a second notification unit notifying that the traveling speed is greater than the speed threshold, when the second determination unit determines that the traveling speed is greater than the speed threshold.

7. The calibration device according to claim 1, wherein:

the positional relationship between the imaging targets in the image acquired by the first acquisition unit is the same as the positional relationship between the imaging targets in the image acquired by the second acquisition unit; and

the second identification unit;

sequentially identifies representative points of the imaging targets shown in the overlap regions of respective images acquired by the image acquisition unit;

makes the identified representative points coincide with the imaging targets located at the corresponding positions to form the virtual link structure; and

when the representative points of the most recent imaging targets are identified along a time axis among the representative points of the imaging targets shown in the overlap regions, the x-coordinate, the y-coordinate, and the yaw of the attitude parameters of the plurality of cameras are identified based on the representative points of the most recent imaging targets.

8. A calibration method performed by a calibration device identifying attitude parameters of a plurality of cameras mounted to a vehicle, wherein:

the plurality of cameras are mounted to the vehicle so as to capture images of specified regions different from each other and defined around the vehicle, each specified region having overlap regions where the specified region partially overlap with another specified region;

the plurality of cameras include a first camera capturing an image in a forward direction of the vehicle, a second camera capturing an image in a rightward direction of the vehicle, a third camera capturing an image in a rearward direction of the vehicle, and a fourth camera capturing an image in a leftward direction of the vehicle;

the calibration method comprises;

an image acquisition step of acquiring images captured by the plurality of cameras;

a first identification step of acting on imaging targets with given shape and size to perform projective transformation for the imaging targets shown in the overlap regions of each image acquired by the image acquisition step and identify a roll, a pitch and a vertical position among the attitude parameters of each of the plurality of cameras so that conversion results are consistent with the shape and size of the imaging target, the roll being a rotation angle of each camera about a lens center axis of the camera, the pitch being a rotation angle of each camera about a horizontal axis of the camera, the vertical position being a coordinate in a height direction of the vehicle; and

a second identification step of acting on a virtual link structure to identify an x-coordinate, a y-coordinate and a yaw among the attitude parameters of each of the plurality of cameras so that one link angle of the virtual link structure is minimized, the x-coordinate being a coordinate in a longitudinal direction of the vehicle, the y-coordinate being a coordinate in a lateral direction of the vehicle, the yaw being a rotation angle of the vehicle about an axis perpendicular to a horizontal plane, the virtual structure being formed of line segments serving as links and virtually connecting representative points of the imaging targets shown in the images acquired by the image acquisition step, where the representative points of the imaging targets located at the same position and shown in the overlap regions of different images are permitted to coincide with each other;

the image acquisition step comprises;

a first acquisition step of acquiring images of the imaging targets ahead of the vehicle captured by the first camera, the second camera, and the fourth camera; and

a second acquisition step of acquiring images of the imaging targets behind the vehicle captured by the second camera, the third camera, and the fourth camera, these images being different from the images acquired by the first acquisition step;

and

the first identification step and the second identification step use images acquired through the first acquisition step and images acquired through the second acquisition step, as images acquired by the image acquisition step to identify attitude parameters of each of the plurality of cameras.

9. A non-transitory computer-readable storage medium containing thereon a program comprising instructions, causing a computer to execute a display control method for identifying attitude parameters of a plurality of cameras mounted to a vehicle by a calibration device, wherein:

the plurality of cameras are mounted to the vehicle so as to capture images of specified regions different from each other and defined around the vehicle, each specified region having overlap regions where the specified region partially overlap with another specified region;

the plurality of cameras include a first camera capturing an image in a forward direction of the vehicle, a second camera capturing an image in a rightward direction of the vehicle, a third camera capturing an image in a rearward direction of the vehicle, and a fourth camera capturing an image in a leftward direction of the vehicle;

the computer is permitted to function as;

an image acquisition step of acquiring images captured by the plurality of cameras;

a first identification step of acting on imaging targets with given shape and size to perform projective transformation for the imaging targets shown in the overlap regions of each image acquired by the image acquisition step and identify a roll, a pitch and a vertical position among the attitude parameters of each of the plurality of cameras so that conversion results are consistent with the shape and size of the imaging target, the roll being a rotation angle of each camera about a lens center axis of the camera, the pitch being a rotation angle of each camera about a horizontal axis of the camera, the vertical position being a coordinate in a height direction of the vehicle; and

a second identification step of acting on a virtual link structure to identify an x-coordinate, a y-coordinate and a yaw among the attitude parameters of each of the plurality of cameras so that one link angle of the virtual link structure is minimized, the x-coordinate being a coordinate in a longitudinal direction of the vehicle, the y-coordinate being a coordinate in a lateral direction of the vehicle, the yaw being a rotation angle of the vehicle about an axis perpendicular to a horizontal plane, the virtual structure being formed of line segments serving as links and virtually connecting representative points of the imaging targets shown in the images acquired by the image acquisition step, where the representative points of the imaging targets located at the same position and shown in the overlap regions of different images are permitted to coincide with each other;

the image acquisition step comprises;

a first acquisition step of acquiring images of the imaging targets ahead of the vehicle captured by the first camera, the second camera, and the fourth camera; and

a second acquisition step of acquiring images of the imaging targets behind the vehicle captured by the second camera, the third camera, and the fourth camera, these images being different from the images acquired by the first acquisition step;

and

the first identification step and the first identification step use images acquired by the first acquisition step and images acquired by the second acquisition step, as images acquired by the image acquisition step to identify attitude parameters of each of the plurality of cameras.