METHOD AND SYSTEM FOR DYNAMICALLY CALIBRATING VEHICULAR CAMERAS

• US 20140043473A1
• Filed: 01/20/2012
• Published: 02/13/2014
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, Rathi, Ghanshyam, He, Daan
• Granted Patent: US 9,357,208 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method of dynamically ascertaining the alignment of a vehicular camera. The method involves acquiring a sequence of images provided by the camera whilst the vehicle is in motion. For each range of steering angles, the method (i) selects a plurality of feature points in the images, (ii) tracks a motion trajectory for each selected feature point, and (iii) determines a vanishing point in the image plane based on the tracked motion trajectories. The method determines a vanishing line in the image plane based on a locus of these vanishing points and determines the alignment of the camera based on the position of a central vanishing point (corresponding to the zero degree angle) and the vanishing line.

Claims

1. A method for dynamically ascertaining the position of a vehicular camera relative to a vehicle to which the camera is attached, the method comprising:

(a) establishing a plurality of vehicular steering angle ranges;

(b) acquiring a set of image frames in a video stream provided by the camera whilst the vehicle is in motion, the image frames defining an image plane;

(c) measuring the steering angle of the vehicle and, for each steering angle range;

(i) selecting a plurality of feature points in the image frames;

(ii) tracking a motion trajectory of each selected feature point in the set of image frames, and

(iii) determining a vanishing point in the image plane for the plurality of tracked motion trajectories;

(d) determining a vanishing line in the image plane provided by the camera based on a locus of said vanishing points; and

(e) determining the position or alignment of the camera based on the position of the vanishing line and the position of a central vanishing point, the central vanishing point being the vanishing point that is determined for the steering angle range that encompasses zero degrees.

2. A method according to claim 1, including removing distortion in the motion trajectories caused by use of a wide angle camera lens.

3. A method according to claim 2, including linearly fitting the undistorted motion trajectories to generate straight lines.

4. A method according to claim 3, including determining the vanishing point in a given steering range by finding a substantial intersection point for the distortion free, straight line tracked motion trajectories in the given steering range.

5. A method according to claim 4, wherein the camera is a front facing camera and rotational angles α

and β

of the camera are determined as a function of d1 and d2, where

α

is the rotation of the camera around an X-axis in a camera coordinate system,

β

is the rotation of the camera around a Y-axis in the camera coordinate system,

d1 is a distance in the image plane from a principal point of the camera to a right angled projection point on the vanishing line, and

d2 is a distance in the image plane along the vanishing line from the projection point to the central vanishing point.

6. A method according to claim 5, wherein α

and β

are determined from a table correlating unique values of d1 and d2 with unique values of α and

β

, said table created by setting the α and

β

positions of the front camera to known quantities and recording the resultant d1 and d2 values.

7. A method according to claim 5, wherein a rotational angle γ

of the camera is determined by solving for γ

as follows;

A cos γ

+B sin γ

=C

where,

A−

f sin α

sin β

B=f cos α

C−

{cvpX−

axisX) sin α

cos β

where γ

is the rotation angle around the Z-axis of a vehicle coordinate system, f is the focal length of the camera, cvpX is an image row coordinate of the central vanishing point in the image plane, and axisX is an image column location of the principal point.

8. A method according to claim 7, wherein rotations Rx, Ry and Rz around X-, Y- and Z-axes, respectively, of the vehicle coordinate system are determined based on the rotations α

, β and

γ

around the X-, Y- and Z-axes of the camera coordinate system.

9. A method according to claim 4, wherein the camera is a side facing camera and rotational angles α

and β

of the side facing camera are determined as a function of d1 and d2, where

α

is a rotation of the side facing camera around an X-axis of a side facing camera coordinate system,

β

is a rotation of the side facing camera around a Y-axis of the side camera coordinate system,

d1 is a distance in the image plane from a principal point of the camera to a right angled projection point on the vanishing line, and

d2 is a distance in the image plane along the vanishing line from the projection point to the central vanishing point.

10. A method according to claim 9, wherein a and are determined from a table correlating unique values of d1 and d2 with unique values of Rx and Rz, said table created by setting the a and positions of the side camera to known quantities and recording the resultant d1 and d2 values.

11. A method according to claim 9, wherein a rotational angle γ

is determined by measuring an angle projected by the central vanishing point on a horizontal axis of the image plane.

12. A method according to claim 11, wherein rotations Rx, Ry and Rz, for the side camera around the X-, Y- and Z-axes, respectively, of the vehicle coordinate system are determined from the rotations α

, β and

γ

around the X-, Y- and Z-axes of the camera coordinate system.

13. A system for dynamically ascertaining the position of a vehicular camera relative to a vehicle to which the camera is attached, the system comprising a controller which:

(a) establishes a plurality of vehicular steering angle ranges;

(b) acquires a set of image frames in a video stream provided by the camera whilst the vehicle is in motion, wherein the image frames define an image plane;

(c) measures the steering angle of the vehicle and, for each steering angle range;

(i) selects a plurality of feature points in the image frames;

(ii) tracks a motion trajectory of each selected feature point in the set of image frames, and

(iii) determines a vanishing point in the image plane for the plurality of tracked motion trajectories;

(d) determines a vanishing line in the image plane provided by the camera based on a locus of said vanishing points; and

(e) determines the position or alignment of the camera based on the position of the vanishing line and the position of a central vanishing point, wherein the central vanishing point is the vanishing point that is determined for the steering angle range that encompasses zero degrees.

14. A system according to claim 13, wherein the controller removes distortion in the motion trajectories caused by use of a wide angle camera lens, linearly fits the undistorted motion trajectories to generate straight lines, and determines the vanishing point in a given steering range by finding a substantial intersection point for the distortion free, straight line tracked motion trajectories in the given steering range.

15. A system according to claim 14, wherein the camera is a front facing camera and rotational angles α

and β

of the camera are determined as a function of d1 and d2, where

α

is the rotation of the camera around an X-axis in a camera coordinate system,

β

is the rotation of the camera around a Y-axis in the camera coordinate system,

d1 is a distance in the image plane from a principal point of the camera to a right angled projection point on the vanishing line, and

d2 is a distance in the image plane along the vanishing line from the projection point to the central vanishing point.

16. A system according to claim 15, wherein α

and β

are determined from a table correlating unique values of d1 and d2 with unique values of α and

β

, said table created by setting the α and

β

positions of the front camera to known quantities and recording the resultant d1 and d2 values.

17. A system according to claim 16, wherein the controller determines a rotational angle γ

of the camera by solving for γ

as follows;

A cos γ

+B sin γ

=C

where,

A=f sin α

sin ’

B=f cos α

C=(cvpX−

axisX)sin α

cos 0β

where γ

is the rotation angle around the Z-axis of a vehicle coordinate system, f is the focal length of the camera, cvpX is an image row coordinate of the central vanishing point in the image plane, and axisX is an image column location of the principal point.

18. A system for dynamically ascertaining the position of a vehicular camera relative to a vehicle, the system comprising:

a camera disposed at a front portion of a vehicle equipped with the system, the camera having a forward field of view, wherein the camera is operable to capture image data;

a controller, wherein the controller is operable to establish a plurality of vehicular steering angle ranges of the equipped vehicle;

wherein the controller is operable to receive image data captured by the camera and to acquire a set of image frames in a video stream of image data captured by the camera while the equipped vehicle is in motion, and wherein the image frames define an image plane;

wherein the controller is operable to measure the steering angle of the equipped vehicle and, for each steering angle range, (i) select a plurality of feature points in the image frames, (ii) track a motion trajectory of each selected feature point in the set of image frames, and (iii) determine a vanishing point in the image plane for the plurality of tracked motion trajectories;

wherein the controller determines a vanishing line in the image plane provided by the camera based on a locus of said vanishing points;

wherein the controller determines the position or alignment of the camera based on the position of the vanishing line and the position of a central vanishing point, wherein the central vanishing point is the vanishing point that is determined for the steering angle range that encompasses zero degrees; and

wherein rotational angles α and

β

of the camera are determined as a function of d1 and d2, where;

α

is the rotation of the camera around an X-axis in a camera coordinate system,

β

is the rotation of the camera around a Y-axis in the camera coordinate system,

d1 is a distance in the image plane from a principal point of the camera to a right angled projection point on the vanishing line, and

d2 is a distance in the image plane along the vanishing line from the projection point to the central vanishing point.

19. A system according to claim 18, wherein α

and β

are determined from a table correlating unique values of d1 and d2 with unique values of α and

β

, said table created by setting the α and

β

positions of the front camera to known quantities and recording the resultant d1 and d2 values.

20. A system according to claim 18, wherein the controller determines a rotational angle γ

of the camera by solving for γ

as follows;

A cos γ

+B sin γ

=C

where,

A=f sin α

sin β

B=f cos α

C=(cvpX−

axisX)sin α

cos β

where γ

is the rotation angle around the Z-axis of a vehicle coordinate system, f is the focal length of the camera, cvpX is an image row coordinate of the central vanishing point in the image plane, and axisX is an image column location of the principal point.