METHOD FOR DYNAMICALLY CALIBRATING VEHICULAR CAMERAS

• US 20190168670A1
• Filed: 02/04/2019
• Published: 06/06/2019
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, He, Daan, Rathi, Ghanshyam
• Granted Patent: US 10,640,041 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for dynamically calibrating without manual intervention a forward viewing vehicular camera with respect to its three rotational degrees of freedom includes disposing a camera at a vehicle and operating the camera to acquire multiple frames of image data as the vehicle is moving and steered through a plurality of steering angles. Feature points are determined in an image frame when the vehicle is steered within a respective range of steering angles and motion trajectories of those feature points are tracked in subsequent image frames for the respective range of steering angles. A vanishing point is established in the image plane for the feature points for the respective range of steering angles. Based on established vanishing points, a vanishing line is determined. When the vanishing line is determined to be non-horizontal, at least one of pitch, roll or yaw of the camera is adjusted.

Claims

1. A method of dynamically calibrating without manual intervention a forward viewing vehicular camera with respect to its three rotational degrees of freedom, the method comprising:

disposing a forward viewing camera at a vehicle, the forward viewing camera, when disposed at the vehicle, having a field of view exterior and forward of the vehicle;

providing a control having a processor operable for processing image data captured by the forward viewing camera;

driving the vehicle so that the vehicle is in motion;

wherein driving the vehicle comprises steering the vehicle so as to establish a plurality of vehicle steering angles;

operating the forward viewing camera to acquire multiple frames of image data captured by the forward viewing camera while the vehicle is in motion and steered within at least two ranges of steering angles, wherein each frame of captured image data defines an image plane having a horizontal axis and a vertical axis;

determining, via processing at the control of a first image frame of image data captured by the forward viewing camera, first and second feature points in the first image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the first image frame, a motion trajectory of the first feature point and of the second feature point while the vehicle is moving and steered within a first range of steering angles;

determining a first vanishing point based on the tracked first feature point and second feature point;

determining, via processing at the control of a second image frame of image data captured by the forward viewing camera, third and fourth feature points in the second image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the second image frame, a motion trajectory of the third feature point and of the fourth feature point while the vehicle is moving and steered within a second range of steering angles that is different from the first range of steering angles;

determining a second vanishing point based on the tracked third feature point and fourth feature point;

determining a vanishing line based on the first and second vanishing points;

determining whether the vanishing line is non-parallel to the horizontal axis of the image plane; and

responsive to determining that the vanishing line is non-parallel to the horizontal axis of the image plane, adjusting at least one selected from the group consisting of pitch of the forward viewing camera, roll of the forward viewing camera and yaw of the forward viewing camera to correct for rotational misalignment of the forward viewing camera.

2. The method of claim 1, wherein the first range of steering angles comprises steering angles close to or at zero degrees such that the vehicle is traveling in a straight line.

3. The method of claim 2, wherein, when the vehicle turns, motion of the vehicle is approximated over relatively short distances as straight line motion.

4. The method of claim 3, wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.

5. The method of claim 4, wherein approximation to straight line motion when the vehicle turns occurs for 0.5 to 2 seconds of travel time after turning from straight line travel.

6. The method of claim 1, wherein image data captured by the forward viewing camera is processed at the control for at least one selected from the group consisting of (i) object detection, (ii) lane departure warning and (iii) automatic high beam control.

7. The method of claim 1, wherein the forward viewing camera comprises a wide angle lens, and wherein the method comprises removing distortion caused by use of the wide angle lens.

8. The method of claim 1, wherein the forward viewing camera is part of a multi-camera vision system of the vehicle.

9. The method of claim 8, wherein the multi-camera vision system of the vehicle comprises the forward viewing camera and three other vehicular cameras.

10. The method of claim 9, wherein image data captured by at least two of the four cameras of the multi-camera vision system is stitched together to provide a composite image for display to a driver of the vehicle.

11. The method of claim 1, comprising receiving at the control vehicle data via a communication bus of the vehicle.

12. The method of claim 11, wherein the received vehicle data includes vehicle speed data and vehicle steering angle data.

13. The method of claim 11, wherein the communication bus of the vehicle comprised a CAN bus.

14. The method of claim 13, wherein the forward viewing camera captures at least 25 frames of image data per second.

15. A method of dynamically calibrating without manual intervention a forward viewing vehicular camera with respect to its three rotational degrees of freedom, the method comprising:

disposing a forward viewing camera at a vehicle, the forward viewing camera, when disposed at the vehicle, having a field of view exterior and forward of the vehicle;

providing a control having a processor operable for processing image data captured by the forward viewing camera;

driving the vehicle so that the vehicle is in motion;

wherein driving the vehicle comprises steering the vehicle so as to establish a plurality of vehicle steering angles;

operating the forward viewing camera to acquire multiple frames of image data captured by the forward viewing camera while the vehicle is in motion and steered within at least two ranges of steering angles, wherein each frame of captured image data defines an image plane having a horizontal axis and a vertical axis;

determining, via processing at the control of a first image frame of image data captured by the forward viewing camera, first and second feature points in the first image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the first image frame, a motion trajectory of the first feature point and of the second feature point while the vehicle is moving and steered within a first range of steering angles, wherein the first range of steering angles comprises steering angles close to or at zero degrees such that the vehicle is traveling in a straight line;

determining a first vanishing point based on the tracked first feature point and second feature point;

determining, via processing at the control of a second image frame of image data captured by the forward viewing camera, third and fourth feature points in the second image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the second image frame, a motion trajectory of the third feature point and of the fourth feature point while the vehicle is moving and steered within a second range of steering angles, wherein the second range of steering angles comprises steering angles that do not include angles close to or at zero degrees such that the vehicle is traveling along a curve;

determining a second vanishing point based on the tracked third feature point and fourth feature point;

determining a vanishing line based on the first and second vanishing points;

determining whether the vanishing line is non-parallel to the horizontal axis of the image plane; and

responsive to determining that the vanishing line is non-parallel to the horizontal axis of the image plane, adjusting at least one selected from the group consisting of pitch of the forward viewing camera, roll of the forward viewing camera and yaw of the forward viewing camera to correct for rotational misalignment of the forward viewing camera.

16. The method of claim 15, wherein image data captured by the forward viewing camera is processed at the control for at least one selected from the group consisting of (i) object detection, (ii) lane departure warning and (iii) automatic high beam control.

17. The method of claim 15, wherein the forward viewing camera comprises a wide angle lens, and wherein the method comprises removing distortion caused by use of the wide angle lens.

18. The method of claim 15, wherein the forward viewing camera is part of a multi-camera vision system of the vehicle, and wherein the multi-camera vision system of the vehicle comprises the forward viewing camera and three other vehicular cameras.

19. The method of claim 18, wherein image data captured by at least two of the four cameras of the multi-camera vision system is stitched together to provide a composite image for display to a driver of the vehicle.

20. The method of claim 15, comprising receiving at the control vehicle data via a communication bus of the vehicle, and wherein the received vehicle data includes vehicle speed data and vehicle steering angle data.

21. A method of dynamically calibrating without manual intervention a forward viewing vehicular camera with respect to its three rotational degrees of freedom, the method comprising:

disposing a forward viewing camera at a vehicle, the forward viewing camera, when disposed at the vehicle, having a field of view exterior and forward of the vehicle;

providing a control having a processor operable for processing image data captured by the forward viewing camera;

driving the vehicle so that the vehicle is in motion;

wherein driving the vehicle comprises steering the vehicle so as to establish a plurality of vehicle steering angles;

operating the forward viewing camera to acquire multiple frames of image data captured by the forward viewing camera while the vehicle is in motion and steered within at least two ranges of steering angles, wherein each frame of captured image data defines an image plane having a horizontal axis and a vertical axis;

determining, via processing at the control of a first image frame of image data captured by the forward viewing camera, first and second feature points in the first image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the first image frame, a motion trajectory of the first feature point and of the second feature point while the vehicle is moving and steered within a first range of steering angles;

determining a first vanishing point based on the tracked first feature point and second feature point;

determining, via processing at the control of a second image frame of image data captured by the forward viewing camera, third and fourth feature points in the second image frame of captured image data;

tracking, via processing at the control of image frames of image data captured by the forward viewing camera subsequent to the second image frame, a motion trajectory of the third feature point and of the fourth feature point while the vehicle is moving and steered within a second range of steering angles that is different from the first range of steering angles;

determining a second vanishing point based on the tracked third feature point and fourth feature point;

determining a vanishing line based on the first and second vanishing points;

determining whether the vanishing line is non-parallel to the horizontal axis of the image plane;

receiving at the control vehicle data via a communication bus of the vehicle, and wherein the received vehicle data includes vehicle speed data and vehicle steering angle data;

responsive to determining that the vanishing line is non-parallel to the horizontal axis of the image plane, adjusting at least one selected from the group consisting of pitch of the forward viewing camera, roll of the forward viewing camera and yaw of the forward viewing camera to correct for rotational misalignment of the forward viewing camera; and

wherein image data captured by the forward viewing camera is processed at the control for at least one selected from the group consisting of (i) object detection, (ii) lane departure warning and (iii) automatic high beam control.

22. The method of claim 21, wherein, when the vehicle turns, motion of the vehicle is approximated over relatively short distances as straight line motion.

23. The method of claim 22, wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.

24. The method of claim 23, wherein, when the vehicle turns from steering angles close to or at zero degrees, approximation to straight line motion occurs for 0.5 to 2 seconds of travel time after the vehicle turns.