METHOD FOR DYNAMICALLY CALIBRATING VEHICULAR CAMERAS

• US 20160267657A1
• Filed: 05/23/2016
• Published: 09/15/2016
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, He, Daan, Rathi, Ghanshyam
• Granted Patent: US 10,202,077 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for dynamically correcting misalignment of a vehicular camera includes fixedly disposing a camera at a vehicle and operating the camera to acquire multiple frames of image data whilst the vehicle is moving generally in a straight line. A plurality of sets of feature points are selected in an image frame, with each set including a first feature point and a second feature point. For each set of feature points, a motion trajectory of that set'"'"'s feature points is tracked in subsequent image frames. For each tracked first and second feature points, a vanishing point is established in the image plane. Based on the established vanishing point, a vanishing line is determined in the image plane. When the vanishing line is determined to be non-horizontal in the image plane, at least one of pitch, roll or yaw of the camera is adjusted to correct rotational misalignment of the camera.

Claims

1. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a vehicle, said camera having a field of view exterior of the equipped vehicle;

(b) driving the equipped vehicle so that the equipped vehicle is in motion and is moving generally in a straight line;

(c) operating said camera to acquire multiple frames of image data captured by said camera whilst the equipped vehicle is in motion;

(d) selecting a plurality of sets of feature points in an image frame, wherein each set comprises a first feature point and a second feature point;

(e) for each set of feature points, tracking in subsequent image frames a motion trajectory of that set'"'"'s first feature point and second feature point;

(f) for each tracked first feature point and second feature point, establishing a vanishing point in the image plane;

(g) based on establishing vanishing points in the image plane by tracking respective first and second feature points for a plurality of sets of feature points, determining a vanishing line in the image plane provided by said camera based on a locus of the vanishing points;

(h) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(i) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

2. A method according to claim 1, wherein, when the equipped vehicle turns from travelling generally in a straight line as a result of a change in its steering angle, motion of the vehicle is approximated over relatively short distances as straight line motion.

3. A method according to claim 2, wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.

4. A method according to claim 3, wherein approximation to straight line motion when the vehicle turns occurs for approximately 0.5 to 2 seconds of travel time after turning from straight line travel.

5. A method according to claim 1, wherein said camera comprises a wide angle lens and wherein said method comprises removing distortion caused by use of the wide angle lens.

6. A method according to claim 1, wherein said camera is part of a multi-camera vision system of the equipped vehicle.

7. A method according to claim 6, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras.

8. A method according to claim 7, wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

9. A method according to claim 1, wherein a control receives and processes image data captured by said camera, said control receiving vehicle data via a communication bus of the equipped vehicle.

10. A method according to claim 9, wherein the received vehicle data includes vehicle speed data and vehicle steering angle data.

11. A method according to claim 10, wherein the communication bus of the equipped vehicle comprised a CAN bus.

12. A method according to claim 11, wherein said camera captures at least 25 frames of image data per second.

13. A method according to claim 12, wherein image data captured by said camera is provided to a digital signal processor that, at least in part, processes received image data to correct rotational misalignment of said camera.

14. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a portion of a vehicle, said camera having a field of view exterior of the equipped vehicle and wherein said camera comprises a wide angle lens;

(b) driving the equipped vehicle so that the equipped vehicle is in motion and is moving generally in a straight line;

(c) operating said camera to acquire multiple frames of image data captured by said camera whilst the equipped vehicle is in motion;

(d) removing distortion caused by use of the wide angle lens;

(e) selecting a plurality of sets of feature points in an image frame, wherein each set comprises a first feature point and a second feature point;

(f) for each set of feature points, tracking in subsequent image frames a motion trajectory of that set'"'"'s first feature point and second feature point;

(g) for each tracked first feature point and second feature point, establishing a vanishing point in the image plane;

(h) based on establishing vanishing points in the image plane by tracking respective first and second feature points for a plurality of sets of feature points, determining a vanishing line in the image plane provided by said camera based on a locus of the vanishing points;

(i) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(j) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

15. A method according to claim 14, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras, and wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

16. A method according to claim 15, wherein image data captured by the four cameras of said multi-camera vision system is stitched together to provide a composite 360 degree image for display to a driver of the equipped vehicle.

17. A method according to claim 16, wherein a control receives and process image data captured by said camera, said control receiving vehicle data via a communication bus of the equipped vehicle.

18. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a portion of a vehicle, said camera having a field of view exterior of the equipped vehicle and wherein said camera comprises a wide angle lens;

(b) driving the equipped vehicle so that the equipped vehicle is in motion and is moving generally in a straight line;

(c) operating said camera to acquire multiple frames of image data captured by said camera whilst the equipped vehicle is in motion;

(d) removing distortion caused by use of the wide angle lens;

(e) receiving vehicle data via a communication bus of the equipped vehicle, the received vehicle data including speed data and steering angle data of the equipped vehicle;

(f) selecting a plurality of sets of feature points in an image frame, wherein each set comprises a first feature point and a second feature point;

(g) for each set of feature points, tracking in subsequent image frames a motion trajectory of that set'"'"'s first feature point and second feature point;

(h) for each tracked first feature point and second feature point, establishing a vanishing point in the image plane;

(i) based on establishing vanishing points in the image plane by tracking respective first and second feature points for a plurality of sets of feature points, determining a vanishing line in the image plane provided by said camera based on a locus of the vanishing points;

(j) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(k) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

19. A method according to claim 18, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras, and wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

20. A method according to claim 19, wherein, when the equipped vehicle turns from travelling generally in a straight line as a result of a change in its steering angle, motion of the vehicle is approximated over relatively short distances as straight line motion, and wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.