Method for dynamically calibrating vehicular cameras

• US 10,202,077 B2
• Filed: 05/23/2016
• Issued: 02/12/2019
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, He, Daan, Rathi, Ghanshyam
• Primary Examiner(s): Hallenbeck-Huber, Jeremiah C
• Application Number:
  US15/161,711
• Publication Number:
  US 20160267657A1
• Time in Patent Office: 995 Days
• Field of Search: 348148
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/8086 for vehicle path indication
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for dynamically correcting misalignment of a vehicular camera includes fixedly disposing a camera at a vehicle and operating the camera to acquire multiple frames of image data while the vehicle is moving generally in a straight line. A plurality of sets of feature points are selected in an image frame, with each set including a first feature point and a second feature point. For each set of feature points, a motion trajectory of that set'"'"'s feature points is tracked in subsequent image frames. For each tracked first and second feature points, a vanishing point is established in the image plane. Based on the established vanishing point, a vanishing line is determined in the image plane. When the vanishing line is determined to be non-horizontal in the image plane, at least one of pitch, roll or yaw of the camera is adjusted to correct rotational misalignment of the camera.

Claims

1. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a vehicle, said camera having a field of view exterior of the equipped vehicle;

(b) driving the equipped vehicle so that the equipped vehicle is in motion;

(c) operating said camera to acquire multiple frames of image data captured by said camera (i) while the equipped vehicle is moving in a straight line and (ii) while the equipped vehicle is moving along a curve;

(d) selecting a plurality of sets of feature points in an image frame of image data captured by said camera, wherein a first set of feature points comprises a first feature point and a second feature point;

(e) for the first set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s first feature point and second feature point while the equipped vehicle is moving in the straight line;

(f) for each tracked first feature point and second feature point while the equipped vehicle is moving in the straight line, establishing a first vanishing point in the image plane;

(g) wherein a second set of feature points comprises a third feature point and a fourth feature point;

(h) for the second set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s third feature point and fourth feature point while the equipped vehicle is moving along the curve;

(i) for each tracked third feature point and fourth feature point while the equipped vehicle is moving along the curve, establishing a second vanishing point in the image plane;

(j) based on establishing first and second vanishing points in the image plane by tracking respective feature points for a plurality of sets of feature points while the equipped vehicle is moving in the straight line and while the equipped vehicle is moving along the curve, determining a vanishing line in the image plane provided by said camera based on a locus of the first and second vanishing points;

(k) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(l) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

2. A method according to claim 1, wherein, when the equipped vehicle turns from travelling in the straight line as a result of a change in its steering angle, motion of the vehicle is approximated over relatively short distances as straight line motion.

3. A method according to claim 2, wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.

4. A method according to claim 3, wherein approximation to straight line motion when the vehicle turns occurs for 0.5 to 2 seconds of travel time after turning from straight line travel.

5. A method according to claim 1, wherein said camera comprises a wide angle lens and wherein said method comprises removing distortion caused by use of the wide angle lens.

6. A method according to claim 1, wherein said camera is part of a multi-camera vision system of the equipped vehicle.

7. A method according to claim 6, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras.

8. A method according to claim 7, wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

9. A method according to claim 1, wherein a control receives and processes image data captured by said camera, said control receiving vehicle data via a communication bus of the equipped vehicle.

10. A method according to claim 9, wherein the received vehicle data includes vehicle speed data and vehicle steering angle data.

11. A method according to claim 10, wherein the communication bus of the equipped vehicle comprised a CAN bus.

12. A method according to claim 11, wherein said camera captures at least 25 frames of image data per second.

13. A method according to claim 12, wherein image data captured by said camera is provided to a digital signal processor that, at least in part, processes received image data to correct rotational misalignment of said camera.

14. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a portion of a vehicle, said camera having a field of view exterior of the equipped vehicle and wherein said camera comprises a wide angle lens;

(b) driving the equipped vehicle so that the equipped vehicle is in motion;

(c) operating said camera to acquire multiple frames of image data captured by said camera (i) while the equipped vehicle is in motion moving in a straight line and (ii) while the equipped vehicle is moving along a curve;

(d) removing distortion caused by use of the wide angle lens;

(e) selecting a plurality of sets of feature points in an image frame of image data captured by said camera, wherein a first set of feature points comprises a first feature point and a second feature point;

(f) for the first set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s first feature point and second feature point while the equipped vehicle is moving in the straight line;

(g) for each tracked first feature point and second feature point while the equipped vehicle is moving in the straight line, establishing a first vanishing point in the image plane;

(h) wherein a second set of feature points comprises a third feature point and a fourth feature point;

(i) for the second set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s third feature point and fourth feature point while the equipped vehicle is moving along the curve;

(j) for each tracked third feature point and fourth feature point while the equipped vehicle is moving along the curve, establishing a second vanishing point in the image plane;

(k) based on establishing first and second vanishing points in the image plane by tracking respective feature points for a plurality of sets of feature points while the equipped vehicle is moving in the straight line and while the equipped vehicle is moving along the curve, determining a vanishing line in the image plane provided by said camera based on a locus of the first and second vanishing points;

(l) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(m) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

15. A method according to claim 14, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras, and wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

16. A method according to claim 15, wherein image data captured by the four cameras of said multi-camera vision system is stitched together to provide a composite 360 degree image for display to a driver of the equipped vehicle.

17. A method according to claim 16, wherein a control receives and process image data captured by said camera, said control receiving vehicle data via a communication bus of the equipped vehicle.

18. A method for dynamically correcting rotational misalignment of a vehicular camera, the method comprising:

(a) fixedly disposing a camera at a portion of a vehicle, said camera having a field of view exterior of the equipped vehicle and wherein said camera comprises a wide angle lens;

(b) driving the equipped vehicle so that the equipped vehicle is in motion;

(c) operating said camera to acquire multiple frames of image data captured by said camera (i) while the equipped vehicle is moving in a straight line and (ii) while the equipped vehicle is moving along a curve;

(d) removing distortion caused by use of the wide angle lens;

(e) receiving vehicle data via a communication bus of the equipped vehicle, the received vehicle data including speed data and steering angle data of the equipped vehicle;

(f) selecting a plurality of sets of feature points in an image frame of image data captured by said camera, wherein a first set of feature points comprises a first feature point and a second feature point;

(g) for the first set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s first feature point and second feature point while the equipped vehicle is moving in the straight line;

(h) for each tracked first feature point and second feature point while the equipped vehicle is moving in the straight line, establishing a first vanishing point in the image plane;

(i) wherein a second set of feature points comprises a third feature point and a fourth feature point;

(j) for the second set of feature points, tracking in subsequent image frames of image data captured by said camera a motion trajectory of that set'"'"'s third feature point and fourth feature point while the equipped vehicle is moving along the curve;

(k) for each tracked third feature point and fourth feature point while the equipped vehicle is moving along the curve, establishing a second vanishing point in the image plane;

(l) based on establishing first and second vanishing points in the image plane by tracking respective feature points for a plurality of sets of feature points while the equipped vehicle is moving in the straight line and while the equipped vehicle is moving along the curve, determining a vanishing line in the image plane provided by said camera based on a locus of the first and second vanishing points;

(m) determining whether the vanishing line is horizontal or non-horizontal in the image plane; and

(n) when the vanishing line is determined to be non-horizontal in the image plane, adjusting at least one of pitch, roll or yaw of said camera to correct rotational misalignment of said camera.

19. A method according to claim 18, wherein said camera is part of a multi-camera vision system of the equipped vehicle, and wherein said multi-camera vision system of the equipped vehicle comprises said camera and three other vehicular cameras, and wherein image data captured by at least two of the four cameras of said multi-camera vision system is stitched together to provide a composite image for display to a driver of the equipped vehicle.

20. A method according to claim 19, wherein, when the equipped vehicle turns from travelling in the straight line as a result of a change in its steering angle, motion of the vehicle is approximated over relatively short distances as straight line motion, and wherein approximation to straight line motion when the vehicle turns is vehicle speed dependent.