Method and system for dynamically ascertaining alignment of vehicular cameras

• US 10,654,423 B2
• Filed: 12/04/2017
• Issued: 05/19/2020
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, He, Daan, Rathi, Ghanshyam
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Jean Baptiste, Jerry T
• Application Number:
  US15/830,116
• Publication Number:
  US 20180086284A1
• Time in Patent Office: 897 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: G03B 43/00 Testing correct operation of photographic apparatus or parts thereof
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method for dynamically ascertaining alignment of a vehicular camera relative to a vehicle to which the camera is attached includes determining a plurality of steering angle ranges for the vehicle, each of which is a range of steering angles that approximates straight vehicle motion over less than two seconds of travel time of the vehicle while the vehicle is in motion and turning. Image data captured by a camera of the vehicle is processed to determine a central vanishing point when the vehicle is in motion and moving straight. A plurality of feature points are selected in the image frames for each steering angle range, and a vanishing point for a plurality of tracked motion trajectories for each steering angle range is determined. An alignment of the camera is determined based at least in part on the determined central vanishing point and a determined vanishing line.

Claims

1. A method for dynamically ascertaining alignment of a vehicular camera relative to a vehicle to which the camera is attached, said method comprising:

(a) determining a plurality of steering angle ranges for the vehicle, wherein a steering angle range is a range of steering angles that approximates straight vehicle motion over less than two seconds of travel time of the vehicle while the vehicle is in motion and turning;

(b) measuring, via a steering angle sensor of a vehicle, steering angle of the vehicle while the vehicle is in motion;

(c) acquiring image frames of image data captured by a camera of the vehicle while the vehicle is in motion;

(d) processing image data captured by the camera via a processor of the vehicle;

(e) determining, via processing by the processor of image data captured by the camera, a central vanishing point when the vehicle is in motion and moving straight;

(f) responsive to processing by the processor of image data captured by the camera, selecting a plurality of feature points in the image frames for each steering angle range of the plurality of steering angle ranges while the vehicle is in motion and turning;

(g) tracking, via processing by the processor of image data captured by the camera, motion trajectory of each selected feature point in the image frames for each steering angle range of the plurality of steering angle ranges;

(h) determining, via processing by the processor of image data captured by the camera, a vanishing point in an image plane of the image frames for the plurality of tracked motion trajectories for each steering angle range of the plurality of steering angle ranges;

(i) determining a vanishing line in the image plane based on a locus of the determined vanishing points; and

(j) determining alignment of the camera based at least in part on the determined central vanishing point and the determined vanishing line.

2. The method of claim 1, wherein the camera comprises a forward viewing camera having a field of view at least forward of the vehicle.

3. The method of claim 1, wherein the camera comprises a rearward viewing camera having a field of view at least rearward of the vehicle.

4. The method of claim 1, wherein the camera comprises a sideward viewing camera having a field of view at least sideward of the vehicle.

5. The method of claim 1, wherein said method comprises determining alignment of the camera by processing image data captured by the camera to determine location of the vanishing point in the image frames and comparing the determined vanishing point location to a predetermined location in the image frames.

6. The method of claim 1, comprising removing distortion in the image frames caused by use of a wide angle camera lens of the camera.

7. The method of claim 1, wherein determining the vanishing line in the image plane comprises extracting and tracking trajectories of feature points for multiple steering angle ranges while the vehicle is in motion and turning.

8. The method of claim 1, wherein the camera is part of a surround view system of the vehicle.

9. The method of claim 1, comprising calibrating the camera responsive to determination of misalignment of the camera.

10. The method of claim 9, wherein the camera is part of a surround view system of the vehicle, and wherein said method comprises determining corresponding features in an overlapping region of the camera'"'"'s field of view that overlaps a region of a field of view of another camera, and wherein calibrating the camera is responsive at least in part to the determined corresponding features in the overlapping region of the camera'"'"'s field of view and the other camera'"'"'s field of view.

11. A method for dynamically ascertaining alignment of a vehicular camera relative to a vehicle to which the camera is attached, said method comprising:

(a) determining a plurality of steering angle ranges for the vehicle, wherein a steering angle range is a range of steering angles that approximates straight vehicle motion over less than two seconds of travel time of the vehicle while the vehicle is in motion and turning;

(b) measuring, via a steering angle sensor of a vehicle, steering angle of the vehicle while the vehicle is in motion;

(c) acquiring image frames of image data captured by a camera of the vehicle while the vehicle is in motion, wherein the camera is part of a surround view system of the vehicle;

(d) processing image data captured by the camera via a processor of the vehicle;

(e) determining, via processing by the processor of image data captured by the camera, a central vanishing point when the vehicle is in motion and moving straight;

(f) responsive to processing by the processor of image data captured by the camera, selecting a plurality of feature points in the image frames for each steering angle range of the plurality of steering angle ranges while the vehicle is in motion and turning;

(g) tracking, via processing by the processor of image data captured by the camera, motion trajectory of each selected feature point in the image frames for each steering angle range of the plurality of steering angle ranges;

(h) determining, via processing by the processor of image data captured by the camera, a vanishing point in an image plane of the image frames for the plurality of tracked motion trajectories for each steering angle range of the plurality of steering angle ranges;

(i) determining a vanishing line in the image plane based on a locus of the determined vanishing points, wherein determining the vanishing line in the image plane comprises extracting and tracking trajectories of feature points for multiple steering angle ranges while the vehicle is in motion and turning; and

(j) determining alignment of the camera based at least in part on the determined central vanishing point and the determined vanishing line.

12. The method of claim 11, wherein the camera comprises a forward viewing camera having a field of view at least forward of the vehicle.

13. The method of claim 11, wherein the camera comprises a rearward viewing camera having a field of view at least rearward of the vehicle.

14. The method of claim 11, wherein the camera comprises a sideward viewing camera having a field of view at least sideward of the vehicle.

15. The method of claim 11, wherein said method comprises determining alignment of the camera by processing image data captured by the camera to determine location of the vanishing point in the image frames and comparing the determined vanishing point location to a predetermined location in the image frames.

16. The method of claim 11, comprising removing distortion in the image frames caused by use of a wide angle camera lens of the camera.

17. A method for dynamically ascertaining alignment of a vehicular camera relative to a vehicle to which the camera is attached, said method comprising:

(a) determining a plurality of steering angle ranges for the vehicle, wherein a steering angle range is a range of steering angles that approximates straight vehicle motion over less than two seconds of travel time of the vehicle while the vehicle is in motion and turning;

(b) measuring, via a steering angle sensor of a vehicle, steering angle of the vehicle while the vehicle is in motion;

(c) acquiring image frames of image data captured by a camera of the vehicle while the vehicle is in motion, wherein the camera is part of a surround view system of the vehicle;

(d) processing image data captured by the camera via a processor of the vehicle;

(e) determining, via processing by the processor of image data captured by the camera, a central vanishing point when the vehicle is in motion and moving straight;

(f) responsive to processing by the processor of image data captured by the camera, selecting a plurality of feature points in the image frames for each steering angle range of the plurality of steering angle ranges while the vehicle is in motion and turning;

(g) tracking, via processing by the processor of image data captured by the camera, motion trajectory of each selected feature point in the image frames for each steering angle range of the plurality of steering angle ranges;

(h) determining, via processing by the processor of image data captured by the camera, a vanishing point in an image plane of the image frames for the plurality of tracked motion trajectories for each steering angle range of the plurality of steering angle ranges;

(i) determining a vanishing line in the image plane based on a locus of the determined vanishing points;

(j) determining a rotational angle of the camera using camera intrinsic data and the determined central vanishing point and determined vanishing line;

(k) determining misalignment of the camera based at least in part on the determined rotational angle of the camera, the determined central vanishing point and the determined vanishing line; and

(l) calibrating the camera responsive to determination of misalignment of the camera.

18. The method of claim 17, wherein the camera comprises a forward viewing camera having a field of view at least forward of the vehicle, and wherein the other camera comprises a sideward viewing camera having a field of view at least sideward of the vehicle.

19. The method of claim 17, wherein the camera comprises a rearward viewing camera having a field of view at least rearward of the vehicle, and wherein the other camera comprises a sideward viewing camera having a field of view at least sideward of the vehicle.

20. The method of claim 17, wherein the camera comprises a sideward viewing camera having a field of view at least sideward of the vehicle, and wherein the other camera comprises a forward viewing camera having a field of view at least forward of the vehicle or a rearward viewing camera having a field of view at least rearward of the vehicle.