Vehicle vision system with targetless camera calibration

• US 9,563,951 B2
• Filed: 05/20/2014
• Issued: 02/07/2017
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Okouneva, Galina
• Primary Examiner(s): Vo, Tung
• Assistant Examiner(s): Jiang, Zaihan
• Application Number:
  US14/282,028
• Publication Number:
  US 20140347486A1
• Time in Patent Office: 994 Days
• Field of Search: 348/148
• US Class Current: 1/1
• CPC Class Codes: B60R 1/22 for viewing an area outside the vehicle, e.g. the exterior of the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A camera calibration system of a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. The camera is operable to capture image data. An image processor operable to process image data captured by the camera. The camera calibration system is operable to generate camera calibration parameters utilizing a bundle adjustment algorithm. Responsive to image processing of captured image data during movement of the vehicle along an arbitrary path, and responsive to the bundle adjustment algorithm, the camera calibration system is operable to calibrate the camera. The bundle adjustment algorithm may iteratively refine calibration parameters starting from a known initial estimation.

Claims

1. A camera calibration system of a vehicle, said camera calibration system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera comprises a two dimensional array having a plurality of photosensing elements;

wherein said camera is operable to capture image data;

an image processor operable to process image data captured by said camera;

wherein said camera calibration system is operable to generate camera calibration parameters utilizing a bundle adjustment algorithm;

wherein said bundle adjustment algorithm uses a kinematic model of vehicle motion derived from image processing by said image processor of multiple frames of captured image data;

wherein said bundle adjustment algorithm generates calibration parameters utilizing the equation;

X_image=P*X_w,

where X_image is a selection of image points in captured image data, P is a projective matrix representing motion between frames of captured image data, and X_w is a vector of world coordinates of image points in captured image data;

wherein, responsive to image processing by said image processor of multiple frames of captured image data, said camera calibration system extracts and matches features determined in multiple frames of captured image data;

wherein, responsive to image processing by said image processor of captured image data during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters utilizing said bundle adjustment algorithm, said camera calibration system is operable to calibrate said camera; and

wherein said camera calibration system calibrates said camera as the vehicle navigates the arbitrary path, and wherein calibration of said camera includes correction for misalignment in yaw, pitch and roll angles of said camera.

2. The camera calibration system of claim 1, wherein said bundle adjustment algorithm iteratively refines calibration parameters starting from a known initial estimation.

3. The camera calibration system of claim 1, wherein said bundle adjustment algorithm solves for calibration parameters and actual coordinates of an object present in the field of view of said camera.

4. The camera calibration system of claim 1, wherein said image processor is operable to calibrate said camera without image processing of image data representative of captured images of a target.

5. The camera calibration system of claim 1, comprising a plurality of cameras disposed at the vehicle and having respective fields of view exterior of the vehicle, wherein said image processor is operable to process image data captured by said cameras, and wherein said image processor is operable to calibrate said cameras without use of targets in the field of view of said cameras and without image processing of image data representative of captured images of targets.

6. The camera calibration system of claim 1, wherein the generated camera calibration parameters comprise at least one of (i) at least one of translational components X, Y, Z and (ii) at least one of rotational components Yaw, Pitch and Roll.

7. The camera calibration system of claim 1, wherein the generated camera calibration parameters comprise (i) translational components X, Y, Z and (ii) rotational components Yaw, Pitch and Roll angles.

8. The camera calibration system of claim 1, wherein said camera calibration system comprises a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle.

9. The camera calibration system of claim 8, wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the vehicle.

10. The camera calibration system of claim 9, wherein one of said cameras comprises a rear camera disposed at a rear portion of the vehicle, and another of said cameras comprises a side camera disposed at a driver-side exterior rearview mirror assembly of the vehicle and another of said cameras comprises a passenger side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle.

11. A camera calibration system of a vehicle, said camera calibration system comprising:

a plurality of cameras disposed at a vehicle, each having a respective field of view exterior of the vehicle;

wherein said plurality of cameras comprises (i) a rear camera disposed at a rear portion of the vehicle and having a rearward field of view, (ii) a driver-side camera disposed at a driver-side exterior rearview mirror assembly of the vehicle and at least having a generally sideward and rearward field of view and (iii) a passenger-side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle and at least having a generally sideward and rearward field of view;

wherein said cameras are operable to capture image data;

an image processor operable to process image data captured by said cameras;

wherein said camera calibration system is operable to generate camera calibration parameters for each of said cameras utilizing a bundle adjustment algorithm;

wherein said bundle adjustment algorithm uses a kinematic model of vehicle motion derived from image processing by said image processor of multiple frames of image data captured by each of said cameras;

wherein said bundle adjustment algorithm generates calibration parameters utilizing the equation;

X_image=P*X_w,

where X_image is a selection of image points in captured image data, P is a projective matrix representing motion between frames of captured image data, and X_w is a vector of world coordinates of image points in captured image data;

wherein, responsive to image processing by said image processor of multiple frames of image data captured by each of said cameras, said camera calibration system extracts and matches features determined in multiple frames of image data captured by each of said cameras;

wherein, responsive to image processing by said image processor of captured image data during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters utilizing said bundle adjustment algorithm, said camera calibration system is operable to calibrate each of said cameras; and

wherein said camera calibration system calibrates each of said cameras as the vehicle navigates the arbitrary path, and wherein calibration of each of said cameras includes correction for misalignment in yaw, pitch and roll angles of each of said cameras.

12. The camera calibration system of claim 11, wherein said bundle adjustment algorithm iteratively refines calibration parameters starting from a known initial estimation.

13. The camera calibration system of claim 11, wherein said bundle adjustment algorithm solves for calibration parameters and actual coordinates of an object present in the field of view of said camera.

14. The camera calibration system of claim 11, wherein the generated camera calibration parameters comprise at least one of (i) at least one of translational components X, Y, Z and (ii) at least one of rotational components Yaw, Pitch and Roll.

15. The camera calibration system of claim 11, wherein the generated camera calibration parameters comprise (i) translational components X, Y, Z and (ii) rotational components Yaw, Pitch and Roll.

16. The camera calibration system of claim 11, wherein each of said cameras comprises a two dimensional array of a plurality of photosensing elements.

17. The camera calibration system of claim 11, wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the vehicle.

18. A camera calibration system of a vehicle, said camera calibration system comprising:

a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle;

wherein each of said cameras comprises a two dimensional array of a plurality of photosensing elements;

wherein said plurality of cameras comprises a rear camera disposed at a rear portion of the vehicle, a driver side camera disposed at a driver-side exterior rearview mirror assembly of the vehicle and a passenger side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle;

wherein each of said cameras is operable to capture image data;

an image processor operable to process image data captured by said cameras;

wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the vehicle;

wherein said camera calibration system is operable to generate camera calibration parameters utilizing a bundle adjustment algorithm;

wherein said bundle adjustment algorithm uses a kinematic model of vehicle motion derived from image processing by said image processor of multiple frames of image data captured by each of said cameras;

wherein said bundle adjustment algorithm generates calibration parameters utilizing the equation;

X_image=P*X_w,

where X_image is a selection of image points in captured image data, P is a protective matrix representing motion between frames of captured image data, and X_w is a vector of world coordinates of image points in captured image data;

wherein, responsive to image processing by said image processor of multiple frames of image data captured by each of said cameras, said camera calibration system extracts and matches features determined in multiple frames of image data captured by each of said cameras;

wherein the generated camera calibration parameters comprise (i) at least one of translational components X, Y, Z and (ii) at least one of rotational components Yaw, Pitch and Roll;

wherein, responsive to image processing by said image processor of captured image data during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters utilizing said bundle adjustment algorithm, said camera calibration system is operable to calibrate at least one of said cameras; and

wherein said camera calibration system calibrates each of said cameras as the vehicle navigates the arbitrary path, and wherein calibration of each of said cameras includes correction for misalignment in yaw, pitch and roll angles of each of said cameras.

19. The camera calibration system of claim 18, wherein said bundle adjustment algorithm iteratively refines calibration parameters starting from a known initial estimation.

20. The camera calibration system of claim 19, wherein said bundle adjustment algorithm solves for calibration parameters and actual coordinates of an object present in the field of view of said at least one of said cameras.