TARGETLESS VEHICULAR CAMERA CALIBRATION METHOD

• US 20180278925A1
• Filed: 05/21/2018
• Published: 09/27/2018
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Okouneva, Galina
• Granted Patent: US 10,567,748 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/22 for viewing an area outside the vehicle, e.g. the exterior of the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method of calibrating a vehicular camera includes disposing a camera at a vehicle, moving the vehicle along an arbitrary path, capturing image data with the camera, and processing, via a processor, image data captured by the camera. Responsive to tracking a determined feature across multiple frames of captured image data during movement of the vehicle along the arbitrary path, and responsive to received vehicle data representative of the movement of the vehicle along the arbitrary path, misalignment of the camera is determined. The determined camera misalignment is corrected (i) without use of a target in the field of view of the camera as the vehicle moves along the arbitrary path and (ii) without processing of image data captured by the camera that is representative of a target in the field of view of the camera as the vehicle moves along the arbitrary path.

Claims

1. A method of calibrating a vehicular camera, said method comprising:

disposing a camera at a vehicle so as to have a field of view exterior of the vehicle, wherein the camera comprises a two-dimensional photosensor array having a plurality of photosensing elements;

moving the vehicle along an arbitrary path;

capturing image data with the camera as the vehicle moves along the arbitrary path;

processing, via a processor, image data captured by the camera;

receiving at the processor vehicle data representative of vehicle kinematic motion as the vehicle moves along the arbitrary path;

responsive to processing by the processor of multiple frames of image data captured by the camera as the vehicle moves along the arbitrary path, determining at least one feature present in the multiple frames of image data captured by the camera and tracking the determined at least one feature across the multiple frames of image data captured by the camera;

responsive to tracking the determined at least one feature across the multiple frames of image data captured by the camera during movement of the vehicle along the arbitrary path, and responsive to the received vehicle data representative of the movement of the vehicle along the arbitrary path, determining misalignment of the camera in one or more of (i) its yaw angle, (ii) its pitch angle and (iii) its roll angle;

correcting the determined misalignment of the camera;

wherein correcting the determined misalignment of the camera starts from at least one of (i) an initial estimation of camera calibration and (ii) an intrinsic parameter of the camera; and

wherein correcting the determined misalignment of the camera is achieved (i) without use of a target in the field of view of the camera as the vehicle moves along the arbitrary path and (ii) without processing of image data captured by the camera that is representative of a target in the field of view of the camera as the vehicle moves along the arbitrary path.

2. The method of claim 1, wherein the camera is one of a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle, and wherein image data captured by at least some cameras of the plurality of cameras is for use by a surround view system of the vehicle.

3. The method of claim 2, wherein correcting the determined misalignment of the camera comprises correcting the determined misalignment of each camera of the plurality of cameras (i) without use of a target in the respective fields of view of the cameras as the vehicle moves along the arbitrary path and (ii) without processing of image data captured by the cameras that is representative of a target in the respective fields of views of the cameras as the vehicle moves along the arbitrary path.

4. The method of claim 2, wherein the camera is disposed at a driver side of the vehicle, and wherein another camera of the plurality of cameras comprises a camera disposed at a rear portion of the vehicle, and wherein another camera of the plurality of camera cameras comprises a passenger side camera disposed at a passenger side of the vehicle.

5. The method of claim 4, wherein the camera is disposed at a driver-side exterior rearview mirror assembly of the vehicle, and wherein the passenger side camera is disposed at a passenger-side exterior rearview mirror assembly of the vehicle.

6. The method of claim 4, wherein each camera of the plurality of cameras comprises a two-dimensional CMOS photosensor array comprising a plurality of photosensing elements.

7. The method of claim 2, wherein the surround view system of the vehicle provides a bird'"'"'s eye view of the environment at or around or at least partially surrounding the vehicle.

8. The method of claim 1, comprising generating camera calibration parameters at least in part by refining calibration parameters starting from an initial estimation of camera calibration.

9. The method of claim 1, comprising generating camera calibration parameters at least in part by refining calibration parameters starting from an intrinsic parameter of the camera.

10. The method of claim 1, wherein processing by the processor of image data captured by the camera provides at least one of (i) feature detection, (ii) feature extraction, (iii) feature matching and (iv) feature filtering.

11. The method of claim 1, wherein processing by the processor of image data captured by the camera eliminates feature pairs that do not satisfy established rules for valid feature pairs in valid frame pairs.

12. The method of claim 1, wherein correcting the determined misalignment of the camera utilizes, at least in part, structure-from-motion during processing of image data captured by the camera as the vehicle moves along the arbitrary path.

13. The method of claim 1, wherein correcting the determined misalignment of the camera utilizes, at least in part, tracking of feature descriptors during processing of image data captured by the camera as the vehicle moves along the arbitrary path.

14. The method of claim 1, wherein correcting the determined misalignment of the camera is achieved without use of reference points on the vehicle.

15. The method of claim 1, wherein the vehicle data comprises at least one of (i) data related to steering of the vehicle and (ii) data related to speed of the vehicle.

16. The method of claim 15, wherein the vehicle data is carried by a communication bus system of the vehicle.

17. The method of claim 16, wherein the communication bus system comprises a CAN system.

18. The method of claim 1, wherein moving the vehicle along the arbitrary path comprises moving the vehicle along an assembly line at a vehicle manufacturer.

19. The method of claim 1, wherein moving the vehicle along the arbitrary path comprises moving the vehicle along a road being traveled by the vehicle.

20. The method of claim 19, wherein the determined at least one feature comprises a sign at the road being traveled by the vehicle.

21. The method of claim 19, wherein the determined at least one feature comprises a mailbox at the road being traveled by the vehicle.

22. The method of claim 19, wherein the determined at least one feature comprises a tree at the road being traveled by the vehicle.

23. A method of calibrating a vehicular camera, said method comprising:

disposing a camera at a vehicle so as to have a field of view exterior of the vehicle, wherein the camera comprises a two-dimensional photosensor array having a plurality of photosensing elements;

moving the vehicle along an arbitrary path;

capturing image data with the camera as the vehicle moves along the arbitrary path;

processing, via a processor, image data captured by the camera;

receiving at the processor vehicle data representative of vehicle kinematic motion as the vehicle moves along the arbitrary path, wherein the vehicle data comprises vehicle steering data and vehicle speed data;

responsive to processing by the processor of multiple frames of image data captured by the camera as the vehicle moves along the arbitrary path, determining at least one feature present in the multiple frames of image data captured by the camera and tracking the determined at least one feature across the multiple frames of image data captured by the camera;

responsive to tracking the determined at least one feature across the multiple frames of image data captured by the camera during movement of the vehicle along the arbitrary path, and responsive to the received vehicle data representative of the movement of the vehicle along the arbitrary path, determining misalignment of the camera in one or more of (i) its yaw angle, (ii) its pitch angle and (iii) its roll angle;

correcting the determined misalignment of the camera;

wherein correcting the determined misalignment of the camera starts from an intrinsic parameter of the camera; and

wherein correcting the determined misalignment of the camera is achieved (i) without use of a target in the field of view of the camera as the vehicle moves along the arbitrary path and (ii) without processing of image data captured by the camera that is representative of a target in the field of view of the camera as the vehicle moves along the arbitrary path.

24. The method of claim 23, wherein the camera is one of a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle, and wherein image data captured by at least some cameras of the plurality of cameras is for use by a surround view system of the vehicle.

25. The method of claim 23, wherein moving the vehicle along the arbitrary path comprises moving the vehicle along an assembly line at a vehicle manufacturer.

26. The method of claim 23, wherein moving the vehicle along the arbitrary path comprises moving the vehicle along a road being traveled by the vehicle.

27. The method of claim 23, wherein the vehicle data is carried by a communication bus system of the vehicle.

28. The method of claim 27, wherein the communication bus system comprises a CAN system.

29. A method of calibrating a vehicular camera, said method comprising:

providing a plurality of cameras, the plurality of cameras comprising a driver-side camera, a passenger-side camera and a rear camera;

disposing the driver-side camera at a driver side of the vehicle so as to have a field of view exterior and sideward of the vehicle;

disposing the passenger-side camera at a passenger side of the vehicle so as to have a field of view exterior and sideward of the vehicle;

disposing the rear camera at a rear side of the vehicle so as to have a field of view exterior and rearward of the vehicle;

wherein each of the cameras comprises a two-dimensional photosensor array having a plurality of photosensing elements;

moving the vehicle along an arbitrary path, wherein moving the vehicle along the arbitrary path comprises moving the vehicle along an assembly line at a vehicle manufacturer;

capturing image data with the cameras as the vehicle moves along the arbitrary path;

processing, via a processor, image data captured by the cameras;

receiving at the processor vehicle data representative of vehicle kinematic motion as the vehicle moves along the arbitrary path;

responsive to processing by the processor of multiple frames of image data captured by the cameras as the vehicle moves along the arbitrary path, determining at least one feature present in the multiple frames of image data captured by a respective camera of the cameras and tracking the determined at least one feature across the multiple frames of image data captured by the respective camera of the cameras;

responsive to tracking the determined at least one feature across the multiple frames of image data captured by the respective camera of the cameras during movement of the vehicle along the arbitrary path, and responsive to the received vehicle data representative of the movement of the vehicle along the arbitrary path, determining misalignment of the respective camera of the cameras in one or more of (i) its yaw angle, (ii) its pitch angle and (iii) its roll angle;

correcting the determined misalignment of the respective camera of the cameras; and

wherein correcting the determined misalignment of the respective camera of the cameras is achieved (i) without use of a target in the respective field of view of the respective camera as the vehicle moves along the arbitrary path and (ii) without processing of image data representative of a target in the respective field of view of respective camera as the vehicle moves along the arbitrary path.

30. The method of claim 29, wherein correcting the determined misalignment of the respective camera of the cameras starts from an intrinsic parameter of the respective camera.

31. The method of claim 29, wherein correcting the determined misalignment of the respective camera of the cameras starts from an initial estimation of camera calibration of the respective camera.

32. The method of claim 29, wherein the driver-side camera is disposed at a driver-side exterior rearview mirror assembly of the vehicle, and wherein the passenger-side camera is disposed at a passenger-side exterior rearview mirror assembly of the vehicle.

33. The method of claim 29, wherein each camera of the plurality of cameras comprises a two-dimensional CMOS photosensor array comprising a plurality of photosensing elements.

34. The method of claim 33, wherein the vehicle data comprises vehicle speed data.

35. The method of claim 34, wherein the vehicle data comprises vehicle steering data.

36. The method of claim 29, wherein image data captured by at least some cameras of the plurality of cameras is for use by a surround view system of the vehicle.

37. The method of claim 36, wherein the surround view system of the vehicle provides a bird'"'"'s eye view of the environment at or around or at least partially surrounding the vehicle.