VEHICLE VISION SYSTEM WITH TARGETLESS CAMERA CALIBRATION

• US 20170134722A1
• Filed: 01/26/2017
• Published: 05/11/2017
• Est. Priority Date: 05/21/2013
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Okouneva, Galina
• Granted Patent: US 9,979,957 B2
• Time in Patent Office: Days
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/22 for viewing an area outside the vehicle, e.g. the exterior of the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/698 : for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A camera calibration system for a vehicle includes a camera disposed at the vehicle and an image processor operable to process image data captured by the camera. The camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from at least one of (i) an initial estimation and (ii) an intrinsic parameter of the camera. The camera calibration system uses a kinematic model of vehicle motion derived at least in part from image processing of multiple frames of captured image data. Responsive to such image processing, the camera calibration system extracts and matches features determined in multiple frames of captured image data. Responsive to image processing during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters, the camera calibration system calibrates the camera as the vehicle navigates the arbitrary path.

Claims

1. A camera calibration system for a vehicle, said camera calibration system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera comprises a two-dimensional photosensor array having a plurality of photosensing elements;

wherein said camera is operable to capture image data;

an image processor operable to process image data captured by said camera;

wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from at least one of (i) an initial estimation and (ii) an intrinsic parameter of said camera;

wherein said camera calibration system uses a kinematic model of vehicle motion derived at least in part from image processing by said image processor of multiple frames of captured image data;

wherein, responsive to image processing by said image processor of multiple frames of captured image data, said camera calibration system extracts and matches features determined in multiple frames of captured image data;

wherein, responsive to image processing of captured image data by said image processor during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters, said camera calibration system calibrates said camera as the vehicle navigates the arbitrary path; and

wherein calibration of said camera includes correction for misalignment in at least one of (i) yaw angle, (ii) pitch angle and (iii) roll angle.

2. The camera calibration system of claim 1, wherein said camera is one of a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle.

3. The camera calibration system of claim 2, wherein image data captured by at least some cameras of said plurality of cameras is used for a surround view system of the vehicle.

4. The camera calibration system of claim 3, wherein said camera calibration system is operable to calibrate said plurality of cameras without use of targets in the field of view of said plurality of cameras and without image processing of image data representative of captured images of targets.

5. The camera calibration system of claim 4, wherein said camera is disposed at a driver-side exterior rearview mirror assembly of the vehicle and wherein another camera of said plurality of cameras comprises a camera disposed at a rear portion of the vehicle, and wherein another camera of said plurality of camera cameras comprises a passenger side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle.

6. The camera calibration system of claim 5, wherein each camera of said plurality of cameras comprises a two-dimensional CMOS photosensor array comprising a plurality of photosensing elements.

7. The camera calibration system of claim 6, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an initial estimation.

8. The camera calibration system of claim 3, wherein said surround view system of the vehicle provides a bird'"'"'s eye view of the environment at or around or at least partially surrounding the vehicle.

9. The camera calibration system of claim 1, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an intrinsic parameter of said camera.

10. The camera calibration system of claim 1, wherein processing by said image processor of captured image data provides at least one of (i) feature detection, (ii) feature extraction, (iii) feature matching and (iv) feature filtering.

11. The camera calibration system of claim 1, wherein processing by said image processor of captured image data eliminates feature pairs that do not satisfy established rules for valid feature pairs in valid frame pairs.

12. The camera calibration system of claim 1, wherein said camera calibration system, at least in part, uses structure-from-motion during processing of captured image data.

13. The camera calibration system of claim 1, wherein said camera calibration system, at least in part, utilize extraction and matching of feature descriptors during processing of captured image data.

14. The camera calibration system of claim 1, wherein said camera calibration system calibrates said camera without use of reference points on the vehicle.

15. The camera calibration system of claim 1, wherein said camera is initially calibrated in the vehicle assembly plant in which the vehicle is assembled.

16. A camera calibration system for a vehicle, said camera calibration system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera is disposed at a driver-side exterior rearview mirror assembly of the vehicle;

wherein said camera comprises a two-dimensional CMOS photosensor array having a plurality of photosensing elements;

wherein said camera is operable to capture image data;

an image processor operable to process image data captured by said camera;

wherein said camera calibration system uses a kinematic model of vehicle motion derived at least in part from image processing by said image processor of multiple frames of captured image data;

wherein, responsive to image processing by said image processor of multiple frames of captured image data, said camera calibration system extracts and matches features determined in multiple frames of captured image data;

wherein, responsive to image processing of captured image data by said image processor during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters, said camera calibration system calibrates said camera as the vehicle navigates the arbitrary path;

wherein calibration of said camera includes correction for misalignment in at least one of (i) yaw angle, (ii) pitch angle and (iii) roll angle;

wherein said camera is one of a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle;

wherein image data captured by at least some cameras of said plurality of cameras is used for a surround view system of the vehicle;

wherein said camera calibration system is operable to calibrate said plurality of cameras without use of targets in the field of view of said plurality of cameras and without image processing of image data representative of captured images of targets; and

wherein one camera of said plurality of cameras comprises a camera disposed at a rear portion of the vehicle, and wherein another camera of said plurality of cameras comprises a passenger side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle.

17. The camera calibration system of claim 16, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an initial estimation.

18. The camera calibration system of claim 16, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an intrinsic parameter of said camera.

19. The camera calibration system of claim 16, wherein said surround view system of the vehicle provides a bird'"'"'s eye view of the environment at or around or at least partially surrounding the vehicle.

20. A camera calibration system for a vehicle, said camera calibration system comprising:

a camera disposed at a vehicle and having a field of view exterior of the vehicle;

wherein said camera is disposed at a driver-side exterior rearview mirror assembly of the vehicle;

wherein said camera comprises a two-dimensional CMOS photosensor array having a plurality of photosensing elements;

wherein said camera is operable to capture image data;

an image processor operable to process image data captured by said camera;

wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from at least one of (i) an initial estimation and (ii) an intrinsic parameter of said camera;

wherein said camera calibration system uses a kinematic model of vehicle motion derived at least in part from image processing by said image processor of multiple frames of captured image data;

wherein, responsive to image processing by said image processor of multiple frames of captured image data, said camera calibration system extracts and matches features determined in multiple frames of captured image data;

wherein, responsive to image processing of captured image data by said image processor during movement of the vehicle along an arbitrary path, and responsive to generation of camera calibration parameters, said camera calibration system calibrates said camera as the vehicle navigates the arbitrary path;

wherein calibration of said camera includes correction for misalignment in at least one of (i) yaw angle, (ii) pitch angle and (iii) roll angle;

wherein said camera is one of a plurality of cameras disposed at the vehicle, each having a respective field of view exterior of the vehicle;

wherein image data captured by at least some of said plurality of cameras is used for a surround view system of the vehicle;

wherein said surround view system of the vehicle provides a bird'"'"'s eye view of the environment at or around or at least partially surrounding the vehicle;

wherein said camera calibration system is operable to calibrate said plurality of cameras without use of targets in the field of view of said plurality of cameras and without image processing of image data representative of captured images of targets;

wherein one camera of said plurality of cameras comprises a camera disposed at a rear portion of the vehicle, and wherein another camera of said plurality of cameras comprises a passenger side camera disposed at a passenger-side exterior rearview mirror assembly of the vehicle; and

wherein processing by said image processor of captured image data provides at least one of (i) feature detection, (ii) feature extraction, (iii) feature matching and (iv) feature filtering.

21. The camera calibration system of claim 20, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an initial estimation.

22. The camera calibration system of claim 20, wherein said camera calibration system generates camera calibration parameters at least in part by iteratively refining calibration parameters starting from an intrinsic parameter of said camera.