Vehicle camera calibration system

• US 10,504,241 B2
• Filed: 12/18/2017
• Issued: 12/10/2019
• Est. Priority Date: 12/19/2016
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Singh, Jagmal
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US15/844,995
• Publication Number:
  US 20180174327A1
• Time in Patent Office: 722 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/24 in front of the vehicle
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A camera calibration system for cameras of a vehicle includes a plurality of cameras disposed at a vehicle and having respective fields of view exterior of the vehicle, with the fields of view of two of the cameras overlapping. While the vehicle is moving, an image processor of a control processes image data captured by the two cameras to determine and track matching features in an overlapping region of the fields of view of the two cameras. The control, responsive to image processing of image data captured by the two cameras, determines motion of matching features and minimizes a cost function of the error in the matching of the features. The camera calibration system uses the determined minimized cost function to refine calibration results of the two cameras using a feedback loop.

Claims

1. A camera calibration system for vehicular cameras, said camera calibration system comprising:

a plurality of cameras disposed at a vehicle, with each camera of the plurality of cameras having a respective field of view exterior of the vehicle, wherein the fields of view of two of said cameras overlap;

a control having an image processor operable to process image data captured by said cameras;

wherein said image processor processes image data captured by said two cameras having overlapping fields of view to determine at least one matching feature of an object present at an overlapping region of the fields of view of said two cameras;

wherein, while the vehicle is moving, said image processor processes multiple frames of image data captured by said two cameras having overlapping fields of view to track the at least one matching feature of the object present in the overlapping region of the fields of view of said two cameras;

wherein said control, responsive to image processing by said image processor of multiple frames of image data captured by said two cameras having overlapping fields of view, determines motion of at least one matching feature of the object and minimizes a cost function of an error in the matching of the at least one feature of the object; and

wherein said camera calibration system uses the determined minimized cost function to refine calibration of said two cameras using a feedback loop.

2. The camera calibration system of claim 1, comprising a display screen disposed in the vehicle for displaying video images derived from image data captured by said cameras for viewing by a driver of the vehicle during a driving maneuver of the vehicle.

3. The camera calibration system of claim 2, wherein images derived from image data captured by adjacent pairs of said cameras are stitched together for display at said display screen for viewing by the driver of the vehicle during a driving maneuver.

4. The camera calibration system of claim 3, wherein the determined minimized cost function and the feedback loop are used to improve stitching quality of the stitched images for display at said display screen for viewing by the driver of the vehicle during a driving maneuver.

5. The camera calibration system of claim 1, wherein said plurality of cameras comprises at least four cameras for a surround view vision system of the vehicle.

6. The camera calibration system of claim 5, wherein said two cameras comprises a rearward viewing camera disposed at a rear of the vehicle and a sideward and rearward viewing camera disposed at a side of the vehicle.

7. The camera calibration system of claim 1, wherein said plurality of cameras comprises multiple cameras disposed at the rear of the vehicle and multiple cameras disposed at each side of the vehicle.

8. The camera calibration system of claim 1, wherein said control uses structure from motion of the ground structure moving through the fields of view of said two cameras while the vehicle is moving to generate a three dimensional point cloud.

9. The camera calibration system of claim 8, wherein the generated three dimensional point cloud provides features that are present at the overlapping region of the fields of view of said two cameras.

10. A camera calibration system for vehicular cameras, said camera calibration system comprising:

a plurality of cameras disposed at a vehicle;

wherein said plurality of cameras comprises a rear camera disposed at a rear portion of the vehicle and having a field of view rearward of the vehicle, a driver-side camera disposed at a driver side of the vehicle and having a field of view sideward and rearward at the driver side of the vehicle, and a passenger-side camera disposed at a passenger side of the vehicle and having a field of view sideward and rearward at the passenger side of the vehicle;

wherein the fields of view of said rear camera and said driver-side camera overlap at a first overlapping region;

wherein the fields of view of said rear camera and said passenger-side camera overlap at a second overlapping region;

a control having an image processor operable to process image data captured by said rear camera, said driver-side camera and said passenger-side camera;

wherein said image processor processes image data captured by said rear camera and said driver-side camera to determine at least one matching feature of an object present at the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein said image processor processes image data captured by said rear camera and said passenger-side camera to determine at least one matching feature of an object present at the second overlapping region of the fields of view of said rear camera and said passenger-side camera;

wherein, while the vehicle is moving, said image processor processes multiple frames of image data captured by said rear camera and said driver-side camera to track the at least one matching feature of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein, while the vehicle is moving, said image processor processes multiple frames of image data captured by said rear camera and said passenger-side camera to track the at least one matching feature of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera;

wherein said control, responsive to image processing by said image processor of multiple frames of image data captured by said rear camera and said driver-side camera, determines motion of at least one matching feature of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera and minimizes a first cost function of an error in the matching of the at least one feature of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein said control, responsive to image processing by said image processor of multiple frames of image data captured by said rear camera and said passenger-side camera, determines motion of at least one matching feature of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera and minimizes a second cost function of an error in the matching of the at least one feature of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera; and

wherein said camera calibration system uses the determined minimized first and second cost functions to refine calibration of said rear camera, said driver-side camera and said passenger-side camera using at least one feedback loop.

11. The camera calibration system of claim 10, comprising a display screen disposed in the vehicle for displaying video images derived from image data captured by said cameras for viewing by a driver of the vehicle during a driving maneuver of the vehicle.

12. The camera calibration system of claim 11, wherein images derived from image data captured by said rear camera and said driver-side camera are stitched together and images derived from image data captured by said rear camera and said passenger-side camera are stitched together for display at said display screen for viewing by the driver of the vehicle during the driving maneuver of the vehicle.

13. The camera calibration system of claim 12, wherein the determined minimized first and second cost function and the at least one feedback loop are used to improve stitching quality of the stitched images for display at said display screen for viewing by the driver of the vehicle during the driving maneuver of the vehicle.

14. The camera calibration system of claim 10, wherein said plurality of cameras comprises a front camera disposed at a front portion of the vehicle and having a forward field of view, and wherein images derived from image data captured by said rear camera, said driver-side camera, said front camera and said passenger-side camera are stitched together and displayed for a surround view vision system of the vehicle.

15. The camera calibration system of claim 10, wherein said plurality of cameras comprises multiple rear cameras disposed at the rear portion of the vehicle and having rearward fields of view, multiple driver-side cameras disposed at the driver side of the vehicle, and multiple passenger-side cameras disposed at the passenger side of the vehicle.

16. A camera calibration system for vehicular cameras, said camera calibration system comprising:

a plurality of cameras disposed at a vehicle;

wherein said plurality of cameras comprises a rear camera disposed at a rear portion of the vehicle and having a field of view rearward of the vehicle, a driver-side camera disposed at a driver side of the vehicle and having a field of view sideward and rearward at the driver side of the vehicle, and a passenger-side camera disposed at a passenger side of the vehicle and having a field of view sideward and rearward at the passenger side of the vehicle;

wherein the fields of view of said rear camera and said driver-side camera overlap at a first overlapping region;

wherein the fields of view of said rear camera and said passenger-side camera overlap at a second overlapping region;

a control having an image processor operable to process image data captured by said rear camera, said driver-side camera and said passenger-side camera;

wherein said image processor processes image data captured by said rear camera and said driver-side camera to determine at least one matching feature of an object present at the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein said image processor processes image data captured by said rear camera and said passenger-side camera to determine at least one matching feature of an object present at the second overlapping region of the fields of view of said rear camera and said passenger-side camera;

wherein, while the vehicle is moving, said image processor processes multiple frames of image data captured by said rear camera and said driver-side camera to track the at least one matching feature of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein, while the vehicle is moving, said image processor processes multiple frames of image data captured by said rear camera and said passenger-side camera to track the at least one matching feature of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera;

wherein said control uses structure from motion of the ground structure moving through the fields of view of said rear camera and said driver-side camera while the vehicle is moving to generate a first three dimensional point cloud, and wherein said control uses structure from motion of the ground structure moving through the fields of view of said rear camera and said passenger-side camera while the vehicle is moving to generate a second three dimensional point cloud;

wherein the generated three dimensional point clouds provide features that are present at respective overlapping regions of the fields of view of adjacent ones of said cameras;

wherein said control, responsive to image processing by said image processor of multiple frames of image data captured by said rear camera and said driver-side camera, determines motion of matching features of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera and minimizes a first cost function of an error in the matching of the features of the object present in the first overlapping region of the fields of view of said rear camera and said driver-side camera;

wherein said control, responsive to image processing by said image processor of multiple frames of image data captured by said rear camera and said passenger-side camera, determines motion of matching features of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera and minimizes a second cost function of an error in the matching of the features of the object present in the second overlapping region of the fields of view of said rear camera and said passenger-side camera; and

wherein said camera calibration system uses the determined minimized first and second cost functions to refine calibration of said rear camera, said driver-side camera and said passenger-side camera using at least one feedback loop.

17. The camera calibration system of claim 16, comprising a display screen disposed in the vehicle for displaying video images derived from image data captured by said cameras for viewing by a driver of the vehicle during a driving maneuver of the vehicle.

18. The camera calibration system of claim 17, wherein images derived from image data captured by said rear camera and said driver-side camera are stitched together and images derived from image data captured by said rear camera and said passenger-side camera are stitched together for display at said display screen for viewing by the driver of the vehicle during the driving maneuver of the vehicle.

19. The camera calibration system of claim 18, wherein the determined minimized first and second cost function and the at least one feedback loop are used to improve stitching quality of the stitched images for display at said display screen for viewing by the driver of the vehicle during the driving maneuver of the vehicle.

20. The camera calibration system of claim 16, wherein said plurality of cameras comprises multiple rear cameras disposed at the rear portion of the vehicle and having rearward fields of view, multiple driver-side cameras disposed at the driver side of the vehicle, and multiple passenger-side cameras disposed at the passenger side of the vehicle.