Multi-camera image stitching calibration system

• US 9,723,272 B2
• Filed: 10/04/2013
• Issued: 08/01/2017
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Byrne, Steven V.
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Matt, Marnie
• Application Number:
  US14/046,174
• Publication Number:
  US 20140098229A1
• Time in Patent Office: 1,397 Days
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A vision system of a vehicle includes a plurality of cameras with exterior overlapping fields of view. The vision system is operable to calibrate at least one of the cameras and includes a plurality of targets disposed at locations near the vehicle and within the fields of views of the cameras. A display device is operable to display images captured by the cameras of the vision system. A plurality of user inputs may be provided and a user may selectively actuate the user inputs to manipulate images captured by the cameras to align portions of a target in overlapping regions of the captured images of adjacent cameras to calibrate the cameras. The vision system may calibrate cameras automatically, such as responsive to the targets moving into the fields of views of the cameras, whereby the vision system may calibrate the cameras as the vehicle is moved along an assembly line.

Claims

1. A vision system of a vehicle, said vision system comprising:

a plurality of cameras having fields of view exterior a vehicle equipped with the vision system, wherein said plurality of cameras comprises a forward camera having a forward field of view, a rearward camera having a rearward facing view, a left side camera having a left sideward field of view at the left side of the equipped vehicle and a right side camera having a right sideward field of view at the right side of the equipped vehicle;

wherein a portion of said forward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view and wherein a portion of said rearward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view;

a display device operable to display images captured by said cameras of said vision system;

a plurality of user inputs actuatable by a user to manipulate the displayed images;

wherein, during manual camera calibration, a plurality of targets are disposed at ground locations near the vehicle and within the overlapping portions of the fields of views of said cameras of said vision system;

wherein, when the displayed image of one of said cameras is misaligned with a displayed image of another of said cameras, a user actuates said user inputs to manipulate the displayed image of said one of said cameras to align portions of at least one of the targets in an overlapping region of the displayed images of said one of said cameras and said another of said cameras to correct for misalignment of said one of said cameras;

wherein, during manual camera calibration, the displayed image captured by a first calibrated camera of said plurality of cameras is manipulated so that lines are displayed as straight and rectangles are displayed in their right shapes and scales; and

wherein, after the displayed image captured by the first calibrated camera is manipulated to reach a satisfactory state, the first calibrated camera is used as a calibration reference for a second calibrated camera, and wherein the displayed image captured by the second calibrated camera is manipulated so that lines are displayed as straight and rectangles are displayed in their right shapes and scales and so that at the border of the displayed images captured by the first and second calibrated cameras, the patterns transition smoothly and continuously across the border or overlapping areas of the displayed images.

2. The vision system of claim 1, wherein said user inputs are part of or associated with said display device.

3. The vision system of claim 1, wherein said user inputs are part of another system of the vehicle.

4. The vision system of claim 1, wherein said display device comprises a remote display device that is remote from the vehicle.

5. The vision system of claim 1, wherein said user inputs are actuated to manipulate the displayed images by stretching or compressing or rotating or shifting the displayed images to align portions of the captured image with at least one of said targets.

6. The vision system of claim 1, wherein said user inputs are actuated to manipulate the displayed images by at least one of (i) stretching the displayed images, (ii) compressing the displayed images, (iii) rotating the displayed images and (iv) shifting the displayed images.

7. A vision system of a vehicle, said vision system comprising:

a plurality of cameras having fields of view exterior a vehicle equipped with the vision system, wherein said plurality of cameras comprises a forward camera having a forward field of view, a rearward camera having a rearward facing view, a left side camera having a left sideward field of view at the left side of the equipped vehicle and a right side camera having a right sideward field of view at the right side of the equipped vehicle;

wherein each of said cameras comprises a lens and an imager;

wherein a portion of said forward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view and wherein a portion of said rearward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view;

a display device operable to display images captured by said cameras of said vision system;

wherein said vision system is operable to calibrate said cameras to correct for mis-alignment of at least one of said cameras;

wherein, during camera calibration, a plurality of targets are disposed at ground locations near the vehicle and within the overlapping portions of the fields of views of said cameras;

wherein the plurality of targets comprises at least two longitudinal lines extending longitudinally along the direction that the vehicle is moving at either side of the vehicle, with a plurality of tick marks extending laterally from the longitudinal lines;

wherein, during camera calibration, said vision system is operable to (i) utilize a math model to at least in part correct for lens distortion based on lens nominal design data, (ii) utilize a lens-to-sensor misalignment assessment and correction by determining a relationship between the degree of curvature of horizontal lines and vertical misalignment of said lens to a center of said imager for each of said cameras, (iii) utilize a perspective correction to at least in part correct for remaining rotation and translation errors and (iv) utilize a fine tuning of smaller mis-matches to achieve smooth stitching for the displayed image;

wherein camera calibration is performed while the vehicle is moving; and

wherein camera calibration is triggered by a target moving into a predetermined position in the camera views while the vehicle is moving.

8. The vision system of claim 7, wherein the perspective correction utilizes a math model of homographic transformation represented by a 3×

3 matrix.

9. The vision system of claim 7, wherein a degree of curvature of horizontal lines can be measured by detecting edges and points of the target patterns.

10. The vision system of claim 7, wherein said fine tuning of smaller mis-matches is achieved by stretching or moving pixel data in overlapping regions of adjacent camera images.

11. A vision system of a vehicle, said vision system comprising:

a plurality of cameras having fields of view exterior a vehicle equipped with the vision system, wherein said plurality of cameras comprises a forward camera having a forward field of view, a rearward camera having a rearward facing view, a left side camera having a left sideward field of view at the left side of the equipped vehicle and a right side camera having a right sideward field of view at the right side of the equipped vehicle;

wherein a portion of said forward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view and wherein a portion of said rearward field of view overlaps respective portions of said left sideward field of view and said right sideward field of view;

wherein said vision system is operable to calibrate said cameras to correct for mis-alignment of at least one of said cameras;

wherein said vision system is operable to calibrate said cameras while the vehicle is moving;

wherein a plurality of targets are disposed at ground locations near the path of the vehicle so that, when the vehicle is at or near the targets, the targets are at least partially within the overlapping portions of the fields of views of said cameras;

wherein the plurality of targets comprises at least two longitudinal lines extending longitudinally along the direction that the vehicle is moving at either side of the vehicle, with a plurality of tick marks extending laterally from the longitudinal lines; and

wherein camera calibration is triggered by at least one triggering target moving into a predetermined position in the field of view of at least one of said cameras as the vehicle moves relative to the targets.

12. The vision system of claim 11, wherein, responsive to triggering of said camera calibration, said cameras will capture and store image for a stitching calibration computation.

13. The vision system of claim 11, comprising a display device operable to display images captured by said cameras.

14. The vision system of claim 11, wherein each of said cameras comprises a lens and an imager.

15. The vision system of claim 14, wherein, during camera calibration, said vision system is operable to utilize a math model to at least in part correct for lens distortion based on lens nominal design data.

16. The vision system of claim 14, wherein, during camera calibration, said vision system is operable to utilize a lens-to-sensor misalignment assessment and correction by determining a relationship between the degree of curvature of horizontal lines and vertical misalignment of said lens to a center of said imager for each of said cameras.

17. The vision system of claim 11, wherein, during camera calibration, said vision system is operable to utilize a perspective correction to at least in part correct for rotation and translation errors.

18. The vision system of claim 11, wherein, during camera calibration, said vision system is operable to, following an initial calibration process, utilize a fine tuning of smaller mis-matches to achieve smooth stitching for the displayed image.