Multi-camera image stitching calibration system

• US 10,284,818 B2
• Filed: 07/31/2017
• Issued: 05/07/2019
• Est. Priority Date: 10/05/2012
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Lu, Yuesheng, Byrne, Steven V.
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US15/663,979
• Publication Number:
  US 20170332052A1
• Time in Patent Office: 645 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method of calibrating cameras of a multi-camera vision system for a being-assembled vehicle includes providing a plurality of targets at a vehicle assembly line at a vehicle assembly plant and moving a being-assembled vehicle along the vehicle assembly line. When the being-assembled vehicle is moved along the vehicle assembly line so as to be at provided calibrating targets, at least some targets of the calibrating targets are encompassed by respective fields of view of the cameras of the being-assembled vehicle. The presence of at least one target is determined via processing of captured image data and as the being-assembled vehicle moves along the vehicle assembly line. Responsive to determination of the presence of at least one target, the cameras are calibrated and as the being-assembled vehicle moves along the vehicle assembly line.

Claims

1. A method of calibrating cameras of a multi-camera vision system for a vehicle moving along a vehicle assembly line, said method comprising:

providing a plurality of targets at a vehicle assembly line at a vehicle assembly plant;

equipping a vehicle that is being conveyed along the vehicle assembly line with a plurality of cameras that includes (i) a front camera disposed at a front portion of the vehicle and viewing at least forward of the vehicle, (ii) a driver-side camera disposed at a driver side of the vehicle and viewing at least sideward of the vehicle, (iii) a passenger-side camera disposed at a passenger side of the vehicle and viewing at least sideward of the vehicle, and (iv) a rear camera disposed at a rear of the vehicle and viewing at least rearward of the vehicle;

moving the vehicle along the vehicle assembly line;

wherein providing a plurality of targets at the vehicle assembly line comprises providing a plurality of calibrating targets;

wherein, when the vehicle is moved along the vehicle assembly line so as to be at the provided calibrating targets, at least some targets of the calibrating targets are encompassed by respective fields of view of the front camera, the driver-side camera, the passenger-side camera and the rear camera;

capturing image data with the cameras as the vehicle moves along the vehicle assembly line;

determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one target of the plurality of targets in the field of view of at least one of the cameras; and

responsive to determination of presence of at least one target in the field of view of at least one of the cameras, calibrating the cameras via processing captured image data and determining presence of the calibrating targets in the respective fields of view of the front camera, the driver-side camera, the passenger-side camera, and the rear camera as the vehicle moves along the vehicle assembly line.

2. The method of claim 1, wherein providing a plurality of calibrating targets comprises providing a plurality of calibrating targets at ground locations at the vehicle assembly line so as to be within overlapping portions of the fields of views of respective pairs of cameras when the vehicle is moved so as to be at the calibrating targets.

3. The method of claim 2, wherein the plurality of calibrating targets comprises at least two longitudinal lines extending longitudinally along the direction that the vehicle is moving at either side of the vehicle assembly line, with a plurality of tick marks extending laterally from the longitudinal lines.

4. The method of claim 1, wherein providing a plurality of targets at the vehicle assembly line comprises providing at least one triggering target at the vehicle assembly line, and wherein calibrating the cameras is responsive to determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one triggering target in the field of view of at least one of the cameras.

5. The method of claim 4, wherein, responsive to determining presence of at least one triggering target, capturing and storing image data for a stitching calibration computation.

6. The method of claim 1, wherein calibrating the cameras comprises, via processing of image data captured by the cameras, determining multi-axial orientation of the cameras.

7. The method of claim 6, wherein determining multi-axial orientation of the cameras comprises determining pitch, yaw and roll of each of the cameras.

8. The method of claim 1, wherein calibrating the cameras comprises utilizing a math model to at least in part correct for lens distortion of a lens of each of the cameras based on lens nominal design data.

9. The method of claim 1, wherein calibrating the cameras comprises utilizing a lens-to-sensor misalignment assessment and correction by determining a relationship between the degree of curvature of horizontal lines and vertical misalignment of a lens to a center of an imager for each of the cameras.

10. The method of claim 1, wherein calibrating the cameras comprises utilizing a perspective correction to at least in part correct for rotation and translation errors.

11. The method of claim 10, wherein utilizing a perspective correction comprises utilizing a math model of homographic transformation represented by a 3×

3 matrix.

12. The method of claim 1, wherein calibrating the cameras comprises calibrating the cameras via an initial calibration process and, following the initial calibration process, a fine tuning process that fine tunes to enhance stitching for images derived from image data captured by the cameras that are displayable at a display screen of the vehicle.

13. The method of claim 1, comprising determining a degree of curvature of horizontal lines by detecting edges and points of the provided targets.

14. The method of claim 1, comprising providing a display device operable to display images derived from image data captured by the cameras.

15. A method of calibrating cameras of a multi-camera vision system for a vehicle moving along a vehicle assembly line, said method comprising:

providing a plurality of targets at a vehicle assembly line at a vehicle assembly plant;

equipping a vehicle that is being conveyed along the vehicle assembly line with a plurality of cameras that includes (i) a front camera disposed at a front portion of the vehicle and viewing at least forward of the vehicle, (ii) a driver-side camera disposed at a driver side of the vehicle and viewing at least sideward of the vehicle, (iii) a passenger-side camera disposed at a passenger side of the vehicle and viewing at least sideward of the vehicle, and (iv) a rear camera disposed at a rear of the vehicle and viewing at least rearward of the vehicle;

moving the vehicle along the vehicle assembly line;

wherein providing a plurality of targets at the vehicle assembly line comprises providing a plurality of calibrating targets;

wherein, when the vehicle is moved along the vehicle assembly line so as to be at the provided calibrating targets, at least some targets of the calibrating targets are encompassed by respective fields of view of the front camera, the driver-side camera, the passenger-side camera and the rear camera;

wherein providing a plurality of calibrating targets comprises providing a plurality of calibrating targets at ground locations at the vehicle assembly line so as to be within overlapping portions of the fields of views of respective pairs of cameras when the vehicle is moved so as to be at the calibrating targets;

capturing image data with the cameras as the vehicle moves along the vehicle assembly line;

determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one target of the plurality of targets in the field of view of at least one of the cameras;

responsive to determination of presence of at least one target in the field of view of at least one of the cameras, calibrating the cameras via processing captured image data and determining presence of the calibrating targets in the respective fields of view of the front camera, the driver-side camera, the passenger-side camera, and the rear camera as the vehicle moves along the vehicle assembly line; and

wherein calibrating the cameras comprises, via processing of image data captured by the cameras, determining multi-axial orientation of the cameras.

16. The method of claim 15, wherein providing a plurality of targets at the vehicle assembly line comprises providing at least one triggering target at the vehicle assembly line, and wherein calibrating the cameras is responsive to determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one triggering target in the field of view of at least one of the cameras.

17. The method of claim 15, wherein determining multi-axial deviation of the cameras comprises determining pitch, yaw and roll of each of the cameras.

18. A method of calibrating cameras of a multi-camera vision system for a vehicle moving along a vehicle assembly line, said method comprising:

providing a plurality of targets at a vehicle assembly line at a vehicle assembly plant;

equipping a vehicle that is being conveyed along the vehicle assembly line with a plurality of cameras that includes (i) a front camera disposed at a front portion of the vehicle and viewing at least forward of the vehicle, (ii) a driver-side camera disposed at a driver side of the vehicle and viewing at least sideward of the vehicle, (iii) a passenger-side camera disposed at a passenger side of the vehicle and viewing at least sideward of the vehicle, and (iv) a rear camera disposed at a rear of the vehicle and viewing at least rearward of the vehicle;

moving the vehicle along the vehicle assembly line;

wherein providing a plurality of targets at the vehicle assembly line comprises providing a plurality of calibrating targets;

wherein, when the vehicle is moved along the vehicle assembly line so as to be at the provided calibrating targets, at least some targets of the calibrating targets are encompassed by respective fields of view of the front camera, the driver-side camera, the passenger-side camera and the rear camera;

wherein providing a plurality of calibrating targets comprises providing a plurality of calibrating targets at ground locations at the vehicle assembly line so as to be within overlapping portions of the fields of views of respective pairs of cameras when the vehicle is moved so as to be at the calibrating targets;

capturing image data with the cameras as the vehicle moves along the vehicle assembly line;

determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one target of the plurality of targets in the field of view of at least one of the cameras;

responsive to determination of presence of at least one target in the field of view of at least one of the cameras, calibrating the cameras via processing captured image data and determining presence of the calibrating targets in the respective fields of view of the front camera, the driver-side camera, the passenger-side camera, and the rear camera as the vehicle moves along the vehicle assembly line; and

wherein calibrating the cameras comprises calibrating the cameras via an initial calibration process and, following the initial calibration process, a fine tuning process that fine tunes to enhance stitching for images derived from image data captured by the cameras that are displayable at a display screen of the vehicle.

19. The method of claim 18, wherein providing a plurality of targets at the vehicle assembly line comprises providing at least one triggering target at the vehicle assembly line, and wherein calibrating the cameras is responsive to determining, via processing of captured image data and as the vehicle moves along the vehicle assembly line, presence of at least one triggering target in the field of view of at least one of the cameras.

20. The method of claim 18, comprising determining a degree of curvature of horizontal lines by detecting edges and points of the provided targets.