Method and system for dynamically calibrating vehicular cameras

• US 9,834,153 B2
• Filed: 04/25/2012
• Issued: 12/05/2017
• Est. Priority Date: 04/25/2011
• Status: Active Grant

• Current Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Original Assignee: Magna Electronics Incorporated (Magna International Incorporated)
• Inventors: Gupta, Nikhil, Faraji, Hilda, He, Daan, Rathi, Ghanshyam
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): Baptiste, Jerry Jean
• Application Number:
  US14/113,412
• Publication Number:
  US 20150049193A1
• Time in Patent Office: 2,050 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: G03B 43/00 Testing correct operation of photographic apparatus or parts thereof
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 17/002 for television cameras

Abstract

A method of dynamically calibrating a given camera relative to a reference camera of a vehicle includes identifying an overlapping region in an image frame provided by the given camera and an image frame provided by the reference camera and selecting at least a portion of an object in the overlapped region of the reference image frame. Expected pixel positions of the selected object portion in the given image frame is determined based on the location of the selected object portion in the reference image frame, and pixel positions of the selected object portion are located as detected in the given image frame. An alignment of the given camera is determined based on a comparison of the pixel positions of the selected object portion in the given image frame to the expected pixel positions of the selected object portion in the given image frame.

Claims

1. A method of dynamically calibrating a given vehicular camera relative to a reference camera attached to the vehicle, the given camera and the reference camera comprising pixelated imaging arrays, said method comprising:

providing a plurality of cameras at the vehicle so that the cameras have respective exterior fields of view, the cameras capturing image data;

selecting from the plurality of cameras a reference camera and a given camera, wherein the selected reference camera and the selected given camera have overlapping fields of view;

capturing an actual given image frame of image data by the given camera;

capturing an actual reference image frame of image data by the reference camera;

identifying an overlapping region in the actual given image frame of image data captured by the given camera and the actual reference image frame of image data captured by the reference camera;

selecting at least one object or at least a portion of an object present in the overlapping region of the actual reference image frame captured by the reference camera;

determining expected pixel positions of the selected object or object portion for a predicted given image frame as would be captured by the given camera based on the location of the selected object or object portion in the actual reference image frame captured by the reference camera;

locating, via processing of the actual given image frame captured by the given camera, actual pixel positions of the selected object or object portion as detected in the actual given image frame captured by the given camera; and

determining alignment of the given camera based at least in part on a comparison of the actual pixel positions of the selected object or object portion in the actual given image frame captured by the given camera to the expected pixel positions of the selected object or object portion for the predicted given image frame.

2. The method of claim 1, wherein the reference camera comprises a forward viewing camera of the vehicle and the given camera comprises a camera disposed at a side of the vehicle.

3. The method of claim 2, wherein said method first determines whether the forward viewing camera is calibrated.

4. The method of claim 3, wherein said method determines the calibration of the forward viewing camera by processing the image frame captured by the forward viewing camera to determine a location of a vanishing point in the image frame and comparing the determined vanishing point to a predetermined location in the image frame.

5. The method of claim 1, including removing distortion in the location of the pixel positions in the actual given image frame for the selected object or object portion caused by use of a wide angle camera lens in the given camera.

6. The method of claim 5, including removing distortion in locations of pixel positions in the actual reference image frame for the selected object or object portion caused by use of a wide angle camera lens in the reference camera.

7. The method of claim 5, wherein the reference camera comprises a forward viewing camera and the given camera comprises a side camera of the vehicle.

8. The method of claim 1, wherein the reference camera comprises a forward viewing camera and the given camera comprises a side camera of the vehicle, and wherein the given camera is disposed at a side mirror of the vehicle.

9. The method of claim 1, wherein the reference camera comprises a rearward viewing camera and the given camera comprises a side camera of the vehicle, and wherein the given camera is disposed at a side mirror of the vehicle.

10. The method of claim 1, wherein the given camera comprises a side camera of the vehicle.

11. The method of claim 1, wherein the reference camera and the given camera are part of a surround view vision system of the equipped vehicle.

12. The method of claim 11, wherein the reference camera comprises one of a forward viewing camera and a rearward viewing camera, and wherein the given camera comprises one of a driver side camera and a passenger side camera.

13. The method of claim 12, wherein the system comprises multiple given cameras including a driver side camera and a passenger side camera.

14. A method of dynamically calibrating a given vehicular camera relative to a reference camera attached to the vehicle, the given camera and the reference camera comprising pixelated imaging arrays, said method comprising:

providing a plurality of cameras at the vehicle so that the cameras have respective exterior fields of view, the cameras capturing image data;

selecting from the plurality of cameras a reference camera and a given camera, wherein the selected reference camera and the selected given camera have overlapping fields of view;

capturing an actual given image frame of image data by the given camera;

capturing an actual reference image frame of image data by the reference camera;

identifying an overlapping region in the actual given image frame of image data captured by the given camera and the reference image frame of image data captured by the reference camera;

wherein the reference camera and the given camera are part of a surround view vision system of the vehicle;

wherein the given camera comprises one of a driver side camera at a driver side mirror of the vehicle and a passenger side camera at a passenger side mirror of the vehicle;

wherein the reference camera comprises one of a forward viewing camera at a front portion of the vehicle and a rearward viewing camera at a rear portion of the vehicle;

selecting at least one object or at least a portion of an object present in the overlapping region of the actual reference image frame captured by the reference camera;

determining expected pixel positions of the selected object or object portion for a predicted given image frame as would be captured by the given camera based on the location of the selected object or object portion in the actual reference image frame captured by the reference camera;

locating, via processing of the actual given image frame captured by the given camera, actual pixel positions of the selected object or object portion as detected in the actual given image frame captured by the given camera; and

determining alignment of the given camera based at least in part on a comparison of the actual pixel positions of the selected object or object portion in the actual given image frame captured by the given camera to the expected pixel positions of the selected object or object portion for the predicted given image frame.

15. The method of claim 14, including removing distortion in the location of the pixel positions in the actual given image frame for the selected object or object portion caused by use of a wide angle camera lens in the given camera.

16. The method of claim 15, including removing distortion in locations of pixel positions in the actual reference image frame for the selected object or object portion caused by use of a wide angle camera lens in the reference camera.

17. A method of dynamically calibrating a given vehicular camera relative to a reference camera attached to the vehicle, the given camera and the reference camera comprising pixelated imaging arrays, said method comprising:

providing a plurality of cameras at the vehicle so that the cameras have respective exterior fields of view, the cameras capturing image data;

selecting from the plurality of cameras a reference camera and a given camera, wherein the selected reference camera and the selected given camera have overlapping fields of view;

capturing an actual given image frame of image data by the given camera;

capturing an actual reference image frame of image data by the reference camera;

determining whether the given camera is calibrated;

responsive to determining that the given camera is calibrated, identifying an overlapping region in the actual given image frame of image data captured by the given camera and the actual reference image frame of image data captured by the reference camera;

selecting at least one object or at least a portion of an object in the overlapping region of the actual reference image frame captured by the reference camera;

determining expected pixel positions of the selected object or object portion for a predicted given image frame as would be captured by the given camera based on the location of the selected object or object portion in the actual reference image frame captured by the reference camera;

locating, via processing of the actual given image frame captured by the given camera, actual pixel positions of the selected object or object portion as detected in the actual given image frame captured by the given camera; and

determining alignment of the given camera based at least in part on a comparison of the actual pixel positions of the selected object or object portion in the actual given image frame captured by the given camera to the expected pixel positions of the selected object or object portion for the predicted given image frame.

18. The method of claim 17, wherein said method determines the calibration of the forward viewing camera by processing the image frame of the forward viewing camera to determine a location of a vanishing point in the image frame and comparing the determined vanishing point to a predetermined location in the image frame.

19. The method of claim 17, wherein the reference camera and the given camera are part of a surround view vision system of the equipped vehicle.

20. The method of claim 19, wherein the reference camera comprises one of a forward viewing camera at a front portion of the vehicle and a rearward viewing camera at a rear portion of the vehicle, and wherein the given camera comprises one of a driver side camera at a driver side of the vehicle and a passenger side camera at a passenger side of the vehicle.