System for calibrating a vision system

• US 9,311,706 B2
• Filed: 05/30/2012
• Issued: 04/12/2016
• Est. Priority Date: 06/01/2011
• Status: Active Grant

• Current Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman Becker Automotive Systems GmbH (Samsung Electronics Co. Ltd.)
• Inventors: Natroshvili, Koba, Scholl, Kay-Ulrich, Quast, Johannes
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): JEAN BAPTISTE, JERRY T
• Application Number:
  US13/483,565
• Publication Number:
  US 20120320190A1
• Time in Patent Office: 1,413 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

A system for calibrating a vision system of an object. For example, a system for calibrating a surround view system of a vehicle. The system may be configured to capture, via a first and second image sensor, a first and second image of a plurality of objects. Upon capturing such information, the system may be configured to determine a position and orientation of the first and the second image sensor relative to the plurality of objects based on the first and the second captured image. Further, the system may be configured to determine a relative position and orientation between the first and the second image sensor based on a result of the determining a position and orientation of the first and the second image sensor. Also, the system may be configured to determine intrinsic and extrinsic parameters of the first and the second image sensor.

Claims

1. A non-transitory computer-readable medium for calibrating a vision system of an object, configured to:

receive, from a first image sensor of the vision system, a first image of a plurality of objects;

receive, from a second image sensor of the vision system, a second image of the plurality of objects, where the plurality of objects is located in an overlapping region of a field of view of the first image sensor and a field of view of the second image sensor;

determine a position and orientation of the first and the second image sensor relative to the plurality of objects based on the first and the second image, respectively, wherein determining the position and orientation of the first image sensor comprises determining a vector which comprises both a plurality of rotation parameters and a plurality of translation parameters of a camera matrix of the first image sensor, said vector being determined by solving a minimization problem under a constraint that a modulus of the vector which comprises both the plurality of rotation parameters and the plurality of translation parameters is fixed;

determine a relative position and orientation between the first and the second image sensor based on a result of the determining a position and orientation of the first and the second image sensor;

determine extrinsic parameters of the first and the second image sensor in a reference frame based on results of the determining a relative position and orientation between the first and the second image sensor;

determine intrinsic parameters of the first and the second image sensor based on the first and the second image, respectively; and

store the determined relative position and orientation between the first and the second image sensor and the determined intrinsic parameters for combining images captured by the vision system in performance of a driver assistance function.

2. The non-transitory computer-readable medium of claim 1, wherein the non-transitory computer-readable medium is further configured to:

repeat the receiving and determinations for a plurality of pairs of image sensors having overlapping fields of view.

3. The non-transitory computer-readable medium of claim 2, wherein a first pair of the plurality of pairs of image sensors includes a first pair image sensor and an intermediate image sensor, wherein a second pair of the plurality of pairs of image sensors includes the intermediate image sensor and a second pair image sensor, and wherein the non-transitory computer-readable medium is further configured to:

determine a relative position and orientation between the first pair image sensor and the second pair image sensor based on a result of the determining a relative position and orientation between the first and the second image sensor for the first pair and the second pair, respectively.

4. The non-transitory computer-readable medium of claim 3, wherein the non-transitory computer-readable medium is further configured to:

compare the relative position and orientation between the first pair image sensor and the second pair image sensor to the relative position and orientation between an other first pair image sensor and an other second pair image sensor of an other plurality of pairs of image sensors.

5. The non-transitory computer-readable medium of claim 3, wherein a field of view of the first pair image sensor and a field of view of the second pair image sensor are disjoint.

6. The non-transitory computer-readable medium of claim 3,

wherein the first pair image sensor and the second pair image sensor are provided at opposite sides of the object.

7. The non-transitory computer-readable medium of claim 1, wherein the object is a vehicle and the vision system is a vehicle vision system.

8. The non-transitory computer-readable medium of claim 1, wherein the determining a position and orientation of an image sensor comprises:

determining the vector {right arrow over (M)}, wherein ∥

L{right arrow over (M)}∥

becomes a minimum under a constraint that the vector {right arrow over (M)} comprises a fixed modulus, wherein L is a matrix having a number of columns that is equal to a number of rows of the vector {right arrow over (M)}, wherein ∥

∥

denotes a vector modulus function, and wherein the vector {right arrow over (M)} comprises matrix elements that are elements of the camera matrix of the first image sensor.

9. The non-transitory computer-readable medium of claim 8, wherein the vector {right arrow over (M)} is defined as

10. The non-transitory computer-readable medium of claim 8, wherein the vector {right arrow over (M)} is independent of intrinsic parameters of the image sensor which define an image distortion.

11. The non-transitory computer-readable medium of claim 8, wherein the matrix L is independent of intrinsic parameters of the image sensor, the matrix L being dependent on coordinates of the plurality of objects and on coordinates of image points in the first image to which the plurality of objects are projected.

12. The non-transitory computer-readable medium of claim 1, wherein determining the extrinsic parameters in the reference frame comprises determining projections of the plurality of objects.

13. The non-transitory computer-readable medium of claim 7, wherein the vehicle vision system is a surround view system and the non-transitory computer-readable medium is further configured to:

render a surround view based on the received images.

21. The non-transitory computer-readable medium of claim 1, wherein the intrinsic parameters of the first and the second image sensor define a non-linear distortion.

14. A method for calibrating a vision system of an object, comprising:

capturing, via a first image sensor of the vision system, a first image of a plurality of objects;

capturing, via a second image sensor of the vision system, a second image of the plurality of objects, where the plurality of objects is located in an overlapping region of a field of view of the first image sensor and a field of view of the second image sensor;

determining a position and orientation of the first and the second image sensor relative to the plurality of objects based on the first and the second captured image, respectively, wherein determining the position and orientation of the first image sensor comprises determining a vector which comprises both a plurality of rotation parameters and a plurality of translation parameters of a camera matrix of the first image sensor, said vector being determined by solving a minimization problem under a constraint that a modulus of the vector which comprises both the plurality of rotation parameters and the plurality of translation parameters is fixed;

determining a relative position and orientation between the first and the second image sensor based on a result of the determining a position and orientation of the first and the second image sensor;

determining a relative position and orientation between a first pair image sensor and a second pair image sensor based on a result of the determining a relative position and orientation between the first and the second image sensor for a first pair and a second pair, respectively, wherein the first pair of a plurality of pairs of image sensors includes the first pair image sensor and an intermediate image sensor, and wherein the second pair of the plurality of pairs of image sensors includes the intermediate image sensor and the second pair image sensor; and

storing the determined relative position and relative orientation between the first and the second image sensor for combining images captured by the vision system in performance of a driver assistance function.

15. The method of claim 14, wherein the plurality of objects includes at least three objects located on a first plane and at least three objects located on a second plane.

16. The method of claim 15, wherein the plurality of objects includes at least three objects located on a third plane, and wherein the first plane, the second plane, and the third plane are orthogonal.

17. A vehicle vision system of a vehicle, comprising:

a plurality of image sensors each comprising an electro-optical component to capture images comprising a non-linear distortion, wherein the image sensors are located at different locations on the vehicle hosting the vehicle vision system;

an image processing unit configured to process image data captured by the plurality of image sensors; and

a calibration device, wherein the calibration device is coupled to the image processing unit, and wherein the calibration device is configured to;

identify, via a first image sensor of the vehicle vision system, a first image of a plurality of objects;

identify, via a second image sensor of the vehicle vision system, a second image of the plurality of objects, where the plurality of objects is located in an overlapping region of a field of view of the first image sensor and a field of view of the second image sensor;

determine a position and orientation of the first and the second image sensor relative to the plurality of objects based on the first and the second captured image, respectively, wherein determining the position and orientation of the first image sensor comprises determining a vector which comprises both a plurality of rotation parameters and a plurality of translation parameters of a camera matrix of the first image sensor, said vector being determined by solving a minimization problem under a constraint that a modulus of the vector which comprises both the plurality of rotation parameters and the plurality of translation parameters is fixed;

determine a relative position and orientation between the first and the second image sensor based on a result of the determining a position and orientation of the first and the second image sensor;

determine extrinsic parameters of the first and the second image sensor in a reference frame based on results of the determining a relative position and orientation between the first and the second image sensor;

determine intrinsic parameters of the first and the second image sensor based on the first and the second captured image, respectively; and

store the determined relative position and orientation between the first and the second image sensor and the determined intrinsic parameters for combining images captured by the vehicle vision system in performance of a driver assistance function.

18. The system of claim 17, wherein the system is a surround view system.

19. The system of claim 17, wherein the plurality of image sensors comprise fisheye lenses.

20. The system of claim 17, wherein at least two of the plurality of image sensors are located at two respective side door mirrors of the vehicle hosting the system.

22. The system of claim 17, wherein the intrinsic parameters of the first and the second image sensor define a non-linear distortion.