Method For Automatically Calibrating A Virtual Camera System

• US 20100110194A1
• Filed: 10/23/2009
• Published: 05/06/2010
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: Magna Electronics Europe Gmbh & Co. Kg
• Original Assignee: Magna Electronics Europe Gmbh & Co. Kg
• Inventors: Adameck, Markus, EULER, Christian
• Granted Patent: US 8,169,480 B2
• Time in Patent Office: Days
• US Class Current: 348/149
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 1/27 providing all-round vision, e.g. using omnidirectional cameras
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/303 using joined images, e.g. multiple camera images
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method for producing a time sequence of images of motor vehicle surroundings from a virtual elevated central perspective on a display wherein several cameras are used to record time sequences of subimage data records from a plurality of real perspectives differently offset and tilted with respect to the virtual perspective. Then, parameterized transformations are applied for obtaining sequences of transformed subimage data records and overall image data for an overall image from the virtual perspective is assembled. Finally, the sequence of the overall image data is displayed as an overall image. To compensate for positional and/or orientation deviations in the recording cameras a selection unit is proposed to recurrently apply a selection criterion for selecting a subimage data record from the sequence and to take this as a basis for redetermining the parameters of the transformations by an optimization method under a quality criterion for the overall image.

Claims

1. Method for producing a chronological sequence of images of a motor vehicle and of the adjacent environment from a virtual elevated central perspective (P_v) above the motor vehicle on a display device (5), wherein

a. a plurality of real recording camera devices (1, 2, 3, 4) are used to record a chronological sequence of subimage data records (DR_{1 . . . N}) with coded subimages (7, 8, 9, 10) from a plurality of real perspectives (P_{1 . . . 4}) which are differently offset and/or tilted with respect to the virtual perspective (P_v),

b. differently parametrized transformations (T_{1 . . . 4}) are applied to the image data (DR_{i,1 . . . 4}) from each subimage data record (DR_i) in an image data processing device (5) in order to obtain a sequence of transformed subimage data records (DT_{1 . . . N}),

c. the sequence of the transformed subimage data records (DT_{1 . . . N}) is assembled to form a sequence of overall image data (DG_{1 . . . N}) for an overall image (11) from the virtual perspective (P_v), and

d. the overall image data (DG_i) from the sequence of the overall image data (DG_{1 . . . N}) are successively displayed on a display device (6),

e. characterized in that a selection unit (6) is used to recurrently apply a selection criterion (C) in order to select from the sequence of subimage data records (DR_{1 . . . N}) a subimage data record (DR_k) for which the parameters of the transformations (T_{1 . . . 4}) are redetermined by an optimization method using a quality criterion (Q) for the overall image.

2. Method according to claim 1, wherein the selection criterion (C) is determined by processing information about the state and the driving situation of the motor vehicle, particularly by processing vehicle state variables and driving state variables from detection devices on the motor vehicle.

3. Method according to claim 2, wherein the number of vehicle state variables detected by the selection criterion (C) is selected from the mileage of the vehicle, the number of vehicle starts and the running time of the vehicle.

4. Method according to claim 2, wherein the number of vehicle state variables detected by the selection criterion (C) is selected from the speed of travel, the acceleration, the steering angle, the GPS position, the exterior lighting conditions, the angle of inclination and the loading.

5. Method according to one of the preceding claims, wherein the selection criterion (C) is determined by processing information from the sequence of the subimage data records (DR_{1 . . . k}).

6. Method according to claim 5, wherein respective fixed points (F1_{1 . . . 4}) for determining the real position and/or perspective (P_{1 . . . 4}) of the respective real camera (1, 2, 3, 4) are extracted from the sequence of the subimage data (DR_{1 . . . k}).

7. Method according to claims 5 and 6, wherein the computation unit (4) is used to extract structural features and, in particular, periodic structures from the subimage data (DR_k,1).

8. Method according to claim 7, wherein the selection criterion (C) suppresses the use of subimage data records (DR_k) with periodic structures for recalculating the parameters of the transformations (T_{1 . . . 4}).

9. Method according to one of claims 5 to 8, wherein the selection criterion (C) favours subimage data records (DR_k) from the recording of flat objects and, in particular, the road surface.

10. Method according to one of the preceding claims, wherein the selection criterion (C) refers to the history of the parameters determined in preceding steps for the purpose of a plausibility check, in particular by applying a statistical method.

11. Method according to one of the preceding claims, wherein the quality criterion (Q) for the overall image (11) involves the calculation of a correlation for the subimage data (DR_k,1) from a subimage data record (DR_k) in the region of a possible region of overlap (12).