MOBILE IMAGING PLATFORM CALIBRATION

• US 20140368651A1
• Filed: 06/14/2013
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Karner, Konrad, Ponticelli, Martin, Irschara, Arnold, Schweighofer, Gerald, Ladstdter, Richard
• Granted Patent: US 9,430,822 B2
• Time in Patent Office: Days
• US Class Current: 348/148
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Mobile platforms are used to capture an area using a variety of sensors (e.g., cameras and laser scanners) while traveling through the area, in order to create a representation (e.g., a navigable set of panoramic images, or a three-dimensional reconstruction). However, such sensors are often precisely calibrated in a controlled setting, and miscalibration during travel (e.g., due to a physical jolt) may result in a corruption of data and/or a recalibration that leaves the platform out of service for an extended duration. Presented herein are techniques for verifying sensor calibration during travel. Such techniques involve the identification of a sensor path for each sensor over time (e.g., a laser scanner path, a camera path, and a location sensor path) and a comparison of the paths, optionally after registration with a static coordinate system, to verify that the continued calibration of the sensors during the mobile operation of the platform.

Claims

1. A system for calibrating a platform traveling through an area and comprising a camera, a laser scanner, and a location sensor, the system comprising:

a path identifier that;

with the laser scanner, compares laser scans of the area captured at respective times to identify a laser scanner path through the area;

with the camera, compares images of the area captured at respective times to identify a camera path through the area; and

with the location sensor, identifies a location sensor path through the area; and

a platform calibrator that compares the laser scanner path, the camera path, and the location sensor path to verify a calibration of the platform.

2. A computer-readable storage device comprising instructions that, when executed on a processor of a platform traveling through an area and comprising a camera, a laser scanner, and a location sensor, calibrate the platform by:

with the laser scanner, comparing laser scans of the area captured at respective times to identify a laser scanner path through the area;

with the camera, comparing images of the area captured at respective times to identify a camera path through the area;

with the location sensor, identifying a location sensor path through the area; and

comparing the laser scanner path, the camera path, and the location sensor path to verify a calibration of the platform.

3. A method of calibrating a platform traveling through an area and comprising a camera, a laser scanner, and a location sensor, the method comprising:

with the laser scanner, comparing laser scans of the area captured at respective times to identify a laser scanner path through the area;

with the camera, comparing images of the area captured at respective times to identify a camera path through the area;

with the location sensor, identifying a location sensor path through the area; and

comparing the laser scanner path, the camera path, and the location sensor path to verify a calibration of the platform.

4. The method of claim 3, wherein:

the method further comprises;

calibrating the laser scanner, the camera, and the location sensor with the platform; and

comparing the laser scanner path and the camera path with the location sensor path further comprises;

comparing the laser scanner path and the camera path with the location sensor path in view of the calibration of the laser scanner, the camera, and the location sensor with the platform.

5. The method of claim 4, wherein calibrating the laser scanner and the camera further comprises:

positioning the platform relative to a fixed position of at least one target in a fixed area;

comparing a first detected position of the target by the laser scanner with the fixed position of the target relative to the platform; and

comparing a second detected position of the target by the camera with the fixed position of the target relative to the platform.

6. The method of claim 3, wherein the comparing further comprises:

comparing the laser scanner path and the camera path with the location sensor path to verify the calibration of the platform while the platform is traveling through the area.

7. The method of claim 3, wherein:

the method further comprises;

upon capturing a laser scan of the area, storing the laser scan;

upon capturing an image of the area, storing the image; and

upon detecting a location of the platform, storing the location of the platform; and

the comparing further comprises;

after the platform has traveled through the area;

retrieving stored laser scans, images, and locations; and

comparing the laser scanner path and the camera path with the location sensor path to verify the calibration of the platform while the platform was traveling through the area.

8. The method of claim 3, wherein verifying the calibration of the platform further comprises:

comparing the laser scanner path with the camera path;

comparing the laser scanner path with the location sensor path; and

comparing the camera path with the location sensor path.

9. The method of claim 8, wherein respective comparisons between a first path and a second path further comprise:

registering the first path with a platform-independent coordinate system to produce a registered first path;

registering the second path with the platform-independent coordinate system to produce a registered second path; and

comparing the registered first path and the registered second path.

10. The method of claim 9, wherein the platform-independent coordinate system comprises a static coordinate system.

11. The method of claim 3, wherein:

the location sensor is configured to perform a detection of a location of the platform at a location detection frequency;

capturing the laser scans of the area further comprises;

synchronizing the capturing of the laser scans with the detection of the location of the platform by the location sensor; and

capturing the images of the area further comprises;

synchronizing the capturing of the images with the detection of the location of the platform by the location sensor.

12. The method of claim 3, wherein the method further comprises:

upon detecting a missing data object from the capturing at a selected time, the missing data object selected from a missing data object set comprising;

a missing laser scan captured by the laser scanner,

a missing image captured by the camera, and

a missing location detected by the location sensor,

interpolating the missing data object at the selected time according to a quaternion interpolation.

13. The method of claim 3, wherein:

the laser scanner comprises a lidar scanner;

the laser scan comprises a lidar point cloud; and

identifying the laser scanner path further comprises;

comparing a first lidar point cloud captured at a first time and a second lidar point cloud captured at a second time to identify a movement of the laser scanner between the first time and the second time.

14. The method of claim 13, wherein identifying the movement further comprises:

performing an iterative closest point evaluation of the first lidar point cloud and the second lidar point cloud to identify the movement of the laser scanner.

15. The method of claim 13, wherein:

the method further comprises;

evaluating the first lidar point cloud to identify a first orientation of the laser scanner at the first time, and

evaluating the second lidar point cloud to identify a second orientation of the laser scanner at the second time; and

identifying the movement further comprises;

comparing the first orientation and the second orientation of the laser scanner to identify a rotation and a translation of the laser scanner between the first time and the second time.

16. The method of claim 15, wherein:

identifying the location sensor path further comprises;

with the location sensor, identifying, between the first time and the second time, a rotation and a translation; and

comparing the laser scanner path and the location sensor path further comprises;

comparing the rotation and the translation of the laser scanner between the first time and the second time with the rotation and the translation of the location sensor path between the first time and the second time.

17. The method of claim 13, wherein comparing the first lidar point cloud and the second lidar point cloud further comprises:

recomparing the first lidar point cloud and the second lidar point cloud in view of a comparison of the laser scanner path and the location sensor path.

18. The method of claim 3, wherein comparing the images to identify the camera path further comprises:

identifying a tie point visible in a first image captured at a first time and a second image captured at a second time; and

comparing a first position of the tie point in the first image and a second position of the tie point in the second image to identify the movement of the camera between the first time and the second time.

19. The method of claim 3, wherein identifying the camera path through the area further comprises:

from the images, generate a reconstruction of the area; and

for respective images captured at a time, identify the position of the camera within the reconstruction at the time.

20. The method of claim 19, wherein the instructions are further configured to perform a photogrammetric bundle adjustment of the reconstruction of the area.