MULTI-SENSOR CAMERA RECALIBRATION

• US 20140300736A1
• Filed: 04/09/2013
• Published: 10/09/2014
• Est. Priority Date: 04/09/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Karner, Konrad, Bauer, Joachim, Ponticelli, Martin, Gruber, Michael, Lawler, Stephen, Reitinger, Bernhard, Hoefler, Mario
• Application Number:
  US13/859,117
• Publication Number:
  US 20140300736A1
• Time in Patent Office: Days
• US Class Current: 348/144
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

One or more techniques and/or systems are providing for facilitating recalibration of a multi-sensor camera. That is, a multi-sensor camera may comprise a nadir sensor and one or more oblique sensors. Temperature, mechanical stress, and other factors can lead to misalignment of one or more sensors within the multi-sensor camera. Accordingly, a set of tie points and/or observations may be generated based upon a search matching technique, a densification technique, and/or a virtual matching technique. A bundle technique may be utilized to generate updated eccentricity information based upon the set of tie points and/or observations. The updated eccentricity information (e.g., orientation and/or position information of a sensor, such as an oblique sensor, with respect to a nadir view) may be used to recalibrate the multi-sensor camera, such as in real-time (e.g., during a flight mission that utilizes the multi-sensor camera to capture aerial images of a city or other scene).

Claims

1. A system for facilitating recalibration of a multi-sensor camera, comprising:

an image identification component configured to;

obtain a first set of images captured from a first sensor of a multi-sensor camera; and

obtain a second set of images captured from a second sensor of the multi-sensor camera; and

a search matching component configured to;

identify a set of image matching pairs between one or more images captured by the multi-sensor camera based upon at least one of the first set of images or the second set of images, a first image matching pair identifying a correspondence region between a first image and a second image of the first image matching pair; and

perform pair-wise image matching upon the set of image matching pairs to generate a set of tie points corresponding to eccentricity information to facilitate recalibration of the multi-sensor camera.

2. The system of claim 1, comprising:

a bundle adjustment component configured to;

iteratively evaluate respective tie points within the set of tie points using initial calibration information of the multi-sensor camera to compute an estimated statistical error distribution for the set of tie points;

generate a set of weights based upon the estimated statistical error distribution; and

apply the set of weights to the set of tie points using a non-linear optimization procedure to generate updated eccentricity information.

3. The system of claim 2, comprising:

a recalibration component configured to recalibrate the multi-sensor camera based upon the updated eccentricity information.

4. The system of claim 1, the first sensor comprising a first oblique sensor, and the search matching component configured to:

identify an image matching pair based upon identifying overlap between a first image and a second image within the first set of images using an image matching technique.

5. The system of claim 1, the first sensor comprising a first oblique sensor and the second sensor comprising a nadir sensor, and the search matching component configured to:

identify an image matching pair based upon identifying overlap between a first image within the first set of images and a second image within the second set of images.

6. The system of claim 1, the image identification component configured to obtain a third set of images captured from a third sensor of the multi-sensor camera, and the search matching component configured to:

identify an image matching pair based upon identifying overlap between a first image of the first set of images, a second image within the second set of images, and a third image within the third set of images, the first sensor comprising a first oblique sensor, the second sensor comprising a second oblique sensor, and the third sensor comprising a nadir sensor.

7. The system of claim 1, the multi-sensor camera comprising an aerial camera.

8. The system of claim 1, the first sensor comprising a first oblique sensor and the second sensor comprising a second wing sensor.

9. The system of claim 1, the first sensor comprising a first oblique sensor and the second sensor comprising a nadir sensor.

10. The system of claim 1, the search matching component configured to:

identify a first tie point based upon a first image matching pair corresponding to a second image matching pair, the first image matching pair comprising a first image and a second image, the second image matching pair comprising the second image and a third image, the first tie point comprised within the first image, the second image, and the third image.

11. The system of claim 7, comprising an image measurement filter comprising at least one of:

a nadir to nadir measurement filter;

a baseline point measurement filter corresponding to an exposure event timing threshold;

or

a redundant point filter.

12. The system of claim 1, comprising:

a densification component configured to;

estimate a 3D point based upon a nadir aerial triangulation of the multi-sensor camera;

re-project the 3D point into an image within the first set of images to obtain a corresponding coordinate of the 3D point within the image, the first sensor comprising a first oblique sensor; and

generate an observation based upon the corresponding coordinate using an image matching technique.

13. The system of claim 12, the search matching component configured to:

update the set of tie points based upon the observation.

14. A method for facilitating recalibration of a multi-sensor camera, comprising:

obtaining a set of images captured by one or more sensors of a multi-sensor camera;

estimating a 3D point based upon a nadir aerial triangulation of the multi-sensor camera;

re-projecting the 3D point into a first image within the set of images to obtain a corresponding coordinate of the 3D point within the first image, the first image captured by a first oblique sensor of the multi-sensor camera; and

generating a first observation based upon the corresponding coordinate using an image matching technique, the first observation indicative of whether the 3D point is comprised within the first image, the first observation used to facilitate recalibration of the multi-sensor camera.

15. The method of claim 14, comprising:

generating a set of observations based upon re-projecting a set of 3D points into respective images within the set of images, the set of observations comprising the first observation.

16. The method of claim 15, comprising:

iteratively evaluating respective observations within the set of observations using initial calibration information of the multi-sensor camera to compute an estimated statistical error distribution for the set of observations;

generating a set of weights based upon the estimated statistical error distribution; and

applying the set of weights to the set of observations using a non-linear optimization procedure to generate updated eccentricity information for the multi-sensor camera.

17. The method of claim 16, comprising:

recalibrating the multi-sensor camera based upon the updated eccentricity information.

18. A method for facilitating recalibration of a multi-sensor camera, comprising:

obtaining a first set of images captured by a first sensor of a multi-sensor camera;

obtaining a second set of images captured by a second sensor of the multi-sensor camera;

constructing a digital surface model (DSM) using a dense image matching technique based upon the first set of images and aerial triangulation associated with the first set of images;

texturing the DSM using the first set of images to create a textured DSM;

generating a set of synthetic rendered images from the textured DSM using a camera pose manifold associated with one or more oblique sensors of the multi-sensor camera; and

generating a set of tie points based upon evaluating the set of synthetic rendered images against the second set of images using an image matching technique, the set of tie points used for facilitating recalibration of the multi-sensor camera.

19. The method of claim 18, comprising:

iteratively evaluating respective tie points within the set of tie points using initial calibration information of the multi-sensor camera to compute an estimated statistical error distribution for the set of tie points;

generating a set of weights based upon the estimated statistical error distribution; and

applying the set of weights to the set of tie points using a non-linear optimization procedure to generate updated eccentricity information.

20. The method of claim 19, comprising:

recalibrating the multi-sensor camera based upon the updated eccentricity information.