Bundle adjustment based on image capture intervals

• US 8,897,543 B1
• Filed: 05/18/2012
• Issued: 11/25/2014
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Michael Hongmai, Reinhardt, Tilman
• Primary Examiner(s): Liew, Alex
• Application Number:
  US13/475,664
• Time in Patent Office: 921 Days
• Field of Search: 382/154
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

Methods, systems, and computer program products are provided for determining camera parameters and three dimensional locations of features from a plurality of images of a geographic area. These include, determining a correlation between a pose of a first camera and a pose of a second camera, generating one or more constraints incorporating the correlation, and determining at least one of camera parameters and three dimensional locations of features using a plurality of constraints including the generated one or more constraints. The first camera and the second camera have substantially rigid positions and poses relative to each other. A strength of the correlation is based at least upon a time interval between respective image captures by the first camera and the second camera.

Claims

1. A method of determining at least one of camera parameters and three dimensional locations of features in images from a plurality of images of an area, comprising:

determining a correlation between a pose of a first camera and a pose of a second camera, wherein a strength of the correlation is based at least upon a time difference between respective image captures by the first camera and the second camera, and wherein positions and poses of the first camera and the second camera are substantially rigid relative to each other;

generating one or more constraints incorporating the correlation; and

determining at least one of camera parameters and three dimensional locations of features using a plurality of constraints including the generated one or more constraints.

2. The method of claim 1, wherein the determining a correlation comprises:

determining the strength of the correlation, wherein the strength is a decreasing function of a duration of the time difference.

3. The method of claim 1, wherein the correlation is defined at least in part by data indicating an arrangement of the cameras in a camera rosette when the images were captured.

4. The method of claim 3, further comprising:

receiving position and timestamp information for each of the plurality of images captured by cameras in the camera rosette from an airborne platform.

5. The method of claim 4, wherein the respective image captures by the first camera and the second camera occur in one iteration of image captures by cameras in the camera rosette.

6. The method of claim 4, wherein the respective image captures by the first camera and the second camera occur with the camera rosette at different positions in a path of the airborne platform.

7. The method of claim 1, wherein the first and second cameras are configured to acquire oblique imagery.

8. The method of claim 1, wherein the generating one or more constraints comprises:

forming one or more constraints relating the pose of the second camera to the first camera.

9. The method of claim 1, wherein the generating one or more constraints comprises:

forming one or more constraints relating the time difference to an estimated pose adjustment.

10. The method of claim 1, wherein the determining at least one of camera parameters and three dimensional locations of features comprises:

determining a pose adjustment for the first camera.

11. The method of claim 10, wherein the determining at least one of camera parameters and three dimensional locations of features further comprises:

determining a three dimensional location of one or more features corresponding to the determined pose adjustment for the first camera.

12. The method of claim 1, wherein the generating one or more constraints include:

prioritizing each of the generated one or more constraints based upon a corresponding image capture interval.

13. The method of claim 12, wherein the prioritizing includes:

weighting constraints of the plurality of constraints, wherein the generated one or more constraints are assigned lower weights relative to a portion of the constraints that are based upon feature matching.

14. The method of claim 12, wherein ones of the generated one or more constraints are assigned lower weights relative to a portion of the constraints that are based upon feature matching and wherein a number of features to be matched exceeds a threshold.

15. A system for determining at least one of camera parameters and three dimensional locations of features in images from a plurality of images of an area, comprising:

at least one processor;

a camera pose adjustment estimator configured to be executed by the processor and further configured to determine a correlation between a pose of a first camera and a pose of a second camera, wherein a strength of the correlation is based at least upon a time difference between respective image captures by the first camera and the second camera, and wherein positions and poses of the first camera and the second camera are substantially rigid relative to each other;

a constraint generator configured to be executed by the processor and further configured to generate one or more constraints incorporating the correlation; and

a bundle adjuster configured to be executed by the processor and further configured to determine at least one of camera parameters and three dimensional locations of features using a plurality of constraints including the generated one or more constraints.

16. The system of claim 15, further comprising:

a camera rosette comprising a plurality of cameras and controlled by the at least one processor, wherein the plurality of cameras includes the first and second cameras.

17. The system of claim 15, wherein the camera pose adjustment estimator is further configured to determine the strength of the correlation, wherein the strength is a decreasing function of a duration of the time difference.

18. The system of claim 15, wherein the constraint generator is further configured to prioritize each of the one or more constraints based upon characteristics of one or more images.

19. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform a method of determining camera parameters and three dimensional locations of features from a plurality of images of an area, wherein the method comprises:

identifying a correlation tying a first camera pose and a second camera pose;

determining a strength of the correlation based at least upon a time difference between an image captured by a camera at the first pose at a first time and an image captured by a camera at a second pose at a second time, the correlation defined so that the strength decreases as the time difference increases;

generating one or more constraints incorporating the correlation as scaled by the determined strength; and

determining at least one of camera parameters and three dimensional locations of features using a plurality of constraints including the generated one or more constraints.

20. The non-transitory computer readable storage medium of claim 19, wherein the generating one or more constraints include:

prioritizing each of the generated one or more constraints based upon a corresponding image capture interval.