Bundle adjustment based on image capture intervals

• US 9,058,538 B1
• Filed: 10/15/2014
• Issued: 06/16/2015
• Est. Priority Date: 05/18/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lin, Michael Hongmai, Reinhardt, Tilman
• Primary Examiner(s): Liew, Alex
• Application Number:
  US14/514,434
• Time in Patent Office: 244 Days
• Field of Search: 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

Methods, systems, and computer program products are provided for determining camera parameters and three dimensional locations of features from a plurality of images of a geographic area. These include, determining a correlation between a pose of a first camera and a pose of a second camera, generating one or more constraints incorporating the correlation, and determining at least one of camera parameters and three dimensional locations of features using a plurality of constraints including the generated one or more constraints. The first camera and the second camera have substantially rigid positions and poses relative to each other. A strength of the correlation is based at least upon a time interval between respective image captures by the first camera and the second camera.

Claims

1. A computer-implemented method for determining three dimensional locations of features in images from a plurality of images of an area, comprising;

receiving, by one or more computing devices, data associated with a pose of a first camera and a pose of a second camera, wherein positions and poses of the first and second cameras are substantially rigid relative to each other;

determining, by the one or more computing devices, a correlation between the pose of the first camera and the pose of the second camera, wherein a strength of the correlation is based at least upon a time difference between respective image captures by the first camera and the second camera;

generating, by the one or more computing devices, one or more constraints incorporating the correlation;

determining, by the one or more computing devices, a three dimensional location of a feature using a plurality of constraints, the plurality of constraints including the generated one or more constraints; and

in response to a request for the three dimensional location of the feature, providing, by the one or more computing devices, bundle adjusted information associated with the three dimensional location of the feature.

2. The computer-implemented method of claim 1, wherein determining the correlation comprises determining, by the one or more computing devices, the strength of the correlation, wherein the strength is a decreasing function of a duration of the time difference.

3. The computer-implemented method of claim 1, further comprising receiving, by the one or more computing devices, position and timestamp information for each of a plurality of images captured by the first and second cameras.

4. The computer-implemented method of claim 1, wherein the correlation is defined at least in part by data indicating an arrangement of the first and second cameras within a camera rosette, the camera rosette including a plurality of cameras.

5. The computer-implemented method of claim 4, wherein the respective image captures by the first and second cameras occur in one iteration of image captures by plurality of cameras in the camera rosette.

6. The computer-implemented method of claim 4, wherein the respective image captures by the first and second cameras occur with the camera rosette located at different positions along a travel path of the camera rosette.

7. The computer-implemented method of claim 1, wherein generating the one or more constraints comprises generating, by the one or more computing devices, one or more constraints relating the time difference to an estimated pose adjustment.

8. The computer-implemented method of claim 1, further comprising prioritizing each of the generated one or more constraints based upon the time difference associated with the respective image captures.

9. The computer-implemented method of claim 1, further comprising weighting constraints of the plurality of constraints, wherein the generated one or more constraints are assigned lower weights relative to a portion of the constraints that are based upon feature matching.

10. A system for determining three dimensional locations of features in images from a plurality of images of an area, the system comprising:

one or more computing devices including one or more processors and associated memory, the memory storing instructions that, when implemented by the one or more processors, configuring the one or more computing devices to;

receive data associated with a pose of a first camera and a pose of a second camera, wherein positions and poses of the first and second cameras are substantially rigid relative to each other;

determine a correlation between the pose of the first camera and the pose of the second camera, wherein a strength of the correlation is based at least upon a time difference between respective image captures by the first camera and the second camera;

generate one or more constraints incorporating the correlation;

determine a three dimensional location of a feature using a plurality of constraints including the generated one or more constraints; and

in response to a request for the three dimensional location of the feature, provide bundle adjusted information associated with the three dimensional location of the feature.

11. The system of claim 10, further comprising a camera rosette including a plurality of cameras communicatively coupled to the one or more computing devices, wherein the plurality of cameras of cameras includes the first and second cameras.

12. The system of claim 11, wherein the respective image captures by the first and second cameras occur in one iteration of image captures by the plurality of cameras in the camera rosette or the respective image captures by the first and second cameras occur with the camera rosette located at different positions along a travel path of the camera rosette.

13. The system of claim 11, wherein the respective image captures by the first and second cameras occur with the camera rosette located at different positions along a travel path of the camera rosette.

14. The system of claim 10, wherein the one or more computing devices are further configured to determine the strength of the correlation, wherein the strength is a decreasing function of a duration of the time difference.

15. The system of claim 10, wherein the one or more computing devices are further configured to receive position and timestamp information for each of a plurality of images captured by the first and second cameras.

16. The system of claim 10, wherein the one or more computing devices are configured to generate one or more constraints relating the time difference to an estimated pose adjustment.

17. A tangible, non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations, comprising:

receiving data associated with a pose of a first camera and a pose of a second camera, wherein positions and poses of the first and second cameras are substantially rigid relative to each other;

determining a correlation between the pose of the first camera and the pose of the second camera, wherein a strength of the correlation is based at least upon a time difference between respective image captures by the first camera and the second camera;

generating one or more constraints incorporating the correlation;

determining camera parameters using a plurality of constraints including the generated one or more constraints; and

in response to a request for the camera parameters, providing bundle adjusted information associated with the camera parameters.

18. The computer-readable medium of claim 17, wherein determining the camera parameters determining a pose adjustment for the first camera.

19. The computer-readable medium of claim 18, further comprising the operation of determining a three dimensional location of one or more features corresponding to the determined pose adjustment for the first camera.

20. The computer-readable medium of claim 17, wherein determining the correlation comprises determining the strength of the correlation, wherein the strength is a decreasing function of a duration of the time difference.