Pose data via user interaction

• US 8,884,950 B1
• Filed: 07/29/2011
• Issued: 11/11/2014
• Est. Priority Date: 07/29/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Brown, Brian Gammon, Fan, Zhe, Reinhardt, Tilman
• Primary Examiner(s): Nguyen, Kimbinh T
• Application Number:
  US13/194,177
• Time in Patent Office: 1,201 Days
• Field of Search: 345/419, 345/420, 345/619
• US Class Current: 345/419
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30184 Infrastructure
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

Embodiments improve pose data for images via user interaction. A user may model a building in a web browser plug in by mapping positions on two-dimensional images to a three-dimensional model of a building shown in the image. Each of the images may have associated original pose data, which includes information related to a position and orientation of a camera which took the image. Upon modeling the building, pose data may be adjusted to more accurately depict the building as a three-dimensional model. If the adjusted pose data is superior to the original pose data, the adjusted pose data may be associated with the image.

Claims

1. A computer implemented method of improving pose data for images, comprising:

(a) receiving, by one or more computing devices, a first user input comprising a user-selected location;

(b) providing, by the one or more computing devices, a plurality of images, each of the plurality of images showing a building at the user-selected location;

(c) providing, by the one or more computing devices, original pose data associated with each of the plurality of images, the original pose data representing at least a position and orientation of a camera that took the associated image;

(d) receiving, by the one or more computing devices, a second user input mapping a selected position on a two-dimensional image from the plurality of images to a selected feature of a three-dimensional model of the building;

(e) calculating, by the one or more computing devices, adjusted pose data for the two-dimensional image such that, when the three-dimensional model is rendered with the two-dimensional image from a perspective specified by the adjusted pose data, the feature of the three-dimensional model appears at the position of the two-dimensional image;

(f) determining, by the one or more computing devices, whether the adjusted pose data is superior to the original pose data;

(g) when the adjusted pose data is determined to be superior to the original pose data, associating, by the one or more computing devices, the adjusted pose data with the two-dimensional image; and

(h) respectively updating, by the one or more computing devices, the original pose data associated with each of the plurality of images other than the two-dimensional image based on the adjusted pose data of the two-dimensional image and information describing a respective change in pose between the two-dimensional image and each of the plurality of images other than the two-dimensional image.

2. The method of claim 1, wherein the pose data further represents a camera focal length.

3. The method of claim 1, wherein the pose data further represents a global positioning system (GPS) location.

4. The method of claim 1, wherein the original pose data comprises pose data determined by an automatic bundle adjustment process.

5. The method of claim 1, wherein the second user input includes constraint points for the selected feature of the three-dimensional model of the building.

6. The method of claim 1, wherein the provided plurality of images comprise one or more of an oblique aerial photograph of the Earth, a panoramic photograph taken from street-level, or a photograph inputted by a user.

7. A system for improving pose data for images, comprising:

one or more processors;

a modeling module implemented by the one or more processors, wherein the modeling module;

displays a plurality of images, each of the plurality of images showing a building at a user-selected location, wherein each of the plurality of images is associated with original pose data, the original pose data representing at least a position and orientation of a camera that took the associated image, and

receives a user input mapping a selected position on a two-dimensional image from the plurality of images to a selected feature of a three-dimensional model of the building;

a pose data adjustment module implemented by the one or more processors, wherein the pose data adjustment module calculates adjusted pose data for the two-dimensional image such that, when the three-dimensional model is rendered with the two-dimensional image from a perspective specified by the adjusted pose data, the feature of the three-dimensional model appears at the position of the two dimensional image;

a quality assurance module implemented by the one or more processors, wherein the quality assurance module;

determines whether the adjusted pose data is superior to the original pose data, and

when the adjusted pose data is determined to be superior to the original pose data, associates the adjusted pose data with the two-dimensional image; and

a bundle adjustment module implemented by the one or more processors, wherein the bundle adjustment module respectively updates the original pose data associated with each of the plurality of images other than the two-dimensional image based on the adjusted pose data of the two-dimensional image and information describing a respective change in pose between the two-dimensional image and each of the plurality of images other than the two-dimensional image.

8. The system of claim 7, wherein the pose data further represents a camera focal length.

9. The system of claim 7, wherein the pose data further represents a global positioning system (GPS) location.

10. The system of claim 7, wherein the original pose data comprises pose data determined by an automatic bundle adjustment process.

11. The system of claim 7, wherein the second user input further comprises constraint points for the selected feature of the three-dimensional model of the building.

12. The system of claim 7, wherein the provided plurality of images comprise one or more of an oblique aerial photograph of the Earth, a panoramic photograph taken from street-level, or a photograph inputted by a user.

13. A non-transitory, computer readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform operations including:

(a) receiving a first user input comprising a user-selected location;

(b) providing a plurality of images, each of the plurality of images showing a building at the user-selected location;

(c) providing original pose data associated with each of the plurality of images, the original pose data representing at least a position and orientation of a camera that took the associated image;

(d) receiving a second user input mapping a selected position on a two-dimensional image from the plurality of images to a selected feature of a three-dimensional model of the building;

(e) calculating adjusted pose data for the two-dimensional image such that, when the three-dimensional model is rendered with the two-dimensional image from a perspective specified by the adjusted pose data, the feature of the three-dimensional model appears at the position of the two-dimensional image;

(f) determining whether the adjusted pose data is superior to the original pose data;

(g) when the adjusted pose data is determined to be superior to the original pose data, associating the adjusted pose data with the two-dimensional image; and

(h) respectively updating the original pose data associated with each of the plurality of images other than the two-dimensional image based on the adjusted pose data of the two-dimensional image and information describing a respective change in pose between the two-dimensional image and each of the plurality of images other than the two-dimensional image.

14. The non-transitory, computer readable storage medium of claim 13, wherein the pose data comprises one or more of a camera position, a camera orientation, or a camera focal length.

15. The non-transitory, computer readable storage medium of claim 13, wherein the pose data comprises a global positioning system (GPS) location.

16. The non-transitory, computer readable storage medium of claim 13, wherein the original pose data comprises pose data determined by an automatic bundle adjustment process.

17. The non-transitory, computer readable storage medium of claim 13, wherein the second user input further comprises constraint points for the one or more three-dimensional primitive shapes.

18. The non-transitory, computer readable storage medium of claim 13, wherein the provided plurality of images comprise one or more of an oblique aerial photograph of the Earth, a panoramic photograph taken from street-level, or a photograph inputted by a user.

19. A computer implemented method of improving pose data for images, comprising:

(a) receiving, by one or more computing devices, a first user input comprising a user-selected location;

(b) providing, by the one or more computing devices, a plurality of images, each of the plurality of images showing a building at the user-selected location, wherein each of the plurality of images is associated with original pose data;

(c) receiving, by the one or more computing devices, adjusted pose data for a two-dimensional image in the plurality of images, wherein the adjusted pose data is calculated based on a mapping of a position on the two-dimensional image to a feature of a three-dimensional model of the building, such that when the three-dimensional model is rendered with the two-dimensional image from a perspective specified by the adjusted pose data, the feature of the three-dimensional model appears at the position of the two-dimensional image;

(d) determining, by the one or more computing devices, whether the adjusted pose data is superior to the original pose data;

(e) when the adjusted pose data is determined to be superior to the original pose data, associating, by the one or more computing devices, the adjusted pose data with the two-dimensional image; and

(f) receiving, by the one or more computing devices, updated pose data for each of plurality of images other than the two-dimensional image, the original nose data for each of the plurality of images having been respectively updated based on the adjusted pose data of the two-dimensional image and information describing a respective change in pose between the two-dimensional image and each of the plurality of images other than the two-dimensional image.