Three-Dimensional Annotations for Street View Data

• US 20090179895A1
• Filed: 01/15/2008
• Published: 07/16/2009
• Est. Priority Date: 01/15/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Vincent, Luc, Filip, Daniel
• Granted Patent: US 8,072,448 B2
• Time in Patent Office: Days
• US Class Current: 345/424
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 2219/004 Annotating, labelling
• CPC Class Codes: G06T 7/75 involving models

Abstract

The present invention relates to annotating images. In an embodiment, the present invention enables users to create annotations corresponding to three-dimensional objects while viewing two-dimensional images. In one embodiment, this is achieved by projecting a selecting object onto a three-dimensional model created from a plurality of two-dimensional images. The selecting object is input by a user while viewing a first image corresponding to a portion of the three-dimensional model. A location corresponding to the projection on the three-dimensional model is determined, and content entered by the user while viewing the first image is associated with the location. The content is stored together with the location information to form an annotation. The annotation can be retrieved and displayed together with other images corresponding to the location.

Claims

1. A method for creating and displaying annotations, comprising:

(1) projecting a selecting object onto a three-dimensional model, the selecting object being input by a user while viewing a first image corresponding to a portion of the three-dimensional model;

(2) determining a location corresponding to the projection on the three-dimensional model;

(3) associating the location with content entered by the user while viewing the first image;

(4) storing the content with the location to form an annotation; and

(5) displaying the annotation with a second image corresponding to the location.

2. The method of claim 1, wherein the first image is a panoramic image taken from street level.

3. The method of claim 1, wherein (5) comprises displaying the annotation with a panoramic image.

4. The method of claim 1, wherein (5) comprises displaying the annotation with a map.

5. The method of claim 1, further comprising:

(6) creating the three-dimensional model from a plurality of two-dimensional images.

6. The method of claim 5, wherein (6) comprises:

(a) identifying a first plurality of features of a third image and a second plurality of features of a fourth image;

(b) determining a plurality of pairs of features, wherein each pair of features includes a first feature, from the first plurality of features, and a second feature, from the second plurality of features, that is similar to the first feature;

(c) determining a plurality of points corresponding to the plurality of pairs of features; and

(d) determining a facade plane based on the plurality of points.

7. The method of claim 6, wherein (a) comprises using a Speeded Up Robust Features (SURF) algorithm.

8. The method of claim 6, wherein (b) comprises:

(i) determining a spill tree for the first plurality of features;

(ii) searching the spill tree for an approximate nearest neighbor and an approximate second nearest neighbor of each feature in the second image; and

(iii) comparing a feature similarity ratio of the approximate nearest neighbor and the approximate second-nearest neighbor to a threshold.

9. The method of claim 8, wherein the threshold is between 0.5 and 0.95 inclusive.

10. The method of claim 6, wherein (c) comprises:

(i) determining, for each pair of features of the plurality of pairs of features, a first ray extending from a first camera viewpoint of the third image through the first feature from the pair and a second ray extending from a second camera viewpoint of the fourth image through the second feature from the pair; and

(ii) determining, for each pair of features of the plurality of pairs of features, a point corresponding to an intersection of the first ray and the second ray.

11. A system for creating and displaying annotations, comprising:

a server that includes an annotation generator, wherein the annotation generator creates a first annotation by

projecting a selecting object, input by a user while viewing a first image, onto a three-dimensional model corresponding to a plurality of two-dimensional images,

determining a location corresponding to the projection on the three-dimensional model, and

associating the location with content input by the user; and

an annotation database, coupled to the server, that stores annotations created by the annotation generator,

wherein the server retrieves the first annotation from the annotation database and forwards the first annotation to a browser for display.

12. The system of claim 11, wherein the first image is a panoramic image taken from street level.

13. The system of claim 11, wherein the server forwards the first annotation to the browser for display with a second image.

14. The system of claim 11, wherein the server forwards the first annotation to the browser for display with a map.

15. A system for creating and displaying annotations, comprising:

an annotation editor that enables a user to input a selecting object and content for an annotation while viewing a first image;

an annotation generator that creates a first annotation by

projecting the selecting object onto a three-dimensional model corresponding to a plurality of two-dimensional images,

determining a location corresponding to the projection on the three-dimensional model, and

associating the location with the content; and

an annotation viewer that displays the first annotation with a second image.

16. The system of claim 16, further comprising:

a feature extractor that identifies a first plurality of features of a third image and a second plurality of features of a fourth image;

a feature matcher that determines a plurality of pairs of features, wherein each pair of features includes a first feature from the first plurality of features and a second feature from the second plurality of features, and wherein the first feature is similar to the second feature;

a point calculator that determines a plurality of points corresponding to the plurality of pairs of features; and

a surface estimator that creates the three-dimensional model based on the plurality of points.

17. The system of claim 16, wherein the feature extractor uses a Speeded Up Robust Features algorithm.

18. The system of claim 16, wherein the feature matcher determines a spill tree for the first plurality of features, searches the spill tree for an approximate nearest neighbor and an approximate second nearest neighbor of each feature in the fourth image, and determines whether a feature similarity ratio of the approximate nearest neighbor and the approximate second-nearest neighbor is below a threshold.

19. The system of claim 18, wherein the threshold of the feature similarity ratio is between 0.5 and 0.95 inclusive.

20. The system of claim 16, wherein the point calculator determines, for each pair of features of the plurality of pairs of features, a first ray extending from a first camera viewpoint of the third image through the first feature from the pair, a second ray extending from a second camera viewpoint of the fourth image through the second feature from the pair, and a point in three-dimensional space corresponding to an intersection of the first ray and the second ray.