Automatic Discovery of Popular Landmarks

• US 20090279794A1
• Filed: 05/12/2008
• Published: 11/12/2009
• Est. Priority Date: 05/12/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Buddemeier, Ulrich, Brucher, Fernando A., Neven, Hartmut
• Granted Patent: US 8,676,001 B2
• Time in Patent Office: Days
• US Class Current: 382/225
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

In one embodiment the present invention is a method for populating and updating a database of images of landmarks including geo-clustering geo-tagged images according to geographic proximity to generate one or more geo-clusters, and visual-clustering the one or more geo-clusters according to image similarity to generate one or more visual clusters. In another embodiment, the present invention is a system for identifying landmarks from digital images, including the following components: a database of geo-tagged images; a landmark database; a geo-clustering module; and a visual clustering module. In other embodiments the present invention may be a method of enhancing user queries to retrieve images of landmarks, or a method of automatically tagging a new digital image with text labels.

Claims

1. A method for populating and updating a database of images of landmarks comprising:

(a) geo-clustering geo-tagged images according to geographic proximity to generate one or more geo-clusters; and

(b) visual-clustering the one or more geo-clusters according to image similarity to generate one or more visual clusters.

2. The method of claim 1, wherein said geo-clustering comprises validating the one or more geo-clusters.

3. The method of claim 2, wherein said validating includes selecting the one or more geo-clusters having at least a predefined number of associated unique user identifiers.

4. The method of claim 1, wherein said visual-clustering includes selecting visual clusters based on a region graph.

5. The method of claim 4, wherein the region graph is generated based on matching images in a geo-cluster.

6. The method of claim 1, wherein said visual-clustering includes generating a text label for at least one visual cluster.

7. The method of claim 6, wherein the text label for at least one visual cluster is based on text labels of individual images in the at least one visual cluster.

8. The method of claim 6, wherein the text label for at least one visual cluster is based on a text label assigned previously to a prior visual cluster, and wherein the prior visual cluster is a cluster having a user assigned text label.

9. The method of claim 1, further comprising:

(c) receiving external data; and

(d) processing visual clusters based on the external data.

10. The method of claim 9, wherein the external data includes text tags.

11. The method of claim 9, wherein the external data includes user input.

12. The method of claim 1 further comprising:

(e) storing of visual clusters.

13. A system for identifying landmarks from digital images, comprising:

(a) a database of geo-tagged images;

(b) a landmark database;

(c) a geo-clustering module in communication with said database of geo-tagged images, wherein the geo-tagged images are grouped into one or more geo-clusters; and

(d) a visual clustering module in communication with said geo-clustering module, wherein the one or more geo-clusters are grouped into one or more visual clusters, and wherein visual cluster data is stored in the landmark database.

14. The system of claim 13 wherein said landmark database includes images of landmarks and associated text labels.

15. The system of claim 13 further comprising:

(e) an interface to receive external data, wherein the external data includes tags for said one or more visual clusters.

16. The system of claim 15 wherein the external data further includes images for said one or more visual clusters.

17. The system of claim 15 wherein said interface is a graphical user interface.

18. The system of claim 13 wherein the visual clustering module further comprises a popularity indexing module.

19. A method of enhancing user queries to retrieve images of landmarks, comprising:

(a) receiving a user query;

(b) identifying one or more trigger words in the user query;

(c) selecting one or more corresponding tags from a landmark database corresponding to the one or more trigger words; and

(d) supplementing the user query with the one or more corresponding tags, generating a supplemented user query.

20. The method of claim 19 further comprising:

(e) retrieving images based on the supplemented user query.

21. The method of claim 20 further comprising:

(f) ordering the retrieved images according to the popularity of landmarks.

22. The method of claim 21 wherein the popularity of landmarks is based on the number of unique user identifiers associated with images having each landmark.

23. A method of automatically tagging a new digital image, comprising:

(a) comparing the new digital image to images in a landmark image database, wherein the landmark image database comprises visual clusters of images of one or more landmarks; and

(b) tagging the new digital image with at least one tag based on at least one of said visual clusters.

24. A computer program product comprising a computer usable medium having control logic stored therein for causing a computer to populate and update a database of images of landmarks, said control logic comprising:

(a) a first computer readable program code that enables the computer to cluster geo-tagged images according to geographic proximity to generate one or more geo-clusters; and

(b) a second computer readable program code that enables the computer to cluster the one or more geo-clusters according to image similarity.

25. A computer program product comprising a computer usable medium having control logic stored therein for causing a computer to enhance user queries, said control logic comprising:

(b) a first computer readable program code that enables the computer to identify one or more trigger words in the user query;

(c) a second computer readable program code that enables the computer to select one or more corresponding tags from a landmark database corresponding to the one or more trigger words; and

(d) a third computer readable program code that enables the computer to supplement the user query with the one or more corresponding tags.

26. A method of maintaining images of landmarks in an image collection, comprising:

(a) selecting a first set of images and a second set of images, wherein the first set and the second set are members of a first landmark set;

(b) displaying a first list that comprises a first list element and a second list element, wherein the first list element comprises a first descriptive data element corresponding to the first set of images and a first input, and wherein the second list element comprises a second descriptive data elements corresponding to the second set of images and a second input; and

(c) receiving user input in the first and second inputs.

27. The method of claim 26, further comprising:

(d) merging the first set and the second set based the user input.

28. The method of claim 26, further comprising:

(e) removing the first set from the first landmark set based on the user input.

29. The method of claim 26, wherein the first descriptive data elements include at least one user-navigable link.

30. The method of claim 26, wherein the selecting is based on selection criteria, and wherein the selection criteria includes user-specified selection criteria.

31. The method of claim 30, wherein the selection criteria includes popularity of landmarks.

32. A computer program product comprising a computer usable medium having control logic stored therein for causing a computer to maintain images of landmarks in an image collection, said control logic comprising:

(a) a first computer readable program code that enables the computer to select a first set of images and a second set of images wherein the first set and the second set are members of a first landmark set;

(b) a second computer readable program code that enables the computer to display a first list that comprises a first list element and a second list element, wherein the first list element comprises a first descriptive data elements corresponding to the first set of images and a first input, and wherein the second list element comprises a second descriptive data elements corresponding to the second set of images and a second input; and

(c) a third computer readable program code that enables the computer to receive user input in the first and second inputs.

33. A method of maintaining images of landmarks in an image collection, comprising:

(a) displaying at least one example image, wherein the example image is included in a first set, and wherein the first set includes images having a first landmark; and

(b) displaying one or more descriptive data groups, wherein each descriptive data group includes a related user input, and wherein each descriptive data group corresponds to one image in the first set;

34. The method of claim 33, further comprising:

(c) receiving user input, wherein user input indicates selection of a first image, and wherein the first image is one of the at least one example image; and

(d) displaying a bounded region-of-interest on the first image, wherein the bounded region-of-interest contains the first landmark.

35. The method of claim 33, further comprising:

(e) receiving user input at the related user input graphic of one or more descriptive data groups; and

(f) removing at least one image from the first set based on the user input.

36. A computer program product comprising a computer usable medium having control logic stored therein for causing a computer to maintain images of landmarks in an image collection, said control logic comprising:

(a) a first computer readable program code that enables the computer to display at least one example image, wherein the example image is included in a first set, and wherein the first set includes images having a first landmark; and

(b) a second computer readable program code that enables the computer to display one or more descriptive data groups, wherein each descriptive data group includes a related user input, and wherein each descriptive data group corresponds to one image in the first set.