Automatic discovery of popular landmarks

• US 9,483,500 B2
• Filed: 04/06/2015
• Issued: 11/01/2016
• Est. Priority Date: 05/12/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Brucher, Fernando A., Buddemeier, Ulrich, Adam, Hartwig, Neven, Hartmut
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US14/680,000
• Publication Number:
  US 20150213057A1
• Time in Patent Office: 575 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

In one embodiment the present invention is a method for populating and updating a database of images of landmarks including geo-clustering geo-tagged images according to geographic proximity to generate one or more geo-clusters, and visual-clustering the one or more geo-clusters according to image similarity to generate one or more visual clusters. In another embodiment, the present invention is a system for identifying landmarks from digital images, including the following components: a database of geo-tagged images; a landmark database; a geo-clustering module; and a visual clustering module. In other embodiments the present invention may be a method of enhancing user queries to retrieve images of landmarks, or a method of automatically tagging a new digital image with text labels.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon to display landmark information, the instructions executable by one or more processors to cause the one or more processors to:

receive a user query;

identify one or more trigger words in the user query;

select one or more tags from a landmark database, the tags corresponding to the one or more trigger words;

supplement the user query with the one or more tags to generate a supplemented user query;

retrieve one or more visual clusters of landmark images for a landmark based on the supplemented user query; and

generate a user interface including information about the landmark and one or more summary lists of visual clusters, wherein each of the one or more summary lists includes, for each respective visual cluster of landmark images in the one or more summary lists, an identification of the respective visual cluster, and one or more of;

a number of images in the respective visual cluster, a number of unique user identifiers associated with the respective visual cluster, a number of authors of images associated with the respective visual cluster, and a link to access images in the respective visual cluster.

2. A method to display landmark information

comprising:

receiving, by one or more computers, a user query;

identifying one or more trigger words in the user query;

selecting one or more tags from a landmark database, the tags corresponding to the one or more trigger words;

supplementing the user query with the one or more tags to generate a supplemented user query;

retrieving, based on the supplemented user query, one or more visual clusters of landmark images;

generating a user interface including the one or more visual clusters of retrieved landmark images; and

causing one or more summary lists of visual clusters of landmark images to be displayed on the user interface, wherein each of the one or more summary lists includes descriptive information and access information for each visual cluster in the one or more summary lists displayed.

3. The method of claim 2, wherein the descriptive information comprises a number of images.

4. The method of claim 2, wherein the descriptive information comprises a number of unique user identifiers.

5. The method of claim 2, wherein the descriptive information comprises popularity information.

6. The method of claim 2, wherein the descriptive information comprises an unique identifier for each visual cluster in the one or more summary lists displayed.

7. The method of claim 2, wherein the access information for each respective visual cluster in the one or more summary lists displayed comprises a respective link to access images associated with the respective visual cluster.

8. The method of claim 2, further comprising ordering the visual clusters in the summary lists according to popularity criteria.

9. The method of claim 2, further comprising:

associating a landmark with the one or more visual clusters of retrieved landmark images;

determining whether there are additional visual clusters corresponding to the associated landmark to display; and

based upon a determination that there are no additional visual clusters to display, displaying information about the associated landmark.

10. The method of claim 9, wherein displaying information about the associated landmark includes displaying one or more of:

name of the associated landmark, location of the associated landmark, popularity, and number of images.

11. The method of claim 2, further comprising providing a user interface element to receive user input corresponding to each visual cluster.

12. A system to display landmark information, comprising:

one or more processors; and

a non-transitory computer-readable medium having instructions stored thereon, the instructions executable by the one or more processors to;

receive a user query comprising one or more keywords;

select one or more corresponding tags from a landmark database corresponding to the one or more keywords;

supplement the user query with the one or more corresponding tags;

retrieve one or more landmark images based on the supplemented user query;

create visual clusters from the one or more retrieved landmark images; and

generate a user interface including one or more summary lists of the created visual clusters for the one or more retrieved landmark images, wherein each summary list corresponds to one of the one or more retrieved landmarks and includes descriptive information and access information for each visual cluster included in the summary list.

13. The system of claim 12, wherein the descriptive information comprises one of a number of images and a number of unique user identifiers.

14. The system of claim 12, wherein the descriptive information comprises popularity information.

15. The system of claim 12, wherein the descriptive information comprises an identifier for one of the visual clusters included in the summary list.

16. The system of claim 12, wherein the access information comprises, for each respective visual cluster included in the summary list, a respective link to access images in the respective visual cluster.

17. The system of claim 12, wherein the instructions are further executable by the one or more processors to evaluate each cluster based on popularity criteria and to order each cluster based on the evaluation.

18. The system of claim 12, wherein the instructions are further executable by the one or more processors to determine whether there are more visual clusters to display, and upon determination that there are no more visual clusters to display corresponding to the one of the one or more retrieved landmarks, display information about the one of the one or more retrieved landmarks.

19. The system of claim 18, wherein the information about the one of the one or more retrieved landmarks includes one or more of:

name of the one landmark, location of the one landmark, popularity of the one landmark, and number of images associated with the one of the one or more retrieved landmarks.

20. The system of claim 12, wherein the instructions are further executable by the one or more processors to provide a checkbox to receive user input corresponding to each visual cluster.