Automatic discovery of popular landmarks

• US 10,289,643 B2
• Filed: 10/03/2016
• Issued: 05/14/2019
• Est. Priority Date: 05/12/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Brucher, Fernando A., Buddemeier, Ulrich, Adam, Hartwig, Neven, Hartmut
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/284,075
• Publication Number:
  US 20170024415A1
• Time in Patent Office: 953 Days
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

In one embodiment the present invention is a method for populating and updating a database of images of landmarks including geo-clustering geo-tagged images according to geographic proximity to generate one or more geo-clusters, and visual-clustering the one or more geo-clusters according to image similarity to generate one or more visual clusters. In another embodiment, the present invention is a system for identifying landmarks from digital images, including the following components: a database of geo-tagged images; a landmark database; a geo-clustering module; and a visual clustering module. In other embodiments the present invention may be a method of enhancing user queries to retrieve images of landmarks, or a method of automatically tagging a new digital image with text labels.

Claims

1. A computer-implemented method comprising:

receiving, by one or more processors, a user query;

identifying one or more trigger words in the user query;

selecting one or more tags from a landmark database, the tags corresponding to the one or more trigger words;

supplementing the user query with the one or more tags to generate a supplemented user query that describe a landmark;

in response to receiving the supplemented user query, identifying a plurality of visual clusters from the landmark database wherein the plurality of visual clusters are associated with a landmark based on the supplemented user query;

causing a user interface to be displayed, wherein the user interface includes the plurality of visual clusters;

receiving user input wherein the user input indicates that a first visual cluster of the plurality of visual clusters and a second visual cluster of the plurality of visual clusters are to be merged, wherein the second visual cluster is different than the first visual cluster; and

in response to receiving the user input, updating the landmark database to merge the first visual cluster and the second visual cluster.

2. The method of claim 1, wherein the user input further indicates that a third visual cluster of the plurality of visual clusters is to be disassociated from the landmark;

and wherein updating the landmark database comprises disassociating the third visual cluster from the landmark in the landmark database.

3. The method of claim 1, wherein the user interface includes a user input graphic enabled to receive the user input.

4. The method of claim 1, wherein the user interface includes a plurality of user input graphics, wherein each of the plurality of user input graphics is associated with a respective visual cluster of the plurality of visual clusters, and is configured to receive the user input.

5. The method of claim 1, wherein the user interface includes a plurality of landmarks and one or more corresponding visual clusters.

6. The method of claim 1, wherein the user interface displays descriptive information comprising:

a number of images;

a popularity of the landmark wherein the popularity is based on a number of one or more authors that have contributed images to the plurality of visual clusters;

an indication that one or more of the plurality of visual clusters have been modified by a user; and

an indication that one or more of the plurality of visual clusters have been verified by the user.

7. The method of claim 1, further comprising:

wherein the user input further indicates that one or more landmarks associated with the plurality of visual clusters are to be merged.

8. A system comprising:

one or more processors; and

logic encoded in one or more tangible media for execution by the one or more processors and when executed operable to perform operations comprising;

receiving a user query;

identifying one or more trigger words in the user query;

selecting one or more tags from a landmark database, the tags corresponding to the one or more trigger words;

supplementing the user query with the one or more tags to generate a supplemented user query that describe a landmark;

in response to receiving the supplemented user query, identifying a plurality of visual clusters from the landmark database wherein the plurality of visual clusters are associated with a landmark based on the supplemented user query;

causing a user interface to be displayed, wherein the user interface includes the plurality of visual clusters;

receiving user input wherein the user input indicates that a first visual cluster of the plurality of visual clusters and a second visual cluster of the plurality of visual clusters to be merged, wherein the second visual cluster is different than the first visual cluster; and

in response to receiving the user input, updating the landmark database to merge the first visual cluster and the second visual cluster.

9. The system of claim 8, wherein the user input further indicates that a third visual cluster of the plurality of visual clusters is to be disassociated from the landmark;

and wherein updating the landmark database comprises disassociating the third visual cluster from the landmark in the landmark database.

10. The system of claim 8, wherein the user interface includes a user input graphic enabled to receive the user input.

11. The system of claim 8, wherein the user interface includes a plurality of user input graphics, wherein each of the plurality of user input graphics is associated with a respective visual cluster of the plurality of visual clusters, and is configured to receive the user input.

12. The system of claim 8, wherein the user interface includes a plurality of landmarks and one or more corresponding visual clusters.

13. The system of claim 8, further comprising applying one or more tags to the updated landmark database.

14. The system of claim 8, further comprising:

wherein the user input further indicates that one or more landmarks associated with the plurality of visual clusters are to be merged.

15. A non-transitory computer readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:

receiving a user query;

identifying one or more trigger words in the user query;

selecting one or more tags from a landmark database, the tags corresponding to the one or more trigger words;

supplementing the user query with the one or more tags to generate a supplemented user query that describe a landmark;

in response to receiving the supplemented user query, identifying a plurality of visual clusters from the landmark database wherein the plurality of visual clusters are associated with a landmark based on the supplemented user query;

causing a user interface to be displayed, wherein the user interface includes the plurality of visual clusters;

receiving user input wherein the user input indicates that a first visual cluster of the plurality of visual clusters and a second visual cluster of the plurality of visual clusters are to be merged; and

in response to receiving the user input, updating the landmark database to merge the first visual cluster and the second visual cluster.

16. The non-transitory computer readable medium of claim 15, wherein the user input further indicates that a third visual cluster of the plurality of visual clusters is to be disassociated from the landmark;

and wherein updating the landmark database comprises disassociating the third visual cluster from the landmark in the landmark database.

17. The non-transitory computer readable medium of claim 15, wherein the user interface includes a user input graphic enabled to receive the user input.

18. The non-transitory computer readable medium of claim 15, wherein the user interface includes a plurality of user input graphics, wherein each of the plurality of user input graphics is associated with a respective visual cluster of the plurality of visual clusters, and is configured to receive the user input.

19. The non-transitory computer readable medium of claim 15, wherein the user interface includes a plurality of landmarks and one or more corresponding visual clusters.

20. The non-transitory computer readable medium of claim 15, wherein the user interface displays descriptive information comprising:

a number of images;

a popularity of the landmark wherein the popularity is based on a number of one or more authors that have contributed images to the plurality of visual clusters;

an indication that one or more of the plurality of visual clusters have been modified by a user; and

an indication that one or more of the plurality of visual clusters have been verified by the user.

21. The non-transitory computer readable medium of claim 15, further comprising:

receiving text input from one or more users wherein the text input includes one or more new text labels to be assigned to a merged visual cluster of the updated landmark database formed by merging the first visual cluster and the second visual cluster.