Landmarks from digital photo collections

• US 10,303,975 B2
• Filed: 07/30/2017
• Issued: 05/28/2019
• Est. Priority Date: 05/15/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Adam, Hartwig, Zhang, Li
• Primary Examiner(s): Dulaney, Kathleen Y
• Application Number:
  US15/663,796
• Publication Number:
  US 20180211134A1
• Time in Patent Office: 667 Days
• Field of Search: 382159, 382160, 382103, 382203, 382305, 707711
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

Methods and systems for automatic detection of landmarks in digital images and annotation of those images are disclosed. A method for detecting and annotating landmarks in digital images includes the steps of automatically assigning a tag descriptive of a landmark to one or more images in a plurality of text-associated digital images to generate a set of landmark-tagged images, learning an appearance model for the landmark from the set of landmark-tagged images, and detecting the landmark in a new digital image using the appearance model. The method can also include a step of annotating the new image with the tag descriptive of the landmark.

Claims

1. A method for detecting and annotating landmarks in digital images:

electronically accessing a plurality of digital images in an image collection;

retrieving an n-gram set generated from one or more texts associated with one or more images in the plurality of digital images;

choosing, from the n-gram set, one or more scored n-grams having at least a minimum reliability measure that is based on a number of unique authors of the one or more texts;

automatically assigning, to at least one of the one or more images in the image collection, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the tag is based upon the one or more chosen scored n-grams from the one or more texts associated with the at least one of the one or more images, wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;

learning an appearance model for the landmark from the set of landmark-tagged images; and

detecting the landmark in a new image that is not part of the image collection using the appearance model, wherein the method is performed by at least one processor.

2. The method of claim 1, further comprising annotating the new image with the tag descriptive of the landmark.

3. The method of claim 1, further comprising:

assigning correlation-weights to the plurality of digital images, wherein each image in the plurality of digital images is assigned a correlation-weight based on correlation of metadata of each image with other images in the plurality of digital images;

generating a matching-images graph from the plurality of digital images, wherein each edge in the matching-images graph is assigned a match score representing a level of match between two images connected by each edge; and

linking said scored n-grams to images in the plurality of digital images based upon said correlation-weights and the matching-images graph to generate links between scored n-grams and images in the plurality of digital images.

4. The method of claim 3, further comprising:

estimating a geo-reliability score for each image of the plurality of digital images using the matching-images graph, wherein the geo-reliability score is an estimation of accuracy of geo-location information of each image based on a comparison of visual consistency of other images with geo-location coordinates within a predetermined distance to each image.

5. The method of claim 4, further comprising:

computing a variance of geo-location for a scored n-gram of said n-gram set, wherein the variance is based on geo-locations of images linked to said scored n-gram in said matching-images graph; and

removing from said n-gram set any scored n-grams having a variance of geo-location exceeding a predetermined threshold.

6. The method of claim 3, wherein an n-gram score of each scored n-gram in the n-gram set is based on the matching-images graph.

7. The method of claim 6, wherein the n-gram score of each scored n-gram in the n-gram set is computed as a ratio of strength of internal edges of said matching-images graph and strength of external edges of said matching-images graph, wherein an internal edge exists between images having at least one common scored n-gram, and wherein an external edge exists between images not having at least one common scored n-gram.

8. The method of claim 3, further comprising merging two or more scored n-grams in said n-gram set.

9. The method of claim 8, wherein the merging is based at least on one of,

a similarity between respective scores of the two or more scored n-grams, and

an overlap of images having the two or more scored n-grams in linked scored n-grams.

10. The method of claim 3, wherein the metadata includes information relating to at least one of:

an author,

a geo-location, or

a time of origin.

11. The method of claim 3, wherein each link in the matching-images graph represents matching feature descriptors between two images of the plurality of digital images.

12. The method of claim 1, wherein the appearance model includes visual information and non-visual information, and wherein the detecting the landmark includes using the visual information and the non-visual information to identify the landmark.

13. The method of claim 1, wherein an n-gram score of each scored n-gram in the n-gram set is based upon a level of match between images associated with each scored n-gram and a level of correlation between each image associated with the landmark with other images associated with the landmark, and wherein each scored n-gram is associated with one or more images having associated text from which each scored n-gram was obtained.

14. A method for detecting and annotating landmarks in digital images:

automatically assigning, to one or more images in an image collection that includes a plurality of digital images, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;

learning an appearance model for the landmark from the set of landmark-tagged images wherein the appearance model includes visual information and non-visual information; and

detecting the landmark in a new image that is not part of the image collection using the appearance model, wherein the detecting the landmark includes using the visual information and the non-visual information to identify the landmark, and wherein the method is performed by at least one processor.

15. A method for detecting and annotating landmarks in digital images:

automatically assigning, to one or more images in an image collection that includes a plurality of digital images, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark,

wherein the tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein an n-gram score of each of the one or more scored n-grams is based upon a level of match between images associated with each scored n-gram and a level of correlation between each image associated with the landmark with other images associated with the landmark, and wherein each scored n-gram is associated with one or more images having associated text from which each scored n-gram was obtained,

wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;

learning an appearance model for the landmark from the set of landmark-tagged images; and

detecting the landmark in a new image that is not part of the image collection using the appearance model, wherein the method is performed by at least one processor.