Landmarks from digital photo collections

• US 9,020,247 B2
• Filed: 02/05/2013
• Issued: 04/28/2015
• Est. Priority Date: 05/15/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Zhang, Li
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US13/759,916
• Publication Number:
  US 20130202198A1
• Time in Patent Office: 812 Days
• Field of Search: 382/159, 382/160, 382/103, 382/203, 382/305, 707/711
• US Class Current: 382/160
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

Methods and systems for automatic detection of landmarks in digital images and annotation of those images are disclosed. A method for detecting and annotating landmarks in digital images includes the steps of automatically assigning a tag descriptive of a landmark to one or more images in a plurality of text-associated digital images to generate a set of landmark-tagged images, learning an appearance model for the landmark from the set of landmark-tagged images, and detecting the landmark in a new digital image using the appearance model. The method can also include a step of annotating the new image with the tag descriptive of the landmark.

Claims

1. A method for detecting and annotating landmarks in digital images:

(a) automatically assigning, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the text tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored by assigning correlation-weights to the plurality of text-associated digital images, and wherein the landmark identifies an object in or feature marking a geographic point or geographic area;

(b) learning an appearance model for the landmark from the set of landmark-tagged images; and

(c) detecting the landmark in a new image using the appearance model,

wherein said stages (a)-(c) are performed by at least one processor.

2. The method of claim 1, further comprising:

(d) annotating the new image with the tag descriptive of the landmark.

3. The method of claim 1, wherein stage (a) comprises:

(i) generating a list of landmark n-grams from the plurality of text-associated digital images;

(ii) computing an n-gram score for each landmark n-gram in an n-gram set, wherein the n-gram set is a subset of the list of landmark n-grams; and

(iii) assigning a tag descriptive of a landmark, wherein the tag descriptive of the landmark is based on at least one landmark n-gram in said n-gram set.

4. The method of claim 3, wherein stage (a)(i) comprises:

electronically accessing the plurality of text-associated digital images; and

retrieving at least one of said landmark n-grams from a text associated with an image in the plurality of text-associated digital images.

5. The method of claim 3, wherein stage (a)(ii) comprises:

assigning said correlation weights to the plurality of text-associated digital images, wherein each image in the plurality of text-associated digital images is assigned a correlation weight based on correlation of metadata of said each image with other images in the plurality of text-associated digital images, and wherein the plurality of text-associated digital images includes the set of text-associated images;

generating a matching-images graph from the plurality of text-associated digital images, wherein each edge in the matching-images graph is assigned a match score representing a level of match between two images connected by said each edge; and

linking said landmark n-grams to images in the plurality of text-associated digital images based upon said correlation-weights and the matching images graph to generate links between landmark n-grams and images in the plurality of text-associated digital images.

6. The method of claim 5, wherein stage (a)(ii) further comprises:

estimating a geo-reliability score for each image of the plurality of text-associated digital images using the matching-images graph, wherein the geo-reliability score is an estimation of accuracy of geo-location information of said each image based on a comparison of visual consistency of other images with geo-location coordinates within a predetermined distance to said each image.

7. The method of claim 6, wherein stage (a)(ii) further comprises:

computing a variance of geo-location for a landmark n-gram of said n-gram set, wherein the variance is based on geo-locations of images linked to said landmark n-gram in said matching-images graph; and

removing from said n-gram set any landmark n-grams having a variance of geo-location exceeding a predetermined threshold.

8. The method of claim 5, wherein the n-gram score is based on the matching-images graph.

9. The method of claim 8, wherein the n-gram score is computed as a ratio of strength of internal edges of said matching-images graph and the strength of external edges of said matching images graph, wherein an internal edge exists between images having at least one common landmark n-gram, and wherein an external edge exists between images not having at least one common landmark n-gram.

10. The method of claim 5, wherein stage (a)(i) further comprises:

merging two or more landmark n-grams in said n-gram set based at least on one of,

a similarity of the score of said two or more landmark n-grams, and

an overlap of images having said two or more landmark n-grams in linked landmark n-grams.

11. The method of claim 5, wherein the metadata includes information relating to at least one of,

an author,

a geo-location, and

a time of origin.

12. The method of claim 5, wherein each link in the matching-graph represents matching feature descriptors between two images of the plurality of text-associated digital images.

13. A system for automatically detecting and annotating landmarks in digital images, comprising:

at least one collection of text-associated digital images stored in a memory medium; and

at least one processor communicatively coupled to said medium, the at least one processor configured to;

automatically assign, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the text tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored by assigning correlation-weights to the plurality of text-associated digital images, and wherein the landmark identifies an object in or feature marking a geographic point or geographic area;

learn an appearance model for the landmark from the set of landmark-tagged images; and

detect the landmark in a new image using the appearance model.

14. The system of claim 13, wherein the at least one processor is further configured to:

annotate the new image with the tag descriptive of the landmark.

15. The system of claim 13, wherein the at least one processor is further configured to:

generate a list of landmark n-grams from the plurality of text-associated digital images;

compute an n-gram score for each landmark n-gram in an n-gram set, wherein the n-gram set is a subset of the list of landmark n-grams; and

assign a tag descriptive of a landmark, wherein the tag descriptive of the landmark is based on at least one landmark n-gram in said n-gram set.

16. The system of claim 15, wherein the at least one processor is further configured to:

assigning said correlation weights to the plurality of text-associated digital images, wherein each image in the plurality of text-associated digital images is assigned a correlation weight based on correlation of metadata of said each image with other images in the plurality of text-associated digital images, and wherein the plurality of text-associated digital images includes the set of text-associated images;

generate a matching-images graph from the plurality of text-associated digital images, wherein each edge in the matching-images graph is assigned a match score representing a level of match between two images connected by said each edge; and

link said landmark n-grams to images in the plurality of text-associated digital images based upon said correlation-weights and the matching images graph to generate links between landmark n-grams and images in the plurality of text-associated digital images.

17. A non-transitory computer readable storage medium having computer program logic recorded thereon for enabling a processor to name images, said computer program logic comprising:

a first module configured to enable the processor to assign, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the text tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored by assigning correlation-weights to the plurality of text-associated digital images, and wherein the landmark identifies an object in or feature marking a geographic point or geographic area;

a second module configured to enable the processor to learn an appearance model for the landmark from the set of landmark-tagged images;

a third module configured to enable the processor to detect the landmark in a new image using the appearance model.

18. The non-transitory computer readable storage medium of claim 17, further comprising:

a fourth module configured to enable the processor to annotate the new image with the tag descriptive of the landmark.

19. The non-transitory computer readable storage medium of claim 17, wherein the first module is further configured to:

generate a list of landmark n-grams from the plurality of text-associated digital images;

compute an n-gram score for each landmark n-gram in an n-gram set, wherein the n-gram set is a subset of the list of landmark n-grams; and

assign a tag descriptive of a landmark, wherein the tag descriptive of the landmark is based on at least one landmark n-gram in said n-gram set.

20. The non-transitory computer readable storage medium of claim 19, wherein the first module is further configured to:

assigning said correlation weights to the plurality of text-associated digital images, wherein each image in the plurality of text-associated digital images is assigned a correlation-weight based on correlation of metadata of said each image with other images in the plurality of text-associated digital images, and wherein the plurality of text-associated digital images includes the set of text-associated images;

generate a matching-images graph from the plurality of text-associated digital images, wherein each edge in the matching images graph is assigned a match score representing a level of match between two images connected by said each edge; and

link said landmark n-grams to images in the plurality of text-associated digital images based upon said correlation-weights and the matching images graph to generate links between landmark n-grams and images in the plurality of text-associated digital images.