Landmarks from digital photo collections

• US 9,721,188 B2
• Filed: 04/10/2015
• Issued: 08/01/2017
• Est. Priority Date: 05/15/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Zhang, Li
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US14/683,643
• Publication Number:
  US 20150213329A1
• Time in Patent Office: 844 Days
• Field of Search: 382160, 382103, 382203, 382305, 707711
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

Methods and systems for automatic detection of landmarks in digital images and annotation of those images are disclosed. A method for detecting and annotating landmarks in digital images includes the steps of automatically assigning a tag descriptive of a landmark to one or more images in a plurality of text-associated digital images to generate a set of landmark-tagged images, learning an appearance model for the landmark from the set of landmark-tagged images, and detecting the landmark in a new digital image using the appearance model. The method can also include a step of annotating the new image with the tag descriptive of the landmark.

Claims

1. A method to detect landmarks in digital images, comprising:

assigning, by one or more processors, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark feature, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored based on image-name links between the one or more n-grams and the one or more images, wherein each of the image-name links is a variable indicative of whether the one or more n-grams are included in text tags of the one or more images;

building an appearance model for the landmark feature based on the assigned text tag descriptive of the landmark feature; and

detecting the landmark feature in a different image using the appearance model.

2. The method of claim 1, further comprising annotating the different image with the text tag descriptive of the landmark feature.

3. The method of claim 1, wherein assigning the text tag descriptive of the landmark feature further comprises generating a list of the one or more n-grams.

4. The method of claim 3, wherein assigning the text tag further comprises computing an n-gram score for one or more of the one or more n-grams.

5. The method of claim 4, wherein the text tag descriptive of the landmark feature is based on one or more of the one or more n-grams.

6. The method of claim 4, wherein assigning the text tag further comprises assigning correlation weights to each of the plurality of text-associated digital images based on correlation of text associated with each image with text associated with one or more other images in the plurality of text-associated digital images.

7. The method of claim 6, further comprising generating a matching-images graph from the plurality of text associated digital images and assigning a match score to each edge in the matching-images graph, wherein the match score represents a level of match between two images connected by the each edge.

8. The method of claim 7, wherein the n-grams are scored based on the matching-images graph.

9. The method of claim 7, further comprising generating the image-name links between the one or more n-grams and the plurality of text-associated digital images based upon the correlation weights and the matching images graph.

10. The method of claim 7, further comprising estimating a geo-reliability score for one or more images in the plurality of text-associated digital images based on the matching-images graph.

11. A system to detect landmarks in digital images, comprising:

one or more processors; and

a non-transitory computer-readable medium having instructions stored thereon, the instructions executable by the one or more processors to;

assign to one or more images in a plurality of text-associated digital images a text tag descriptive of a landmark object, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images that are scored based on image-name links between the one or more n-grams and the one or more images, wherein the image-name links are variables indicative of whether the one or more n-grams are included in text tags of the one or more images;

learn an appearance model for the landmark object from the assigned text tag;

detect the landmark object in a new image using the appearance model; and

annotate the new image with the assigned text tag descriptive of the landmark object.

12. The system of claim 11, wherein the instructions further comprise instructions to generate a list of the one or more n-grams from the one or more texts associated with the one or more images.

13. The system of claim 12, wherein the instructions further comprise instructions to compute an n-gram score for one or more of the one or more n-grams from the one or more texts.

14. The system of claim 13, wherein the text tag descriptive of the landmark object is assigned to the one or more images based on the n-gram score meeting a threshold.

15. The system of claim 13, wherein the instructions further comprise instructions to assign correlation weights to the plurality of text-associated digital images based on correlation of text associated with each image with other images in the plurality of text-associated digital images.

16. The system of claim 15, wherein the instructions further comprise instructions to generate a matching-images graph from the plurality of text associated digital images and assign a match score to each edge in the images graph representing a level of match between two images connected by each edge.

17. The system of claim 16, wherein the n-grams are scored based at least in part on the matching-images graph.

18. The system of claim 16, wherein the instructions further comprise instructions to generate the image-name links between the one or more n-grams and the plurality of text-associated digital images based upon the correlation weights and the matching images graph.

19. The system of claim 16, wherein the instructions further comprise instructions to estimate a geo-reliability score for one or more images in the plurality of text-associated digital images based on the matching-images graph.

20. A method to detect landmarks in digital images, comprising:

assigning, by one or more processors, to one or more images in a plurality of text-associated digital images, a text tag descriptive of a landmark feature, wherein the text tag is based upon one or more n-grams from one or more texts associated with the one or more images, wherein the n-grams are scored based on image-name links between the one or more n-grams and the one or more images and wherein the assigning comprises assigning correlation weights to each of the plurality of text-associated digital images based on correlation of text associated with each image with text associated with one or more other images in the plurality of text-associated digital images;

building an appearance model for the landmark feature based on the assigned text tag descriptive of the landmark feature; and

detecting the landmark feature in a different image using the appearance model.