Determining location information for images using landmark, caption, and metadata location data

• US 9,069,794 B1
• Filed: 10/11/2013
• Issued: 06/30/2015
• Est. Priority Date: 10/11/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Adam, Hartwig, Flynn, John, Preetham, Arcot J., Bandukwala, Farhana, Sikora, Krzysztof, Andreetto, Marco
• Primary Examiner(s): Jalil, Neveen Abel
• Assistant Examiner(s): Levitian, Karina
• Application Number:
  US14/052,157
• Time in Patent Office: 627 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location

Abstract

Methods, systems, and apparatus are provided for determining location information for images. In one aspect, a method includes obtaining landmark location data from content depicted in an image and corresponding confidence scores. Also, the method includes obtaining caption location data from user input and corresponding confidence scores, and obtaining metadata location data from data provided by an image capturing device. Further, the method includes identifying location pairs from the landmark, caption, and metadata location data, and generating, for each location pair, a geographic consistency score. Additionally, the method includes selecting a location pair based on the geographic consistency scores, and selecting an image location for the image from the selected location pair. Moreover, the method includes determining an image location score based on a confidence score for one of the locations in the selected location pair, and associating the image location and image location score with the image.

Claims

1. A method implemented by data processing apparatus, the method comprising:

obtaining landmark location data for an image, the landmark location data specifying one or more landmark locations derived from content depicted in the image and, for each landmark location, a landmark confidence score that indicates a measure of confidence that the image depicts content associated with the landmark location;

obtaining caption location data for the image, the caption location data specifying one or more caption locations derived from user input and, for each caption location, a caption confidence score that indicates a measure of confidence that the image depicts content associated with the caption location;

obtaining metadata location data for the image, the metadata location data specifying a metadata location derived from data provided by an image capturing device that captured the image;

identifying location pairs from the landmark location data, caption location data, and metadata location data, each location pair including two of a landmark location, caption location, and metadata location;

generating, for each location pair, a geographic consistency score that indicates a measure of consistency between locations specified in the location pair;

ranking each location pair according to a predetermined hierarchy of location types, the predetermined hierarchy specifying a first priority for landmark locations, a second priority for caption locations, and third priority for metadata locations, wherein the ranking of each location air is based on a highest priority of the locations specified in the location pair;

selecting a location pair based at least in part on the geographic consistency scores and location pair rankings;

selecting an image location for the image from the locations specified in the selected location pair;

determining an image location score for the image location based on a confidence score for one of the locations specified in the selected location pair; and

associating the image location and image location score with the image.

2. The method of claim 1, further comprising:

identifying a confidence score for a location specified in the selected location pair;

identifying a location type for the location; and

generating a topicality score for the image based on the identified confidence score and the identified location type.

3. The method of claim 2, wherein identifying a confidence score for a location specified in the selected location pair comprises identifying a confidence score for a location specified in the selected location pair based on the pre-determined hierarchy of location types.

4. The method of claim 1, wherein selecting an image location for the image from the locations specified in the selected location pair comprises selecting an image location for the image from the locations specified in the selected location pair based on the pre-determined hierarchy of location types.

5. The method of claim 2, wherein generating a topicality score for the image comprises:

inputting the identified confidence score into a function that corresponds to the identified location type, wherein the function is one of three different functions that each correspond to a different location type; and

obtaining the topicality score for the image as output from the function.

6. The method of claim 1, wherein:

the landmark data is obtained from a landmark processor that i) identifies landmark images that are visually similar to the image and ii) provides, as landmark locations for the image, landmark locations that are associated with the identified landmark images;

the caption location data is obtained from a caption processor that derives one or more caption locations from user input that includes one or more of an image caption, an album title, and an image tag; and

the metadata location data is obtained from a metadata processor that derives a metadata location from EXIF data for the image.

7. A system comprising:

a data processing apparatus; and

a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising;

obtaining landmark location data for an image, the landmark location data specifying one or more landmark locations derived from content depicted in the image and, for each landmark location, a landmark confidence score that indicates a measure of confidence that the image depicts content associated with the landmark location;

obtaining caption location data for the image, the caption location data specifying one or more caption locations derived from user input and, for each caption location, a caption confidence score that indicates a measure of confidence that the image depicts content associated with the caption location;

obtaining metadata location data for the image, the metadata location data specifying a metadata location derived from data provided by an image capturing device that captured the image;

identifying location pairs from the landmark location data, caption location data, and metadata location data, each location pair including two of a landmark location, caption location, and metadata location;

generating, for each location pair, a geographic consistency score that indicates a measure of consistency between locations specified in the location pair;

ranking each location pair according to a predetermined hierarchy of location types, the predetermined hierarchy specifying a first priority for landmark locations, a second priority for caption locations, and third priority for metadata locations, wherein the ranking of each location pair is based on a highest priority of the locations specified in the location pair;

selecting a location pair based at least in part on the geographic consistency scores and location pair rankings;

selecting an image location for the image from the locations specified in the selected location pair;

determining an image location score for the image location based on a confidence score for one of the locations specified in the selected location pair; and

associating the image location and image location score with the image.

8. The system of claim 7, wherein the operations further comprise:

identifying a confidence score for a location specified in the selected location pair;

identifying a location type for the location; and

generating a topicality score for the image based on the identified confidence score and the identified location type.

9. The system of claim 8, wherein identifying a confidence score for a location specified in the selected location pair comprises identifying a confidence score for a location specified in the selected location pair based on the pre-determined hierarchy of location types.

10. The system of claim 7, wherein selecting an image location for the image from the locations specified in the selected location pair comprises selecting an image location for the image from the locations specified in the selected location pair based on the pre-determined hierarchy of location types.

11. The system of claim 8, wherein generating a topicality score for the image comprises:

inputting the identified confidence score into a function that corresponds to the identified location type, wherein the function is one of three different functions that each correspond to a different location type; and

obtaining the topicality score for the image as output from the function.

12. The system of claim 7, wherein:

the landmark data is obtained from a landmark processor that i) identifies landmark images that are visually similar to the image and ii) provides, as landmark locations for the image, landmark locations that are associated with the identified landmark images;

the caption location data is obtained from a caption processor that derives one or more caption locations from user input that includes one or more of an image caption, an album title, and an image tag; and

the metadata location data is obtained from a metadata processor that derives a metadata location from EXIF data for the image.

13. A computer readable storage medium comprising instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:

obtaining landmark location data for an image, the landmark location data specifying one or more landmark locations derived from content depicted in the image and, for each landmark location, a landmark confidence score that indicates a measure of confidence that the image depicts content associated with the landmark location;

obtaining caption location data for the image, the caption location data specifying one or more caption locations derived from user input and, for each caption location, a caption confidence score that indicates a measure of confidence that the image depicts content associated with the caption location;

obtaining metadata location data for the image, the metadata location data specifying a metadata location derived from data provided by an image capturing device that captured the image;

identifying location pairs from the landmark location data, caption location data, and metadata location data, each location pair including two of a landmark location, caption location, and metadata location;

generating, for each location pair, a geographic consistency score that indicates a measure of consistency between locations specified in the location pair;

ranking each location pair according to a predetermined hierarchy of location types, the predetermined hierarchy specifying a first priority for landmark locations, a second priority for caption locations, and third priority for metadata locations, wherein the ranking of each location pair is based on a highest priority of the locations specified in the location pair;

selecting a location pair based at least in part on the geographic consistency scores and location pair rankings;

selecting an image location for the image from the locations specified in the selected location pair;

determining an image location score for the image location based on a confidence score for one of the locations specified in the selected location pair; and

associating the image location and image location score with the image.

14. The computer readable storage medium of claim 13, wherein the operations further comprise:

identifying a confidence score for a location specified in the selected location pair;

identifying a location type for the location; and

generating a topicality score for the image based on the identified confidence score and the identified location type.

15. The computer readable storage medium of claim 14, wherein identifying a confidence score for a location specified in the selected location pair comprises identifying a confidence score for a location specified in the selected location pair based on the pre-determined hierarchy of location types.

16. The computer readable storage medium of claim 13, wherein selecting an image location for the image from the locations specified in the selected location pair comprises selecting an image location for the image from the locations specified in the selected location pair based on the pre-determined hierarchy of location types.

17. The computer readable storage medium of claim 14, wherein generating a topicality score for the image comprises:

inputting the identified confidence score into a function that corresponds to the identified location type, wherein the function is one of three different functions that each correspond to a different location type; and

obtaining the topicality score for the image as output from the function.