GENERATING LABELS FOR IMAGES ASSOCIATED WITH A USER

• US 20170185670A1
• Filed: 12/28/2015
• Published: 06/29/2017
• Est. Priority Date: 12/28/2015
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: DUA, Robin, RAVI, Sujith
• Application Number:
  US14/980,790
• Publication Number:
  US 20170185670A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/538 Presentation of query results
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video

Abstract

A method includes identifying an image associated with a user, where the image is identified as at least one of captured by a user device associated with the user, stored on the user device associated with the user, and stored in cloud storage associated with the user. The method also includes determining one or more labels for the image, where the one or more labels are based on at least one of metadata, a primary annotation, and a secondary annotation and the secondary annotation is generated by performing label expansion on at least one of the metadata and the primary annotation. The method also includes generating a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to the image.

Claims

1. A computer-implemented method comprising:

identifying an image associated with a user, wherein the image is identified as at least one of captured by a user device associated with the user, stored on the user device associated with the user, and stored in cloud storage associated with the user;

determining one or more labels for the image, wherein;

the one or more labels are based on at least one of metadata, a primary annotation, and a secondary annotation, and

the secondary annotation is generated by performing label expansion on at least one of the metadata and the primary annotation; and

generating a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to the image.

2. The method of claim 1, wherein the one or more labels are based on the primary annotation and further comprising:

generating the primary annotation by performing at least one of;

image recognition to determine one or more of an entity that appears in the image and a characteristic associated with the image, and

conversion of the metadata to the primary annotation based on an inference about the metadata.

3. The method of claim 1, wherein the label expansion includes expanding the at least one of the metadata and the primary annotation based on a hierarchical taxonomy.

4. The method of claim 1, wherein the label expansion includes expanding the at least one of the metadata and the primary annotation based on at least one of a semantic similarity of the at least one of the metadata and the primary annotation to the secondary annotation and a visual similarity of the at least one of the metadata and the primary annotation to the secondary annotation.

5. The method of claim 1, further comprising:

receiving from the user a search query for the image associated with the user, wherein the search query includes one or more user-generated search terms; and

generating, based on the one or more user-generated search terms and the mapping, one or more suggested search terms for the user that autocomplete the one or more user-generated search terms.

6. The method of claim 1, further comprising:

receiving from the user a search query for the image associated with the user, wherein the search query includes one or more user-generated search terms; and

retrieving additional information to translate the one or more user-generated search terms into categorized search terms, the categorized search terms including at least one of a date, a time, latitude and longitude coordinates, an altitude, and a direction.

7. The method of claim 6, further comprising:

identifying search results by determining a match between the categorized search terms and the one or more labels in the mapping; and

ranking the search results based on the match between the categorized search terms and the one or more labels.

8. The method of claim 1, further comprising:

identifying a user activity to associate with a location; and

associating a user activity annotation with the image that are associated with the location.

9. The method of claim 1, further comprising:

receiving from the user a first search query for the image associated with the user;

providing the user with first search results that match the first search query;

receiving from the user a second search query;

determining, based on one or more terms in the second search query, that the second search query is to be applied to the first search results; and

providing the user with second search results that are filtered from the first search results and that match the second search query.

10. The method of claim 1, wherein:

the image includes multiple image,

the mapping includes a graph of the images, and

the images represent nodes and each edge between the nodes is based on the one or more labels associated with corresponding images.

11. The method of claim 10, further comprising:

receiving one or more user-generated search terms from the user for the images associated with the user;

identifying search results that include images from the mapping;

generating a ranked list of search results based on confidence scores associated with each corresponding image; and

providing at least a portion of the ranked list of search results to the user.

12. A computer system comprising:

one or more processors coupled to a memory;

an image processing module stored in the memory and executable by the one or more processors, the image processing module operable to identify images associated with a user and operable to for each image, determine one or more labels for the image, wherein the one or more labels are based on at least one of metadata, a primary annotation, and a secondary annotation and the secondary annotation is generated by performing label expansion on at least one of the metadata and the primary annotation;

an indexing module stored in the memory and executable by the one or more processors, the indexing module operable to generate a mapping of the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to corresponding images; and

a search module stored in the memory and executable by the one or more processors, the search module operable to receive from the user a search query for the image associated with the user, wherein the search query includes one or more user-generated search terms.

13. The system of claim 12, wherein the search module is further operable to:

generate, based on the one or more user-generated search terms and the mapping, one or more suggested search terms for the user that autocomplete the one or more user-generated search terms.

14. The system of claim 12, wherein the search module is further operable to:

retrieve additional information to translate the one or more user-generated search terms into categorized search terms, the categorized search terms including at least one of a date, a time, latitude and longitude coordinates, an altitude, and a direction.

15. The system of claim 14, wherein the search module is further operable to:

identify search results by determining a match between the categorized search terms and the one or more labels in the mapping; and

ranking the search results based on the match between the categorized search terms and the one or more labels.

16. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

identifying images associated with a user;

for each of the images determining one or more labels, wherein;

the one or more labels are based on at least one of metadata, a primary annotation, and a secondary annotation, and

generating the primary annotation by performing at least one of;

image recognition to determine one or more of an entity that appears in the image and a characteristic associated with the image and conversion of the metadata based on inferences, and

conversion of the metadata based on inferences; and

generating a mapping the one or more labels to one or more confidence scores, wherein the one or more confidence scores indicate an extent to which the one or more labels apply to corresponding images.

17. The computer storage medium of claim 16, wherein the instructions are further operable to perform operations comprising:

for at least one of the images that includes the entity, determining a boundary of the entity; and

responsive to receiving a selection by the user that is within the boundary of the entity in the image, retrieving additional information about the entity.

18. The computer storage medium of claim 16, wherein the instructions are further operable to perform operations comprising:

receiving a request from the user for additional information;

determining that the request is for additional information about the entity in one of the images;

responsive to receiving the request, obtaining the additional information from a server-hosted knowledge graph; and

providing the additional information to the user.

19. The computer storage medium of claim 16, wherein the instructions are further operable to perform operations comprising:

associating one or more of the one or more labels with a boundary of an entity in a first image of the images;

providing the user with the first image;

receiving a selection within the boundary of the entity;

determining the one or more labels that correspond to the entity;

searching for additional information for the entity; and

providing the user with the additional information.

20. The computer storage medium of claim 16, wherein the instructions are further operable to perform operations comprising:

receiving from the user a first search query for one or more of the images associated with the user;

providing the user with first search results that match the first search query;

receiving from the user a second search query;

determining, based on one or more terms in the second search query, that the second search query is to be applied to the first search results; and

providing the user with second search results that are filtered from the first search results and that match the second search query.