Associating classifications with images

• US 9,760,803 B2
• Filed: 11/08/2013
• Issued: 09/12/2017
• Est. Priority Date: 05/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kulick, Matthew, Adam, Hartwig, Lee, Taehee, Potetz, Brian
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Azima, Shaghayegh
• Application Number:
  US14/075,424
• Publication Number:
  US 20140341476A1
• Time in Patent Office: 1,404 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/24 Classification techniques

Abstract

One or more devices may receive multiple images; determine one or more classifications for each of the multiple images; generate a confidence score for each classification of the one or more classifications for each of the multiple images. The confidence score may include a value between two numbers and may identify the likelihood that that a particular classification, of the one or more classifications, correctly corresponds to a particular image, of the multiple images. The confidence score may be based on information, associated with a user, and pixel-based classification information. The one or more devices may sort the multiple images by the one or more classifications and by corresponding confidence scores to form sorted multiple images; and provide the sorted multiple images.

Claims

1. A method comprising:

receiving, by one or more devices, a plurality of images;

determining, by the one or more devices, one or more classifications for each of the plurality of images wherein the one or more classifications are structured and linked with at least one other classification, wherein the one or more classifications for each image of the plurality of images are based on data inputs including one or more of pixel-based information determined based on pixels of the image, metadata information determined based on metadata associated with the image, and user information of a user associated with the image;

generating, by the one or more devices, a confidence score for each classification of the one or more classifications for each of the plurality of images, the confidence score identifying a likelihood that a particular classification, of the one or more classifications, correctly corresponds to a particular image, of the plurality of images, the confidence score being based on the data inputs, wherein each data input is assigned a respective weight and wherein the confidence score is generated by combining weighted data values;

sorting, by the one or more devices, the plurality of images by the one or more classifications and by corresponding confidence scores to form a sorted plurality of images; and

providing, by the one or more devices, the sorted plurality of images.

2. The method of claim 1, further comprising:

receiving a search query;

identifying images, of the plurality of images, based on the one or more classifications and the search query;

generating a ranking score for each of the identified images based on the confidence scores for the images; and

generating a ranked list of the identified images, the identified images being sorted in the ranked list based on the ranking scores,

wherein providing the sorted plurality of images includes providing the ranked list.

3. The method of claim 2, where the ranking score is further based on at least one of a relevancy score and a quality score value,

the relevancy score being based on the search query,

the quality score value being based on a size, a resolution, or a measure of popularity of the images.

4. The method of claim 1, where determining the one or more classifications is based on an object in respective image, of the plurality of images, that identifies a particular event and based on event-related user information that identifies the particular event.

5. The method of claim 1, is based on wherein the metadata identifies a particular time and the user information identifies an event occurring at the particular time.

6. The method of claim 1, wherein the metadata identifies a particular time, the user information identifies a particular event occurring at the particular time, and an object in a respective image, of the plurality of images, that identifies the particular event.

7. The method of claim 1, where each classification of the one or more classifications is indicative of an event, an object, a time of day, an activity, a geographic location, a person, or a facial expression associated with a particular image of the plurality of images.

17. The method of claim 1, wherein at least a first classification and a second classification of the one or more classifications overlap, and wherein a first confidence score for the first classification is higher than a second confidence score for the second classification that is more specific than the first classification.

18. The method of claim 1, further comprising determining a time-span of the plurality of images based at least in part on respective timestamps associated with each of the plurality of images.

19. The method of claim 18, wherein at least one classification of the one or more classifications is based on the time-span.

20. The method of claim 1, wherein at least one image of the plurality of images is assigned a classification based on data inputs associated with one or more other images of the plurality of images.

8. A system comprising:

a storage device; and

a hardware processor coupled to the storage device, the processor configured to;

store a data structure on the storage device, the data structure associating a plurality of images with one or more classifications for each image of the plurality of images, wherein the one or more classifications are structured and linked with at least one other classification, and wherein the one or more classifications for each image of the plurality of images are based on data inputs that include one or more of pixel-based information determined based on pixels of the image, metadata information determined based on metadata associated with the image, and user information of a user associated with the image, the data structure associating a confidence score for each classification of the one or more classifications for each image of the plurality of images, the confidence score being based on the data inputs, wherein each data input is assigned a respective weight and wherein the confidence score is generated by combining weighted data values, the data structure sorting the plurality of images by the one or more classifications and by corresponding confidence scores to form a sorted plurality of images;

receive a search query;

identify images, of the plurality of images, based on the one or more classifications stored in the data structure on the storage device, and the search query;

generate a ranking score for each image of the identified images based on the confidence scores for the images;

generate a ranked list of the identified images, the identified images being sorted in the ranked list based on the ranking scores; and

provide the ranked list to a user device to cause the user device to display the ranked list.

9. The system of claim 8, wherein the one or more classifications are based on an object in a particular image, of the plurality of images, that identifies a particular event and wherein the user information identifies the particular event.

10. The system of claim 8, the metadata identifies a particular time and the user information identifies an event occurring at the particular time.

11. The system of claim 8, where each classification of the one or more classifications is indicative of an event, an object, a time of day, an activity, a geographic location, a person, or a facial expression associated with a particular image of the plurality of images.

15. The system of claim 8, wherein the data structure stores overlapping classifications for the image.

12. A non-transitory computer-readable medium for storing instructions, the instructions comprising:

a plurality of instructions that, when executed by one or more processors associated with a device, causes the one or more processors to;

store a data structure, the data structure associating a plurality of images with one or more classifications-for each image of the plurality of images, wherein the one or more classifications are structured and linked with at least one other classification, and wherein the one or more classifications for each image of the plurality of images are based on data inputs including one or more of pixel-based information determined based on pixels of the image, metadata information determined based on metadata associated with the image, and user information of a user associated with the image, the data structure associating a confidence score for each classification of the one or more classifications for each image of the plurality of images, the confidence score identifying a likelihood that that a particular classification, of the one or more classifications, correctly corresponds to a particular image, of the plurality of images, the confidence score being based on the data inputs, wherein each data input is assigned a respective weight and wherein the confidence score is generated by combining weighted data values;

receive a search query;

identify images, of the plurality of images, based on the one or more classifications stored in the data structure, and the search query;

generate a ranking score for each image of the identified images based on the confidence scores for the images;

generate a ranked list of the identified images, the identified images being sorted in the ranked list based on the ranking scores; and

provide the ranked list to a user device to cause the user device to display the ranked list.

13. The non-transitory computer-readable medium of claim 12, wherein the user information includes at least one of an e-mail account associated with the user, a calendar associated with the user, a web chat associated with the user, a voice call associated with the user, a web browsing history profile associated with the user, or a social networking profile associated with the user.

14. The non-transitory computer-readable medium of claim 12, wherein the one or more classifications are based on an object in a particular image, of the plurality of images, that identifies a particular event and wherein the user information that identifies the particular event.

16. The non-transitory computer-readable medium of claim 12, wherein the data structure associates a classification in a form of an entity identifier associated with a knowledge base or a collection of structured data.