System and method for determining an annotation for an image

• US 8,971,644 B1
• Filed: 01/18/2012
• Issued: 03/03/2015
• Est. Priority Date: 01/18/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jing, Yushi, Pasca, Alexandru Marius, Cai, Wei
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Lesnick, Alexander J
• Application Number:
  US13/352,867
• Time in Patent Office: 1,140 Days
• Field of Search: None
• US Class Current: 382/224
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

A plurality of images related to a particular image are determined, and a plurality of annotations associated with the plurality of images are identified. An ontology is determined based on the plurality of annotations, and an image annotation for the particular image is determined based on the ontology. The ontology may be a semantic ontology, for example.

Claims

1. A method of determining an annotation for a particular image, the method comprising:

determining a plurality of images related to the particular image, the plurality of images stored in one or more computer systems;

identifying a plurality of annotations associated with the plurality of images;

generating an ontology for the particular image wherein the ontology comprises;

a plurality of terms, the plurality of annotations, and the plurality of images arranged in a hierarchy with the plurality of annotations being downstream from the plurality of terms associated with a highest level of the hierarchy, and the plurality of images being downstream from the plurality of annotations, and

a plurality of links defining relationships between respective terms, annotations, or images, wherein each link is associated with a respective relevance value indicating a measure of relevance between two respective terms, annotations, or images connected by a respective link;

determining a total relevance value for each term associated with the highest level, wherein for each term associated with the highest level, the total relevance value is a sum of relevance values of links downstream from the term; and

associating one of the plurality of terms having a highest total relevance value with the particular image as an image annotation.

2. The method of claim 1, wherein the step of determining a plurality of images related to the particular image further comprises:

generating a fingerprint associated with the particular image; and

determining the plurality of images related to the particular image, based on the fingerprint associated with the particular image and fingerprints associated with the plurality of images.

3. The method of claim 1, further comprising receiving the particular image and a request for annotation of the particular image,

wherein generating an ontology for the particular image is in response to receiving the particular image and a request for annotation of the particular image.

4. The method of claim 1, wherein each image of the plurality of images is associated with an image relevance value, wherein the image relevance value is an indication of a measure of relevance to the particular image, and wherein the total relevance value is further based on the sum of image relevance values of images downstream from each term associated with the highest level.

5. The method of claim 1, further comprising:

determining a total relevance value for each term in a plurality of terms in a next highest level of the ontology; and

associating one of the terms in the plurality of terms in the next highest level of the ontology with the particular image as an image annotation.

6. The method of claim 1, further comprising:

Obtaining information defining the relationships between the plurality of terms from an information resource available via a network.

7. The method of claim 1, wherein the semantic ontology assigns each of the plurality of terms to a respective level within the hierarchy.

8. The method of claim 7, comprising:

identifying as the image annotation a term within the semantic ontology associated with the highest level within the hierarchy.

9. The method of claim 8, wherein the step of determining the image annotation for the particular image further comprises:

identifying the plurality of terms within the semantic ontology associated with the highest level within the hierarchy; and

identifying as the image annotation a term associated with the highest level that has a greatest number of links.

10. A non-transitory computer readable medium having program instructions stored thereon, that, in response to execution by a processor, cause a computing device to perform operations comprising:

determining a plurality of images related to the particular image;

identifying a plurality of annotations associated with the plurality of images;

generating an ontology for the particular image wherein the ontology comprises;

a plurality of terms, the plurality of annotations, and the plurality of images arranged in a hierarchy with the plurality of annotations being downstream from the plurality of terms associated with a highest level of the hierarchy, and the plurality of images being downstream from the plurality of annotations, and

a plurality of links defining relationships between respective terms, annotations, or images, wherein each link is associated with a respective relevance value indicating a measure of relevance between two respective terms, annotations, or images connected by a respective link;

determining a total relevance value for each term associated with the highest level, wherein for each term associated with the highest level, the total relevance value is a sum of relevance values of links downstream from the term; and

associating one of the plurality of terms having a highest total relevance value with the particular image as an image annotation.

11. The non-transitory computer readable medium of claim 10, further comprising instructions defining the steps of:

generating a fingerprint associated with the particular image; and

determining the plurality of images related to the particular image, based on the fingerprint associated with the particular image and fingerprints associated with the plurality of images.

12. The non-transitory computer readable medium of claim 10, further comprising instructions defining the step of receiving a request for annotation of the particular image,

wherein generating an ontology for the particular image is in response to receiving the request for annotation of the particular image.

13. The non-transitory computer readable medium of claim 10, further comprising instructions defining the step of:

obtaining information defining relationships between the plurality of terms from an information resource available via a network.

14. The non-transitory computer readable medium of claim 10, wherein the semantic ontology assigns each of the plurality of terms to a respective level within the hierarchy.

15. The non-transitory computer readable medium of claim 14, further comprising instructions defining the step of:

identifying as the image annotation a term within the semantic ontology associated with the highest level within the hierarchy.

16. The non-transitory computer readable medium of claim 15, further comprising instructions defining the step of:

identifying the plurality of terms within the semantic ontology associated with the highest level within the hierarchy; and

identifying as the image annotation a term associated with the highest level that has a greatest number of links.

17. A system comprising one or more processors and a non-transitory computer readable medium having program instructions stored thereon, that, in response to execution by a processor, cause a computing device to perform operations comprising:

determining a plurality of images related to the particular image;

identifying a plurality of annotations associated with the plurality of images;

generating an ontology for the particular image wherein the ontology comprises;

a plurality of terms, the plurality of annotations, and the plurality of images arranged in a hierarchy with the plurality of annotations being downstream from the plurality of terms associated with a highest level of the hierarchy, and the plurality of images being downstream from the plurality of annotations, and

a plurality of links defining relationships between respective terms, annotations, or images, wherein each link is associated with a respective relevance value indicating a measure of relevance between two respective terms, annotations, or images connected by a respective link;

determining a total relevance value for each term associated with the highest level, wherein for each term associated with the highest level, the total relevance value is a sum of relevance values of links downstream from the term; and

associating one of the plurality of terms having a highest total relevance value with the particular image as an image annotation.

18. The system of claim 17, wherein the non-transitory computer readable medium further comprises program instructions stored thereon, that, in response to execution by a processor, cause a computing device to perform operations comprising:

generating a fingerprint associated with the particular image; and

determining the plurality of images related to the particular image, based on the fingerprint associated with the particular image and fingerprints associated with the plurality of images.