Image tagging

• US 9,607,014 B2
• Filed: 10/31/2013
• Issued: 03/28/2017
• Est. Priority Date: 10/31/2013
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wang, Zhaowen, Yang, Jianchao, Lin, Zhe, Brandt, Jonathan
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US14/068,238
• Publication Number:
  US 20150120760A1
• Time in Patent Office: 1,244 Days
• Field of Search: 707/E17.03, 707/741, 707/744, 707/769, 382/118, 382/157, 706/12
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A system is configured to annotate an image with tags. As configured, the system accesses an image and generates a set of vectors for the image. The set of vectors may be generated by mathematically transforming the image, such as by applying a mathematical transform to predetermined regions of the image. The system may then query a database of tagged images by submitting the set of vectors as search criteria to a search engine. The querying of the database may obtain a set of tagged images. Next, the system may rank the obtained set of tagged images according to similarity scores that quantify degrees of similarity between the image and each tagged image obtained. Tags from a top-ranked subset of the tagged images may be extracted by the system, which may then annotate the image with these extracted tags.

Claims

1. A method comprising:

accessing an untagged image that is to be tagged;

generating a set of vectors by mathematically transforming the untagged image and regions of the untagged image;

querying a database of tagged images by submitting the set of vectors as search criteria to a search engine configured to search the database, the querying of the database obtaining a set of tagged images;

ranking the set of tagged images according to similarity scores that quantify degrees of similarity between the untagged image and each tagged image in the set of tagged images, wherein the similarity scores are calculated based on a similarity matrix that provides a higher similarity score for two images within the set of tagged images that have similar tags than for two images within the set of tagged images that have dissimilar tags; and

annotating the untagged image with tags from a subset of the ranked set of tagged images, the annotating being performed by a processor of a machine based on the similarity scores and ranks of tagged images within the subset.

2. The method of claim 1, wherein:

the tagged images are reference images each annotated by at least one tag; and

the querying of the database includes indexing the tagged reference images.

3. The method of claim 1, wherein:

the querying of the database includes retrieving the set of tagged images based on a bag-of-words analysis of visual features in each tagged image in the database.

4. The method of claim 1, wherein:

the querying of the database includes retrieving the set of tagged images based on spatial arrangement analysis of visual features in each tagged image in the database.

5. The method of claim 1, wherein the similarity matrix comprises

a bilinear similarity matrix.

6. The method of claim 5 further comprising:

generating the bilinear similarity matrix in satisfaction of a condition that the bilinear similarity matrix causes a first similarity score between a first tagged image in the database and a second tagged image in the database to exceed a second similarity score between the first tagged image and a third tagged image in the database as a consequence of a first mathematical distance between tags of the first tagged image and tags of the second tagged image being less than a second mathematical distance between tags of the first tagged image and tags of the third tagged image.

7. The method of claim 6, wherein:

the generating of the bilinear similarity matrix includes;

accessing a first version of the bilinear similarity matrix; and

generating a second version of the bilinear similarity matrix in response to a determination that an image triplet inclusive of the first tagged image, the second tagged image, and the third tagged image satisfies the condition.

8. The method of claim 7, wherein:

the generating of the bilinear similarity matrix includes setting the first version of the bilinear similarity matrix to an identity matrix.

9. The method of claim 6 further comprising:

calculating a rank score of a tagged image among the set of tagged images based on a rank of the tagged image.

10. The method of claim 9 further comprising:

determining the tags based on the rank score of the tagged image, a set of annotations stored in metadata of the tagged image, and a threshold function applicable to the set of annotations stored in the metadata of the tagged image.

11. The method of claim 10, wherein:

the determining of the tags is based on a weight vector that reduces a likelihood that an annotation is included in the tags based on a frequency with which the annotation occurs in the database of tagged images.

12. The method of claim 6, wherein:

the generating of the bilinear similarity matrix includes;

accessing the first tagged image from the database of tagged images;

generating a first vector from the first tagged image;

querying the database to obtain a further set of tagged images based on the first vector; and

determining that an image triplet inclusive of the first tagged image, the second tagged image, and the third tagged image satisfies the condition.

13. The method of claim 7, wherein:

the generating of the bilinear similarity matrix includes;

calculating a difference that corresponds to the image triplet between the first similarity score and the second similarity score; and

determining that the difference that corresponds to the image triplet is a minimum difference among image triplets that include the first tagged image and two further tagged images from the further set of tagged images.

14. The method of claim 5 further comprising:

determining a first feature vector of a first candidate image and a second feature vector of a second candidate image; and

calculating a further similarity score based on the bilinear similarity matrix, the further similarity score quantifying a degree of similarity between the first and second candidate images.

15. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing an untagged image that is to be tagged;

generating a set of vectors by mathematically transforming the untagged image and regions of the untagged image;

querying a database of tagged images by submitting the set of vectors as search criteria to a search engine configured to search the database, the querying of the database obtaining a set of tagged images;

ranking the set of tagged images according to similarity scores that quantify degrees of similarity between the untagged image and each tagged image in the set of tagged images, wherein the similarity scores are calculated based on a similarity matrix that provides a higher similarity score for two images within the set of tagged images that have similar tags than for two images within the set of tagged images that have dissimilar tags; and

annotating the untagged image with tags from a subset of the ranked set of tagged images, the annotating being based on the similarity scores and ranks of tagged images within the subset.

16. The non-transitory machine-readable storage medium of claim 15, wherein:

the similarity matrix is a bilinear similarity matrix; and

the bilinear similarity matrix is generated in satisfaction of a condition that the bilinear similarity matrix causes a first similarity score between a first tagged image in the database and a second tagged image in the database to exceed a second similarity score between the first tagged image and a third tagged image in the database as a consequence of a first mathematical distance between tags of the first tagged image and tags of the second tagged image being less than a second mathematical distance between tags of the first tagged image and tags of the third tagged image.

17. The non-transitory machine-readable storage medium of claim 16, wherein the operations further comprise:

determining a first feature vector of a first candidate image and a second feature vector of a second candidate image; and

calculating a further similarity score based on the bilinear similarity matrix, the further similarity score quantifying a degree of similarity between the first and second candidate images.

18. A system comprising:

one or more processors; and

one or more computer storage media storing computer-executable instructions that, when executed by the one or more processors, implement a method comprising;

accessing an untagged image that is to be tagged;

generating a set of vectors by mathematically transforming the untagged image and regions of the untagged image;

querying a database of tagged images by submitting the set of vectors as search criteria to a search engine configured to search the database, the querying of the database obtaining a set of tagged images;

ranking the set of tagged images according to similarity scores that quantify degrees of similarity between the untagged image and each tagged image in the set of tagged images, wherein the similarity scores are calculated based on a similarity matrix that provides a higher similarity score for two images within the set of tagged images that have similar tags than for two images within the set of tagged images that have dissimilar tags; and

annotating the untagged image with tags from a subset of the ranked set of tagged images, the annotating being performed by a processor of a machine based on the similarity scores and ranks of tagged images within the subset.

19. The system of claim 18, wherein:

the similarity matrix is a bilinear similarity matrix; and

the bilinear similarity matrix is generated in satisfaction of a condition that the bilinear similarity matrix causes a first similarity score between a first tagged image in the database and a second tagged image in the database to exceed a second similarity score between the first tagged image and a third tagged image in the database as a consequence of a first mathematical distance between tags of the first tagged image and tags of the second tagged image being less than a second mathematical distance between tags of the first tagged image and tags of the third tagged image.

20. The system of claim 19, wherein the method further comprises:

determining a first feature vector of a first candidate image and a second feature vector of a second candidate image; and

calculating a further similarity score based on the bilinear similarity matrix, the further similarity score quantifying a degree of similarity between the first and second candidate images.