Dual cross-media relevance model for image annotation

• US 8,571,850 B2
• Filed: 12/13/2007
• Issued: 10/29/2013
• Est. Priority Date: 09/13/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Mingjing, Wang, Bin, Li, Zhiwei, Ma, Wei-Ying, Lui, Jing
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US11/956,331
• Publication Number:
  US 20090076800A1
• Time in Patent Office: 2,147 Days
• Field of Search: 707/999.003
• US Class Current: 704/9
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/242 Dictionaries

Abstract

A dual cross-media relevance model (DCMRM) is used for automatic image annotation. In contrast to the traditional relevance models which calculate the joint probability of words and images over a training image database, the DCMRM model estimates the joint probability by calculating the expectation over words in a predefined lexicon. The DCMRM model may be advantageous because a predefined lexicon potentially has better behavior than a training image database. The DCMRM model also takes advantage of content-based techniques and image search techniques to define the word-to-image and word-to-word relations involved in image annotation. Both relations can be estimated by using image search techniques on the web data as well as available training data.

Claims

1. A method comprising:

for each word v of a lexicon V;

obtaining a word-to-word correlation between a candidate word w and each word v in the lexicon V measured by P(w|v) which comprises a conditional probability of the candidate word w given the word v; and

obtaining a word-to-image correlation between the word v and a target image I_u independently of a training set of images, based at least on an image representation of the word v that is developed through an analysis of one or more top ranked images obtained by an image search engine searching on the word v, the word-to-image correlation between each word v in the lexicon V and the target image I_u measured by P(I_u|v) which comprises a conditional probability of the target image I_u given the word v;

determining a value of a collective word-to-image correlation between the candidate word w and the target image I_u based on the word-to-word correlations between the candidate word w and each word v in the lexicon V and the word-to-image correlations between each word v in the lexicon V and the target image I_u; and

annotating the target image I_u using the candidate word w if the collective word-to-image correlation between the candidate word w and the target image I_u satisfies a preset condition, the value of the collective word-to-image correlation between the candidate word w and the target image I_u calculated using;

2. The method as recited in claim 1, wherein obtaining the word-to-word correlation between the word v and the candidate word w and determining the value of the collective word-to-image correlation between the candidate word w and the target image I_u are repeated for a plurality of candidate words selected from the lexicon V, and wherein annotating the target image I_u comprises:

comparing the values of the collective word-to-image correlations of the plurality of candidate words; and

annotating the target image I_u with at least one of the candidate words whose value of the collective word-to-image correlation is among the highest of the plurality of candidate words.

3. The method as recited in claim 1, wherein annotating the target image I_u comprises:

calculating the value of the collective word-to-image correlation between the candidate word w and the target image using the following formula;

4. The method as recited in claim 1, wherein obtaining the word-to-image correlation between each word v and the target image I_u comprises:

performing an image search on an image data source using the word v as a query word;

selecting a plurality of images from results of the image search to form the image representation of the word v;

determining an image-to-image correlation between the target image I_u and the image representation of the word v; and

obtaining the word-to-image correlation between the word v and the target image I_u based on the image-to-image correlation between the target image I_u and the image representation of the word v, wherein the word-to-image correlation is the conditional probability of the target image I_u given the word v.

5. The method as recited in claim 1, wherein the target image I_u is associated with a characterizing text, and obtaining the word-to-image correlation between each word v and the target image I_u comprises:

determining a word-to-word correlation between the word v and the characterizing text; and

determining the word-to-image correlation between the word v and the target image I_u based on the word-to-word correlation between the word v and the characterizing text.

6. The method as recited in claim 1, wherein obtaining the word-to-word correlation between each word v and the candidate word w comprises:

calculating a visual distance or a visual similarity between the image representation of the word v and an image representation of the candidate word w.

7. The method as recited in claim 1, wherein obtaining the word-to-word correlation between each word v and the candidate word w comprises:

providing the image representation of the word v and an image representation of the candidate word w; and

determining the word-to-word correlation between the word v and the candidate word w at least partially based on visual features of the image representations of the word and the candidate word w.

8. The method as recited in claim 7, wherein providing the image representation of each word v comprises:

conducting an image search using the word v as a query word; and

selecting a plurality of images from search results associated with the query word.

9. The method as recited in claim 7, wherein providing the image representation of the candidate word w comprises:

conducting an image search using the candidate word w as a query word; and

selecting a plurality of images from search results associated with the query word.

10. The method as recited in claim 1, wherein obtaining the word-to-word correlation between each word v and the candidate word w comprises:

providing a conjunctive image representation of the word v and the candidate word w; and

determining the word-to-word correlation between the word v and the candidate word w at least partially based on visual features of the conjunctive image representation of the word v and the candidate word w.

11. The method as recited in claim 1, wherein obtaining the word-to-word correlation between each word v and the candidate word w comprises:

calculating a first correlation between the word v and the candidate word w based on visual features of the image representation of the word v and an image representation of the candidate word w;

calculating a second correlation between the word v and the candidate word w using an alternative method that is at least one of a text-based method or a conjunctive image representation method; and

determining the word-to-word correlation between the word v and the candidate word w by combining the first correlation and the second correlation.

12. A method performed by a computing device for image annotation, the method comprising:

obtaining a plurality of word-to-word correlations, each word-to-word correlation being defined between a pair of words selected from a lexicon V, each word-to-word correlation between a candidate word w and each word v in the lexicon V measured by P(w|v) which comprises a conditional probability of the word w given the word v;

obtaining a plurality of word-to-image correlations independently of a training set of images, each word-to-image correlation being defined between the word v in the lexicon V and a target image I_u being annotated, each word-to-image correlation between each word v and the target image I_u measured by P(I_u|v) which comprises a conditional probability of the target image I_u given the word v;

for a candidate word w selected from the lexicon V, determining value of a collective word-to-image correlation between the candidate word w and the target image I_u based on the word-to-word correlations between the candidate word w and each word v in the lexicon V, and the word-to-image correlations between each word v in the lexicon V and the target image I_u, wherein the collective word-to-image correlation is a probability of the target image I_u given the candidate word w; and

annotating the target image I_u using the candidate word w if the value of the collective word-to-image correlation between the candidate word w and the target image I_u satisfies a preset condition.

13. The method as recited in claim 12, wherein determining a value of a collective word-to-image correlation between the candidate word w and the target image I_u is performed for a plurality of candidate words respectively, and annotating the target image I_u comprises:

comparing the values of the collective word-to-image correlations of the plurality of candidate words; and

annotating the target image I_u with at least one of the candidate words whose value of the collective word-to-image correlation is among the highest of the plurality of candidate words.

14. The method as recited in claim 12, wherein obtaining the word-to-image correlation between each word v and the target image I_u comprises:

performing an image search on an image data source using the word v as a query word;

selecting a plurality of images from results of the image search to form an image representation of the word v;

determining an image-to-image correlation between the target image I_u and the image representation of the word v; and

determining the word-to-image correlation between the word v and the target image I_u based on the image-to-image correlation between the target image I_u and the image representation of the word v.

15. The method as recited in claim 12, wherein obtaining the word-to-word correlation between each pair of words comprises:

calculating a visual distance or a visual similarity between an image representation of first word of the pair and an image representation of second word of the pair.

16. The method as recited in claim 12, wherein obtaining the word-to-word correlation between each pair of words comprises:

providing a conjunctive image representation of the pair of words; and

determining the word-to-word correlation between the pair of words at least partially based on visual features of the conjunctive image representation of the pair of words.

17. The method as recited in claim 12, wherein obtaining the word-to-word correlation between each pair of words comprises:

for each of the pair of words, conducting an image search on an image data source using the word as a query word, and selecting a plurality of images from search results to form an image representation of the word; and

determining the word-to-word correlation between each pair of words based on an image-to-image correlation between the image representations of the pair of words.

18. One or more computer readable memory devices having stored thereupon a plurality of instructions that, when executed by a processor, causes the processor to:

obtain a plurality of word-to-word correlations, each word-to-word correlation being defined between a pair of words selected from a lexicon V, each word-to-word correlation between a candidate word w and each word v in the lexicon V measured by P(w|v) which comprises a conditional probability of the word w given the word v;

obtain a plurality of word-to-image correlations, each word-to-image correlation being defined between a word v in the lexicon V and the target image I_u being annotated, based at least on an image representation of the word v, each word-to-image correlation between each word v and the target image I_u measured by P(I_u|v) which comprises a conditional probability of the target image I_u given the word v;

for the candidate word w selected from lexicon V, determine a value of a collective word-to-image correlation between the candidate word w and the target image I_u based on the word-to-word correlations between each word v in the lexicon V and the candidate word w, and the word-to-image correlations between each word v in the lexicon V and the target image I_u, wherein the collective word-to-image correlation is a probability of the target image I_u given the candidate word w; and

annotate the target image I_u using the candidate word w if the collective word-to-image correlation between the candidate word w and the target image I_u satisfies a preset condition.

19. The one or more computer readable memory devices as recited in claim 18, wherein:

for a candidate word w selected from the lexicon V, determining the value of the collective word-to-image correlation between the candidate word w and the target image I_u is performed for a plurality of candidate words, and

annotating the target image I_u comprises;

comparing the values of the collective word-to-image correlations of the plurality of candidate words; and

annotating the target image I_u with at least one of the candidate words whose value of the collective word-to-image correlation is among the highest of the plurality of candidate words.

20. The one or more computer readable memory devices as recited in claim 18, wherein obtaining the word-to-image correlation between each word and the target image I_u comprises:

performing an image search on an image data source using the word v as a query word;

selecting a plurality of images from results of the image search to form an image representation of the word v;

determining an image-to-image correlation between the target image I_u and the image representation of the word v; and

determining the word-to-image correlation between the word and the target image I_u based on the image-to-image correlation between the target image I_u and the image representation of the word v.