Estimating word correlations from images

• US 8,457,416 B2
• Filed: 12/13/2007
• Issued: 06/04/2013
• Est. Priority Date: 09/13/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Liu, Jing, Wang, Bin, Li, Zhiwei, Li, Mingjing, Ma, Wei-Ying
• Primary Examiner(s): ZARKA, DAVID PETER
• Application Number:
  US11/956,333
• Publication Number:
  US 20090074306A1
• Time in Patent Office: 2,000 Days
• Field of Search: 382/229
• US Class Current: 382/229
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

Word correlations are estimated using a content-based method, which uses visual features of image representations of the words. The image representations of the subject words may be generated by retrieving images from data sources (such as the Internet) using image search with the subject words as query words. One aspect of the techniques is based on calculating the visual distance or visual similarity between the sets of retrieved images corresponding to each query word. The other is based on calculating the visual consistence among the set of the retrieved images corresponding to a conjunctive query word. The combination of the content-based method and a text-based method may produce even better result.

Claims

1. A system for estimating word correlations, comprising:

a processor; and

a computer readable storage media having stored thereupon a plurality of instructions that, when executed by the processor, cause the processor to perform acts comprising;

providing a first image representation set having a first plurality of images representing a first word in response to a first search based at least in part on the first word and a second image representation set having a second plurality of images representing a second word in response to a second search based at least in part on the second word;

extracting, from at least one image of the first image representation set and from at least one image of the second image representation set, multi-modal visual features; and

estimating a correlation between the first word and the second word at least partially based on a calculation of a mathematical function having at least the extracted multi-modal visual features corresponding to the first image representation set and at least the extracted multi-modal visual features corresponding to the second image representation set as variables thereto, wherein estimating the correlation includes;

selecting one of a first correlation function for a calculation of an inter-set visual aspect correlation based on multiple images of the first image representation set and multiple images of the second image representation set and a second correlation function for a calculation of an intra-set visual aspect correlation based on multiple images within one of the first image representation set and the second image representation set; and

calculating the selected correlation function.

2. The system as recited in claim 1, wherein the selected correlation function is the first correlation function, and wherein calculating the selected correlation function comprises:

calculating a visual distance or a visual similarity between the first image representation set and the second image representation set.

3. The system as recited in claim 1, wherein providing the first and the second image representation sets representing the first word and the second word comprises:

conducting an image search using the respective first word or second word as query word; and

selecting a third plurality of images from search results.

4. The system as recited in claim 3, wherein the image search is conducted on the Internet using an image search engine.

5. The system as recited in claim 3, wherein the third plurality of images are selected from top returns of the search results.

6. The system as recited in claim 1, further comprising:

providing a set of images representing conjunction of the first word and the second word, the set of images comprising a third plurality of images,

wherein estimating the correlation between the first word and second word is further based on visual features of the set of images representing conjunction of the first word and the second word.

7. The system as recited in claim 6, wherein estimating the correlation between the first word and second word comprises calculating a visual consistence of the set of images representing conjunction of the first word and the second word.

8. The system as recited in claim 6, wherein providing the set of images representing conjunction of the first word and the second word comprises:

performing an image search using the first word and the second word as a conjunctive query word; and

selecting a fourth plurality of images from search results.

9. The system as recited in claim 6, wherein the selected correlation function is the second correlation function, and wherein calculating the selected correlation function comprises:

calculating a first visual consistence of the first image representation set representing the first word;

calculating a second visual consistence of the second image representation set representing the second word; and

calculating a third visual consistence of the set of images representing conjunction of the first word and the second word; and

wherein the estimating the correlation between the first word and the second word comprises evaluating a function having the first visual consistence, the second visual consistence and the third visual consistence as terms thereof.

10. A system for estimating word correlations, comprising:

a processor; and

a computer readable storage media having stored thereupon a plurality of instructions that, when executed by the processor, cause the processor to perform acts comprising;

providing a first image representation set having a first plurality of images representing a first word in response to a first search based at least in part on the first word and a second image representation set having a second plurality of images representing a second word in response to a second search based at least in part on the second word;

extracting, from at least one image of the first image representation set and from at least one image of the second image representation set, multi-modal visual features; and

estimating a correlation between the first word and the second word at least partially based on a calculation of a mathematical function having at least the extracted multi-modal visual features corresponding to the first image representation set and at least the extracted multi-modal visual features corresponding to the second image representation set as variables thereto, wherein estimating the correlation between the first word and the second word includes;

calculating a content correlation between the multi-modal visual features of the image representations of the first word and the second word;

calculating a text correlation between the first word and the second word using a text-based method; and

estimating the correlation between the first word and the second word by combining the content correlation and the text correlation.

11. The system as recited in claim 10, wherein providing the first and the second image representation sets representing the first word and the second word comprises:

conducting an image search using the respective first word or second word as query word; and

selecting a third plurality of images from search results.

12. The system as recited in claim 11, wherein the image search is conducted on the Internet using an image search engine.

13. The system as recited in claim 11, wherein the third plurality of images are selected from top returns of the search results.

14. A method for estimating word correlations, the method comprising:

providing a computing device with a conjunctive image representation of a first word and a second word, the conjunctive image representation comprising a first set of multiple images;

providing the computing device with a first image representation of the first word and a second image representation of the second word, the first image representation of the first word comprising a second set of multiple images, and the second image representation of the second word comprising a third set of multiple images; and

estimating, by the computing device, a correlation between the first word and second word at least partially based on a calculation of visual features calculated from the first set of multiple images comprising the conjunctive image representation of the first word and the second word, the second set of multiple images comprising the first image representation of the first word and the third set of multiple images comprising the second image representation of the second word.

15. The method as recited in claim 14, wherein estimating, by the computing device, the correlation between the first word and second word comprises calculating, by the computing device, a visual consistence of the first set of multiple images.

16. The method as recited in claim 14, wherein estimating, by the computing device, the correlation between the first word and the second word comprises:

selecting a subset of image features from a plurality of image features extracted from the first set of multiple images, wherein the subset of image features includes at least one image feature having a low variance according to a preset variance threshold; and

calculating a partial variance of the subset of image features.

17. The method as recited in claim 16, wherein selecting the subset of image features is performed dynamically as the conjunctive image representation varies.

18. The method as recited in claim 14, wherein providing the conjunctive image representation of the first word and the second word comprises:

conducting an image search using the first word and the second word as a conjunctive query word; and

selecting a plurality of images from search results.

19. One or more computer readable storage media device having stored thereupon a plurality of instructions that, when executed by a processor, causes the processor to:

provide a first image representation of a first word in response to a first search based at least in part on the first word and a second image representation of a second word in response to a second search based at least in part on the second word, the first image representation comprising a first set of images having a first plurality of images, the second image representation comprising a second set of images having a second plurality of images;

select one of a first correlation function for a calculation of an inter-set visual aspect correlation based on multiple images of the first set of images and multiple images of the second set of images or a second correlation function for a calculation of an intra-set visual aspect correlation based on multiple images within one of the first set of images and the second set of images;

calculate the selected correlation function; and

estimate a correlation between the first word and the second word at least partially based on the calculation of the selected correlation function.

20. The computer readable storage media device as recited in claim 19, wherein the processor estimates the correlation between the first word and second word by calculating a visual distance or a visual similarity between the first image representation of the first word and the second image representation of the second word.

21. The computer readable storage media device as recited in claim 19, wherein the plurality of instructions, when executed by a processor, further causes the processor to:

provide a conjunctive image representation of the first word and the second word, wherein the processor estimates the correlation between the first word and second word further based on visual features of the conjunctive image representation of the first word and the second word.