Text to Image Translation

• US 20120296897A1
• Filed: 05/18/2011
• Published: 11/22/2012
• Est. Priority Date: 05/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Lei, Ma, Wei-Ying, Xin-Jing, Wang
• Granted Patent: US 9,678,992 B2
• Time in Patent Office: Days
• US Class Current: 707/728
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Techniques are described for online real time text to image translation suitable for virtually any submitted query. Semantic classes and associated analogous items for each of the semantic classes are determined for the submitted query. One or more requests are formulated that are associated with analogous items. The requests are used to obtain web based images and associated surrounding text. The web based images are used to obtain associated near-duplicate images. The surrounding text of images is analyzed to create high-quality text associated with each semantic class of the submitted query. One or more query dependent classifiers are trained online in real time to remove noisy images. A scoring function is used to score the images. The images with the highest score are returned as a query response.

Claims

1. A method comprising:

under control of one or more processors configured with executable instructions;

receiving a search query;

determining semantic classes applicable to the search query;

identifying analogous terms associated with each of the semantic classes;

obtaining web images corresponding to a subset of the analogous terms associated with each of the semantic classes;

obtaining near-duplicate images of a subset of the web images associated with each of the semantic classes;

determining representative text associated with the near-duplicate images;

training at least one query-dependent classifier based in part on the representative text;

creating a denoised image set for each of the semantic classes by employing the at least one query-dependent classifier;

selecting at least one representative image from each denoised image set as an image answer for the search query; and

serving the image answer for presentation to a user as a response to the search query.

2. The method of claim 1, the method further comprising determining the subset of the analogous terms associated with each of the semantic classes by:

ranking the analogous terms based in part on a relevance to an associated one of the semantic classes; and

selecting the subset of the analogous terms based on the ranking the analogous terms.

3. The method of claim 2, the obtaining web images comprising:

formulating requests, each formulated request having a unique analogous term of the subset of the analogous terms; and

submitting the formulated requests to one or more search engines to obtain the web images.

4. The method of claim 1, the method further comprising determining the subset of the web images associated with each of the semantic classes by:

ranking the web images associated with a corresponding one of the semantic classes; and

selecting the subset of the web images for each of the semantic classes based on the ranking the web images.

5. The method of claim 4, the method further comprising determining a number of near-duplicate images available for each of the web images, the ranking the web images being based in part on the number of near-duplicate images associated with each of the web images.

6. The method of claim 4, the ranking the web images being based in part on an analysis of surrounding text associated with each of the web images.

7. The method of claim 1, the method further comprising:

compiling surrounding text of the near-duplicate images associated with each of the semantic classes into a corresponding training document;

removing stop words from each training document; and

removing terms having a frequency of occurrence that is less than a predetermined threshold from each training document.

8. The method of claim 7, the analysis of surrounding text comprising:

using at least one of a chi-squared scheme, a term frequency-inverse document frequency (td-idf) scheme or a term frequency scheme to weight textual features in each training document; and

selecting the representative text based in part on the weight associated with the textual features in each training document.

9. The method of claim 1, the selecting the at least one representative image including:

ranking images in each denoised image set based in part on a relevance score that measures a similarity of each image with an associated semantic class and a confidence score proportional to a number of nearest neighbors associated with each image; and

selecting the at least one representative image from each denoised image set based on the ranking images in each denoised image set.

10. A system comprising:

memory;

one or more processors communicatively coupled to the memory;

modules stored on the memory and executed by the one or more processors, the modules comprising;

a query expansion module to;

receive a search query;

obtain semantic classes for the search query;

identify analogous terms associated with each of the semantic classes and the search query; and

formulate requests associated with a subset of the analogous terms;

an image collection module to collect web based images corresponding to the requests; and

a near-duplicate extraction module to obtain near-duplicate images for a subset of the web based images;

a local classifier training module to;

train at least one query-dependent classifier or multiple one-on-one query-dependent classifiers based in part on representative text associated with the near-duplicate images;

and

create a denoised image set for each of the one or more semantic classes by employing the at least one query-dependent classifier or multiple one-on-one query-dependent classifiers; and

a ranking module to select at least one representative image from each denoised image set as an image answer for the search query.

11. The system of claim 10, the query expansion module further being to:

determine the subset of the analogous terms associated with a corresponding one of the semantic classes by ranking the analogous terms based in part on a relevance to the corresponding one of the semantic classes; and

select the subset of the analogous terms based on the ranking the analogous terms.

12. The system of claim 10, the near-duplicate extraction module further being to determine a number of available near-duplicate images associated with each of the web based images.

13. The system of claim 12, the image collection module further being to rank the web based images based in part on the number of the available near-duplicate images associated with each of the web images.

14. The system of claim 13, the near-duplicate extraction module to determine the subset of the web based images based of the rank of the web based images.

15. The system of claim 10, the system further comprising an image feature extraction module to format the web based images.

16. The system of claim 15, the near-duplicate extraction module to obtain the near-duplicate images for the subset of the web based images having the format.

17. The system of claim 10, the system further comprising:

a representative text determination module to determine the representative text based in part on an analysis of surrounding text associated with the near-duplicate images; and

a heuristics module to;

analyze terms in the requests to identify terms that result in a semantic gap between text and image domains; and

replace, modify or remove identified terms in the requests to reduce the semantic gap.

18. The system of claim 10, the ranking module further being to:

use a scoring function to rank each image in each denoised image set based in part on a relevance score that measures a similarity of each image of an associated denoised image set with its corresponding semantic class, and a confidence score proportional to a number of nearest neighbors associated with each image of an associated denoised image set; and

determine the at least one representative image from each denoised image set based on the rank of each image in each denoised image set.

19. The system of claim 18, the relevance score measures the similarity based in part on a cosine similarity between the representative text of each image of an associated denoised image set and a corresponding one of the semantic classes.

20. One or more computer-readable storage media storing computer-executable instructions that, when executed by at least one processor of a computing device, configure the at least one processor to perform acts comprising:

receiving a search query;

determining semantic classes of the query;

identifying analogous terms associated with each of the semantic classes and the search query;

for each of the semantic classes of the query;

ranking each of the analogous terms to create ranked analogous terms;

formulating requests, each request including at least one of a subset of the ranked analogous terms;

replacing, modifying or removing terms in the requests to minimize a semantic gap;

obtaining web images corresponding to the requests;

obtaining near-duplicate images for a subset of the web images; and

determining representative text associated with the near-duplicate images;

training at least one query-dependent classifier based in part on the representative text after the receiving of the search query;

creating a denoised image set for each of the semantic classes by employing the at least one query-dependent classifier;

selecting at least one representative image from each denoised image set as an image answer for the search query based in part on a relevance score and a confidence score; and

providing the image answer in response to the search query.