Generating search requests from multimodal queries

• US 7,457,825 B2
• Filed: 09/21/2005
• Issued: 11/25/2008
• Est. Priority Date: 09/21/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Xie, Xing, Li, Zhiwei, Li, Ming Jing, Fan, Xin
• Primary Examiner(s): Ly; Cheyne D
• Application Number:
  US11/233,352
• Publication Number:
  US 20070067345A1
• Time in Patent Office: 1,161 Days
• Field of Search: 707/3
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and system for generating a search request from a multimodal query that includes a query image and query text is provided. The multimodal query system identifies images of a collection that are textually related to the query image based on similarity between words associated with each image and the query text. The multimodal query system then selects those images of the identified images that are visually related to the query image. The multimodal query system may formulate a search request based on keywords of web pages that contain the selected images and submit that search request to a search engine service.

Claims

1. A method in a device for generating a search request for a multimodal query with a query image and query text, the query image being stored in electronic form, the method comprising:

providing a collection of images and associated words;

generating a word-to-image index that maps words to associated images and an image-to-related-information index that maps images to associated keywords;

receiving a multimodal query that includes a query image and query text;

identifying images of the collection based on textual relatedness between an associated word and the query text;

wherein the images are identified by searching the word-to-image index to locate images with associated words that are related to the query text;

selecting images of the identified images based on visual relatedness between an identified image and the query image, wherein the selecting comprising extracting a feature vector for the identified image, determining the distance between the extracted feature vector and the feature vector of each image of the collection, and selecting the images based on the determined distance;

generating a search request based on keywords associated with the selected images, wherein the search request is generated from keywords of the image-to-related-information index;

submitting the generated search request to a search engine for identifying documents related to the multimodal query; and

providing an indication of the identified documents as a search result for the multimodal query.

2. The method of claim 1 wherein the collection includes a collection of web pages with images and words.

3. The method of claim 1 wherein visual relatedness is based on similarity in color space and wavelet coefficients.

4. The method of claim 1 including:

before identifying images of the collection,

determining whether the query image is a duplicate of an image of the collection; and

when the query image is a duplicate of an image, generating a search request based on a keyword associated with that image.

5. The method of claim 4 wherein the query image is a duplicate when the images are identical.

6. The method of claim 4 wherein the query image is a duplicate when the images are of the same content but from different points of view.

7. The method of claim 4 wherein the collection includes a signature-to-image index and wherein images are duplicates when they have the same signature.

8. The method of claim 1 wherein the query text is derived from audio information.

9. A computer-readable storage medium containing instructions for controlling a computer system to find images related to a multimodal query, the instructions perform a method comprising:

providing web pages with images;

generating a word-to-image index that maps words to associated images and an image-to-related-information index that maps images to associated keywords from the web pages with images;

receiving a query image and query text of the multimodal query;

identifying images of the web pages based on textual relatedness between a web page and the query text;

wherein the images are identified by searching the word-to-image index to locate images with associated words that are related to the query text;

selecting images of the identified images of the web pages based on visual relatedness between an identified image and the query image, wherein the selecting comprising extracting a feature vector for the identified image,

determining the distance between the extracted feature vector and the feature vector of each image of the collection, and selecting the images based on the determined distance;

generating a search request based on keywords associated with the selected images, wherein the search request is generated from keywords of the image-to-related-information index;

submitting the generated search request to a search engine for identifying documents related to the multimodal query; and

providing an indication of the identified documents as a search result for the multimodal query.

10. The computer-readable medium of claim 9 including:

before identifying web pages,

determining whether the query image is a duplicate of an image of a web page; and

when the query image is a duplicate of an image of a web page, generating a search request based on words of the web page that contains the duplicate image.

11. The computer-readable medium of claim 9 wherein visual relatedness is based on similarity in color space and wavelet coefficients.