GENERATING SEARCH REQUESTS FROM MULTIMODAL QUERIES

• US 20090041366A1
• Filed: 10/08/2008
• Published: 02/12/2009
• Est. Priority Date: 09/21/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Wei-Ying, Xie, Xing, Li, Zhiwei, Li, Ming Jing, Fan, Xin
• Granted Patent: US 8,081,824 B2
• Time in Patent Office: Days
• US Class Current: 382/250
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and system for generating a search request from a multimodal query that includes a query image and query text is provided. The multimodal query system identifies images of a collection that are textually related to the query image based on similarity between words associated with each image and the query text. The multimodal query system then selects those images of the identified images that are visually related to the query image. The multimodal query system may formulate a search request based on keywords of web pages that contain the selected images and submit that search request to a search engine service.

Claims

1-20. -20. (canceled)

21. A method in a computing device for generating a compact representation of an image, the method comprising:

for each of a plurality of blocks of pixels of the image, calculating for the block a representative intensity level derived from intensities of the pixels of the block;

performing a discrete transform on the representative intensity levels for the blocks to generate transformed coefficients;

applying a principal component analysis to generate PCA coefficients of a reduced dimensionality; and

setting each bit of a vector with an element corresponding a PCA coefficient based on the value of the PCA coefficient

wherein the bits represent the compact representation of the image.

22. The method of claim 21 wherein the compact representation is a signature of the image.

23. The method of claim 22 including identifying images that are duplicates of a target image by comparing signatures of the images to the signature of the target image.

24. The method of claim 21 wherein the calculating of the representative intensity levels includes setting the pixels of the images to a grey scale and wherein the representative intensity level is an average intensity level for the pixels of a block.

25. The method of claim 21 wherein the transform is a discrete cosine transform and the transformed coefficients do not include a DC coefficient.

26. The method of claim 25 wherein the transformed coefficients are selected in a zigzag pattern.

27. The method of claim 21 wherein the calculating of the representative intensity levels includes setting the pixels of the images to a grey scale and wherein the representative intensity level is an average intensity level for the pixels of a block, wherein the discrete transform is a discrete cosine transform and the transformed coefficients are AC coefficients selected in a zigzag pattern, and wherein the bits are set based on whether the PCA coefficients are greater than or less than zero.

28. The method of claim 27 wherein the blocks include 64 pixels, wherein 48 AC coefficients are selected, and wherein the reduce dimensionality is 32.