System and methods of integrating visual features with textual features for image searching

• US 9,075,825 B2
• Filed: 09/26/2012
• Issued: 07/07/2015
• Est. Priority Date: 09/26/2011
• Status: Active Grant

• Current Assignee: University of Kansas
• Original Assignee: University of Kansas
• Inventors: Chen, Xue-wen, Luo, Bo
• Primary Examiner(s): MOSER, BRUCE M
• Application Number:
  US13/627,716
• Publication Number:
  US 20130080426A1
• Time in Patent Office: 1,014 Days
• Field of Search: 707/723
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

The present invention includes a system and methods for image searching. Embodiments of the system integrate textual features and visual features for improved search performance. The system represents text terms in the visual feature space and develops a text-guided weighting scheme for visual features. The weighting scheme infers intention from query terms and enhances the visual features that are significant in light of such intention.

Claims

1. A system for image searching in a vertical content area to generate a ranked image search list, comprising:

a memory;

a display;

a processor connected to the memory and the display; and

a data input device for receiving an input of a text query into the memory;

the processor processing the text query by;

collecting a set of information based on the text query, the set of information including one or more images and one or more text terms;

analyzing the one or more images of the set for one or more visual features;

determining one or more text features of the one or more text terms of the set;

obtaining keywords using the one or more visual features and the one or more text features;

assigning to each keyword a weight vector, the weight vector representing a visual perception of the one or more visual features and the one or more text features;

using the weight vector of each keyword to rank each visual feature of the one or more visual features and the one or more text features to obtain a ranked list of items, wherein a higher weight is given to a feature when the keyword is used as a text query; and

displaying the ranked list of items on the display.

2. The system of claim 1, further comprising a database configured to store the set of information including one or more images and one or more text terms.

3. The system of claim 1 further comprising a crawler configured to fetch one or more web pages from the world wide web, wherein the crawler may be configured to fetch only certain types of web pages.

4. The system of claim 1, wherein the processor receives the text query as entered by a user through the data input device.

5. The system of claim 1, wherein the processor optimizes the weight vector to obtain an optimized weight vector.

6. The system of claim 1, wherein the processor produces a visual thesaurus using the weight vector.

7. The system of claim 1 further comprising an image processor for extracting from the one or more images the one or more visual features including a shape, a color, and a texture.

8. The system of claim 7, wherein the one or more visual features obtained using the image processor are normalized to permit comparison among features that take values from different domains.

9. The system of claim 1 further comprising a parser for identifying from the one or more text terms the one or more text features including a description associated with the one or more images.

10. The system of claim 1, wherein the processor further analyzes the one or more images by segmenting the one or more images to remove background information.

11. A computer for image searching, the computer method comprising the steps of:

receiving by a processor a text query input using a data input device;

collecting by the processor set of information including one or more web pages, wherein each web page includes an image and a text term;

analyzing by an image processor the image to ascertain at least one visual feature including a shape, a color, or a texture of the image;

assessing by a parser the text term to obtain at least one text feature;

obtaining keywords using the at least one visual feature and the at least one text feature;

calculating by the processor a weight vector for each keyword, the weight vector representing a visual perception of the at least one visual feature and the at least one text feature;

ranking by the processor the at least one visual feature and the at least one text feature using the weight vector to obtain ranked items; and

displaying on a user interface the ranked items.

12. The computer method of claim 11 further comprising the step of creating by the processor a visual thesaurus for the keywords.

13. The computer method of claim 12 further comprising the step of optimizing by the processor the weight vector by associating keyword synonyms identified in the visual thesaurus with other keyword synonyms.

14. The computer method of claim 11 further comprising the step of segmenting by the processor the image to remove background information from the image.

15. The computer method of claim 11 further comprising the step of normalizing by the processor the at least one visual f turcsfeature to permit comparison with other visual features having values from different domains.

16. The computer method of claim 11 further comprising the step of forming by the processor at least two feature distributions for each keyword.

17. The computer method of claim 16, further comprising the step of measuring by the processor differences between the at least two feature distributions using a Kolmogorov-Smirnov test.

18. The computer method of claim 11, wherein the weight vector is a query weight vector calculated by combining the weight vector for each keyword when text query includes more than one keyword.

19. The computer method of claim 11 further comprising the step of performing by the processor an evaluation of the ranked items compared to ranked items generated by a content-based image retrieval (CBIR) technique.

20. The computer method of claim 11 further comprising the step of deriving by the processor semantic meaning of the at least one visual feature using a technique chosen from the group consisting of:

a neural network technique, a support vector machine technique, a Bayesian technique, or a bootstrapping technique.