SYSTEM AND METHODS OF INTEGRATING VISUAL FEATURES AND TEXTUAL FEATURES FOR IMAGE SEARCHING

• US 20130080426A1
• Filed: 09/26/2012
• Published: 03/28/2013
• Est. Priority Date: 09/26/2011
• Status: Active Grant

• Current Assignee: University of Kansas
• Original Assignee: University of Kansas
• Inventors: Chen, Xue-wen, Luo, Bo
• Granted Patent: US 9,075,825 B2
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

The present invention includes a system and methods for image searching. Embodiments of the system integrate textual features and visual features for improved search performance. The system represents text terms in the visual feature space and develops a text-guided weighting scheme for visual features. The weighting scheme infers intention from query terms and enhances the visual features that are significant in light of such intention.

Claims

1. A system for image searching in a vertical content area to generate a ranked image search list, comprising:

a memory;

a display;

a processor connected to said memory and said database; and

a data input device configured to permit input one or more text queries into said memory;

said processor processing the one or more text queries by;

providing a ranked list of relevant items as an initial result set;

constructing an expanded query by integrating textual features and visual features for each result in the initial result set;

evaluating each expanded query to determine items that are similar in the ranked list of relevant items to generate the ranked image search list, wherein a weighted vector representing visual perception of one of the one or more text queries is enforced during the evaluating of each expanded query; and

displaying the ranked image search list on said display.

2. The system of claim 1, further comprising a database configured to store information.

3. The system of claim 1 further comprising a crawler configured to fetch web pages from the world wide web, wherein said crawler may be configured to fetch only certain types of web pages.

4. The system of claim 1, wherein the processor receives a text query as entered by a user through the data input device.

5. The system of claim 1, wherein the processor optimizes the weighted vector to obtain an optimized weighted vector.

6. The system of claim 1, wherein the processor produces a visual thesaurus using the weighted vector.

7. The system of claim 1, wherein one or more visual features including shapes, color, and texture of an image are extracted from images using an image processor.

8. The system of claim 1, wherein one or more textual features including a keyword associated with an image is identified using a parser.

9. The system of claim 1, wherein the one or more visual features obtained using the image processor are normalized to permit comparison among features that take values from different domains.

10. The system of claim 1, wherein the processor segments images to remove background information from the image.

11. A method for an improved search, comprising the steps of:

inputting a text query using a data input device;

crawling a network of information to obtain pages from the network, wherein each page includes an image and text information;

analyzing the image using an image processor to ascertain at least one visual feature including shape, color, or texture of an image;

assessing the text information using a parser to obtain a text feature including a keyword of the text information;

calculating a weight vector for each keyword by integrating the visual features and text features;

preparing an initial data set based using a text-based search method;

ranking the initial data set using the weight vector to obtain ranked items; and

displaying the ranked items in a user interface context shown in a display unit.

12. The method of claim 11, further comprising the step of creating a visual thesaurus for the keywords. 16.

13. The method of claim 12, further comprising the step of optimizing the weighted vector by associating keyword synonyms identified in the visual thesaurus with other keyword synonyms.

14. The method of claim 11, further comprising the step of segmenting the image to remove background information from the image.

15. The method of claim 11, further comprising the step of normalizing visual features to permit comparison with other visual features having values from different domains.

16. The method of claim 11, further comprising the steps of forming at least two feature distributions for each keyword and measuring the differences between the two feature distributions using a Kolmogorov-Smirnov test across each feature dimension.

17. The method of claim 11, wherein, if a text query includes more than one keyword, the weighted vector is computed by combining the weight vector for each keyword to construct a query weight vector, which is configured to signify intention of a user to obtain results in line with that intention.

18. The method of claim 11, further comprising the step of performing an evaluation of the ranked items compared to ranked items generated by CBIR systems and methods.

19. The method of claim 11, wherein at least one of a computationally expensive step is parallelized to improve performance.

20. The method of claim 11, further comprising the step of employing a supervised or unsupervised learning technique to derive semantic meaning in low level visual features, wherein the technique is chosen from the group of neural networks, support vector machines, Bayesian, or bootstrapping.