Multi-modal fusion in content-based retrieval

• US 20040267774A1
• Filed: 06/30/2003
• Published: 12/30/2004
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Lin, Ching-Yung, Smith, John R., Naphade, Milind R., Tseng, Belle L., Natsev, Apostol
• Granted Patent: US 7,610,306 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

The present invention relates to the use of search fusion methods for querying multimedia databases and more specifically to a method and system for constructing a multi-modal query of a multimedia repository by forming multiple uni-modal searches and explicitly selecting fusion methods for combining their results. The present invention also relates to the integration of search methods for content-based retrieval, model-based retrieval, text-based retrieval, and metadata search, and the use of graphical user interfaces allowing the user to form queries fusing these search methods.

Claims

1. An apparatus for constructing a multi-modal query, said apparatus comprising:

an arrangement for forming a plurality of uni-modal searches;

an input interface for permitting the selection of at least one fusion method to be used in combining the results of said uni-modal searches; and

an arrangement for combining results of said uni-modal searches.

2. The apparatus according to claim 1, wherein said selection is accomplished through the use of a graphical user interface.

3. The apparatus according to claim 2, wherein the query is constructed as a text string.

4. The apparatus according to claim 3, wherein a user constructs said query by interactively and sequentially forming said searches while viewing intermediate results.

5. The apparatus according to claim 1, wherein said fusion method involves normalization of scores of at least one of said search results.

6. The apparatus according to claim 1, wherein said fusion method involves aggregation of scores of at least one of said search results.

7. The apparatus according to claim 1, wherein said plurality of uni-modal searches includes at least one uni-modal search selected from the group consisting of content-based retrieval, model-based retrieval, text-based retrieval, metadata search, structured database search, relational database search, or XML data search.

8. A method of constructing a multi-modal query, said method comprising the steps of:

forming a plurality of uni-modal searches;

selecting at least one fusion method to be used in combining the results of said uni-modal searches; and

combining results of said uni-modal searches.

9. The method according to claim 8, wherein said step of selecting is accomplished through the use of a graphical user interface.

10. The method according to claim 9, wherein the query is constructed as a text string.

11. The apparatus according to claim 10, wherein a user constructs said query by interactively and sequentially forming said searches while viewing intermediate results.

12. The method according to claim 8, wherein said fusion method involves normalization of scores of at least one of said search results.

13. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for constructing a multi-modal query, said method comprising the steps of:

forming a plurality of uni-modal searches;

selecting at least one fusion method to be used in combining the results of said uni-modal searches; and

combining results of said uni-modal searches.