Multi-modal fusion in content-based retrieval

• US 7,610,306 B2
• Filed: 06/30/2003
• Issued: 10/27/2009
• Est. Priority Date: 06/30/2003
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Natsev, Apostol, Lin, Ching-Yung, Smith, John R., Naphade, Milind R., Tseng, Belle L.
• Primary Examiner(s): Jalil; Neveen Abel
• Assistant Examiner(s): RADTKE, MARK A
• Application Number:
  US10/611,235
• Publication Number:
  US 20040267774A1
• Time in Patent Office: 2,311 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

The present invention relates to the use of search fusion methods for querying multimedia databases and more specifically to a method and system for constructing a multi-modal query of a multimedia repository by forming multiple uni-modal searches and explicitly selecting fusion methods for combining their results. The present invention also relates to the integration of search methods for content-based retrieval, model-based retrieval, text-based retrieval, and metadata search, and the use of graphical user interfaces allowing the user to form queries fusing these search methods.

Claims

1. An apparatus for constructing and executing a fusion search for searching a multimedia database, said apparatus comprising:

a graphical user interface for forming a plurality of searches that produce intermediate search results, wherein a user builds a fusion search constructed as a text string interactively by sequentially choosing among descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of each individual search technique, and wherein each of the plurality of searches utilizes a different search technique, and wherein said search techniques consist of;

content-based retrieval which allows searching and matching based on perceptual similarity of multimedia content, wherein a user provides at least one example of multimedia content and result similarity is based on distance in feature space; and

model-based retrieval which allows searching based on automatically extracted concept labels produced by statistical models, wherein a user provides desired label text and matches are ranked using D=1−

C, with distance D derived from a confidence score C associated with each automatically assigned label;

an arrangement for a graphical user interface interactively integrating the plurality of searches to selectively direct each search to multiple individual search tools and provide desired matches from multiple disparate data sources based on the search technique and content type;

an input interface for permitting the selection of at least one fusion method to be used in combining a plurality of results of said searches the intermediate search results;

an arrangement for combining the intermediate search results of said searches interactively based on the intermediate search results of the searches until a final results list is achieved;

an arrangement for returning the plurality of results intermediate search results to a user; and

an arrangement for returning a fused result the final results list to a user;

wherein the plurality of intermediate search results of said searches include multimedia; and

wherein the apparatus is configured to execute a cascade of sequential operations, the cascade of sequential operations comprising conducting the plurality of searches that produce the intermediate search results, normalizing the intermediate search results, fusing the intermediate search results to produce fused results, and using the fused results as input for further normalization and fusion with subsequent intermediate results of a subsequent search until the final results list is achieved.

2. The apparatus according to claim 1, wherein said fusion method involves normalization of at least one of said plurality of intermediate search results, the normalization comprising one or more of invert, studentize, and range normalize operations.

3. The apparatus according to claim 1, wherein said fusion method involves aggregation of at least one of said plurality of intermediate search results, said aggregation comprising one or more of average, minimum, maximum, sum, and product operations.

4. A method for constructing and executing a fusion search for searching a multimedia database, said method comprising the steps of:

forming a plurality of searches that produce intermediate search results, wherein a user builds a fusion search constructed as a text string interactively by sequentially choosing among descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of each individual search technique, and wherein each of the plurality of searches utilizes a different search technique, and wherein said search techniques consist of;

content-based retrieval which allows searching and matching based on perceptual similarity of multimedia content, wherein a user provides at least one example of multimedia content and result similarity is based on distance in feature space; and

model-based retrieval which allows searching based on automatically extracted concept labels produced by statistical models, wherein a user provides desired label text and matches are ranked using D=1−

C, with distance D derived from a confidence score C associated with each automatically assigned label;

interactively integrating the plurality of searches to selectively direct each search to multiple individual search tools and provide desired matches from multiple disparate data sources based on the search technique and content type;

selecting at least one fusion method to be used in combining a plurality of the intermediate search results of said searches;

combining the plurality of intermediate search results of said searches interactively based on the intermediate results of the searches until a final results list is achieved;

returning the plurality of intermediate search results to a user; and

returning a fused result the final results list to a user;

wherein the plurality of intermediate search results of said searches include multimedia data; and

wherein combining the intermediate search results interactively further comprises executing a cascade of sequential operations, the cascade of sequential operations comprising conducting the plurality of searches that produce the intermediate search results, normalizing the intermediate search results, fusing the intermediate search results to produce fused results, and using the fused results as input for further normalization and fusion with subsequent intermediate results of a subsequent search until the final results list is achieved.

5. The method according to claim 4, wherein said fusion method involves normalization of at least one of said plurality of intermediate search results, the normalization comprising one or more of invert, studentize, and range normalize operations.

6. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for constructing and executing a fusion search for searching a multimedia database, said method comprising the steps of:

forming a plurality of searches that produce intermediate search results, wherein a user builds a fusion search constructed as a text string interactively by sequentially choosing among descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of each individual search technique, and wherein each of the plurality of searches utilizes a different search technique, and wherein said search techniques consist of;

content-based retrieval which allows searching and matching based on perceptual similarity of multimedia content, wherein a user provides at least one example of multimedia content and result similarity is based on distance in feature space; and

model-based retrieval which allows searching based on automatically extracted concept labels produced by statistical models, wherein a user provides desired label text and matches are ranked using D=1−

C, with distance D derived from a confidence score C associated with each automatically assigned label;

interactively integrating the plurality of searches to selectively direct each search to multiple individual search tools and provide desired matches from multiple disparate data sources based on the search technique and content type;

selecting at least one fusion method to be used in combining a plurality of the intermediate search results of said searches;

combining the plurality of intermediate search results of said searches interactively based on the intermediate results of the searches until the final results list is achieved;

returning the plurality of intermediate search results to a user; and

returning a fused result the final results list to a user;

wherein the plurality of intermediate search results of said searches includes include multimedia data; and

wherein combining the intermediate search results interactively further comprises executing a cascade of sequential operations, the cascade of sequential operations comprising conducting the plurality of searches that produce the intermediate search results, normalizing the intermediate search results, fusing the intermediate search results to produce fused results, and using the fused results as input for further normalization and fusion with subsequent intermediate results of a subsequent search until the final results list is achieved.