TEXTUAL QUERY BASED MULTIMEDIA RETRIEVAL SYSTEM

• US 20120179704A1
• Filed: 09/16/2010
• Published: 07/12/2012
• Est. Priority Date: 09/16/2009
• Status: Abandoned Application

• Current Assignee: Nanyang Technological University
• Original Assignee: Nanyang Technological University
• Inventors: Xu, Dong, Tsang, Wai Hung, Liu, Yiming
• Application Number:
  US13/496,447
• Publication Number:
  US 20120179704A1
• Time in Patent Office: Days
• US Class Current: 707/766
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/483 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text

Abstract

A system and method are proposed for identifying multimedia files in a first database which are related to a textual term specified by a user. The textual term is used to search a second database of multimedia files, each of which is associated with a portion of text. The “second database” is usually composed of files from the databases of a very large number of servers connected via the internet. The multimedia files identified in the search are ones for which the corresponding associated text is relevant to the textual term. The identified multimedia files are used to generate a classifier engine. The classifier engine is then applied to the first database of multimedia files, thereby retrieving multimedia files in the first database which are relevant to the textual term. The user can optionally specify whether the retrieved multimedia files are relevant or not, and this permits a feedback process to improve the classifier engine.

Claims

1. A method of searching a first database of multimedia files, the method comprising:

(i) receiving from a user data specifying at least one textual term;

(ii) using the textual term to search a second database of multimedia files, each multimedia file in the second database being associated with respective text, said search identifying a first set of multimedia files in the second database for which the respective text is related to the textual term and said second database being different from said first database;

(iii) constructing a first multimedia file classifier engine using the first set of multimedia files; and

(iv) searching the first database of multimedia files using the first multimedia file classifier engine, thereby identifying one or more multimedia files in the first database related to the textual term;

wherein the method further includes at least once performing the steps of;

(a) receiving from a user relevance data which specifies, for each of a second set of one or more multimedia files in the first database, an indication of whether the second set of multimedia files are respectively related to the textual term; and

(b) using the relevance data to modify the first multimedia file classifier engine to form a modified classifier engine, and repeating said step (iv) using the modified classifier engine.

2. The method of claim 1, further including a step of using the textual term to search the second database of multimedia files to identify at least one third set of multimedia files, the third set of multimedia files being multimedia files for which the respective associated text is unrelated to the textual term,

said third set of multimedia files being used in said step (iii) of constructing said first multimedia file classifier engine.

3. The method of claim 2 in which said step of using the textual term to search the second database to identify the third set of multimedia files includes the sub-steps of:

consulting a lexical database using the textual term to obtain an enlarged group of textual terms; and

searching the second database for multimedia files for which the associated text does not include any of the enlarged group of textual terms.

4. The method of claim 1 in which the second database contains indexing data which indicates, for each of a plurality of predefined textual terms, those multimedia files in the second database for which the associated text includes the corresponding one of the predefined textual terms.

5. The method of claim 1 in which the first multimedia file classifier engine is arranged, upon operating on a multimedia file, to generate a numerical relevance value indicative of the relevance of the multimedia file to the textual term, the method further including ranking at least some of the multimedia files in the first database according to the corresponding relevance values.

6. The method of claim 2, further comprising:

selecting a plurality of said third sets of multimedia files;

for each of the third sets of multimedia files, constructing a corresponding non-linear function using that third set of multimedia files and also the first set of multimedia files;

and generating the first multimedia file classifier engine as a sum of the non-linear functions.

7. The method of claim 1 in which the first multimedia file classifier engine comprises an ensemble of decision stumps, each decision stump, when applied to a certain multimedia file, generating a non-linear output indicative of the presence of a respective characteristic in that multimedia file,

the first multimedia file classifier engine combining the outputs of the decision stumps to generate a numerical value.

8. The method of claim 1 in which the first multimedia file classifier engine comprises a linear and/or non-linear function of a product of a weight vector composed of weights, and a vector representing a multimedia file input to the first multimedia file classifier engine.

10. The method of claim 5 which includes presenting to the user a plurality of multimedia files from the first database having a high ranking according to the corresponding relevance values, the relevance data relating to one or more of said plurality of multimedia files from the first database.

11. The method of claim 1 in which the first file classifier engine is modified by at least one of the following sub-steps:

(I) training an adaptive system using the relevance data and the third second set of multimedia files, the modified classifier engine generating an output by combining an output generated by the first multimedia file classifier engine with an output generated by the adaptive system;

or

(II) generating a set of weight values defining the modified classifier engine, the set of weight values being generated to minimize a cost function including a term indicating disparity between the outputs of the modified classifier engine when operating on the second set of multimedia files and the corresponding relevance data.

12. The method of claim 11 in which, in sub-step (II) the cost function further includes a term indicative of the disparity between the outputs of the modified classifier engine and the corresponding outputs of the first multimedia file classifier engine when respectively operating on multimedia files in the first database which are not included in the second set of multimedia files.

13. The method of claim 11 in which the weight values are generated using a closed form expression which is a function of a set of data structures, steps (a) and (b) being performed repeatedly,

and, in each step (b), sub-step (II) comprising updating the data structures using the second set of multimedia files specified by the relevance data obtained in the preceding step (a).

9. (canceled)

14. A computer apparatus having a processor and a memory, the memory storing program instructions operative, when implemented by the processor, to cause the processor to search a first database of multimedia files, by:

(i) receiving from a user data specifying at least one textual term;

(ii) using the textual term to search a second database of multimedia files, each multimedia file in the second database being associated with respective text, said search identifying a first set of multimedia files in the second database for which the respective text is related to the textual term and said second database being different from said first database;

(iii) constructing a first multimedia file classifier engine using the first set of multimedia files;

(iv) searching the first database of multimedia files using the first multimedia file classifier engine, thereby identifying one or more multimedia files in the first database related to the textual term;

and at least once performing the steps of;

(a) receiving from a user relevance data which specifies, for each of a second set of one or more multimedia files in the first database, an indication of whether the second set of multimedia files are respectively related to the textual term; and

(b) using the relevance data to modify the first multimedia file classifier engine to form a modified classifier engine, and repeating said step (iv) using the modified classifier engine.

15. A recording medium, such as a tangible recording medium, storing program instructions operative to cause a processor performing the instructions to search a first database of multimedia files, by:

(i) receiving from a user data specifying at least one textual term;

(ii) using the textual term to search a second database of multimedia files, each multimedia file in the second database being associated with respective text, said search identifying a first set of multimedia files in the second database for which the respective text is related to the textual term and said second database being different from said first database;

(iii) constructing a first multimedia file classifier engine using the first set of multimedia files;

(iv) searching the first database of multimedia files using the first multimedia file classifier engine, thereby identifying one or more multimedia files in the first database related to the textual term;

and at least once performing the steps of;

(a) receiving from a user relevance data which specifies, for each of a second set of one or more multimedia files in the first database, an indication of whether the second set of multimedia files are respectively related to the textual term; and

(b) using the relevance data to modify the first multimedia file classifier engine to form a modified classifier engine, and repeating said step (iv) using the modified classifier engine.