Apparatus and methods for semantic representation and retrieval of multimedia content

• US 20040111432A1
• Filed: 12/10/2002
• Published: 06/10/2004
• Est. Priority Date: 12/10/2002
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Smith, John Richard, Lin, Ching-Yung, Naphade, Milind R., Tseng, Belle L., Adams, Hugh William Jr., Iyengar, Giridharan, Neti, Chalapathy Venkata, Nock, Harriet Jane
• Application Number:
  US10/315,334
• Publication Number:
  US 20040111432A1
• Time in Patent Office: Days
• US Class Current: 707/104.1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/73 Querying
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content

Abstract

An apparatus and method for analyzing multimedia content to identify the presence of audio, visual and textual cues that together correspond to one or more high-level semantics are provided. The apparatus and method make use of one or more analysis models that are trained to analyze audio, visual and textual portions of multimedia content to generate scores associated with the audio, visual and textual portions with respect to various high-level semantic concepts. These scores are used to generate a vector of scores. The apparatus is trained with regard to relationships between audio, visual and textual scores to thereby take the vector of scores generated for the multimedia content and classify the multimedia content into one or more high-level semantic concepts. Based on the scores for the various audio, video and textual portions of the multimedia content, a level of certainty regarding the high-level semantic concepts may be generated. These high-level semantic concepts are then used to generate one or more labels for the multimedia content that may be used to retrieve the multimedia content using a conceptual search engine. These semantic concept labels and their associated certainty levels may be stored in a file, associated with the multimedia content, for use in retrieving the multimedia content using the conceptual search engine.

Claims

1. A method of representing multimedia content, comprising:

performing feature extraction on one or more modalities of the multimedia content to extract one or more features of the multimedia content;

identifying one or more generic cues based on the one or more extracted features;

identifying a semantic based on a combination of the one or more generic cues; and

generating a model for the multimedia content based on the identified semantic.

2. The method of claim 1, wherein the one or more modalities include at least one of audio, visual, and textual modalities.

3. The method of claim 1, wherein generating a model for the multimedia content based on the identified semantic includes:

identifying one or more searchable labels based on the semantic; and

storing the one or more labels in a data structure associated with the multimedia content.

4. The method of claim 1, wherein identifying a semantic based on a combination of the one or more generic cues includes:

identifying a plurality of semantics based on the one or more generic cues;

identifying a confidence measure associated with each semantic in the plurality of semantics; and

selecting one or more semantics based on the confidence measure associated with the one or more semantics.

5. The method of claim 4, wherein selecting one or more semantics includes selecting only a semantic having a highest confidence measure.

6. The method of claim 4, wherein selecting one or more semantics includes selecting a subset of semantics in the plurality of semantics.

7. The method of claim 3, further comprising:

storing a confidence measure for each of the searchable labels in association with the searchable labels in the data structure.

8. The method of claim 1, wherein identifying one or more generic cues based on the one or more extracted features includes using at least one of a rule based system, expert system, and a neural network to identify the one or more generic cues based on an internal model generated through training of the rule based system, expert system or neural network.

9. A computer program product in a computer readable medium for representing multimedia content, comprising:

first instructions for performing feature extraction on one or more modalities of the multimedia content to extract one or more features of the multimedia content;

second instructions for identifying one or more generic cues based on the one or more extracted features;

third instructions for identifying a semantic based on a combination of the one or more generic cues; and

fourth instructions for generating a model for the multimedia content based on the identified semantic.

10. The computer program product of claim 9, wherein the one or more modalities include at least one of audio, visual, and textual modalities.

11. The computer program product of claim 9, wherein the fourth instructions for generating a model for the multimedia content based on the identified semantic include:

instructions for identifying one or more searchable labels based on the semantic; and

instructions for storing the one or more labels in a data structure associated with the multimedia content.

12. The computer program product of claim 9, wherein the third instructions for identifying a semantic based on a combination of the one or more generic cues include:

instructions for identifying a plurality of semantics based on the one or more generic cues;

instructions for identifying a confidence measure associated with each semantic in the plurality of semantics; and

instructions for selecting one or more semantics based on the confidence measure associated with the one or more semantics.

13. The computer program product of claim 12, wherein the instructions for selecting one or more semantics include instructions for selecting only a semantic having a highest confidence measure.

14. The computer program product of claim 12, wherein the instructions for selecting one or more semantics include instructions for selecting a subset of semantics in the plurality of semantics.

15. The computer program product of claim 11, further comprising:

instructions for storing a confidence measure for each of the searchable labels in association with the searchable labels in the data structure.

16. The computer program product of claim 9, wherein the second instructions for identifying one or more generic cues based on the one or more extracted features include instructions for using at least one of a rule based system, expert system, and a neural network to identify the one or more generic cues based on an internal model generated through training of the rule based system, expert system or neural network.

17. An apparatus for representing multimedia content, comprising:

means for performing feature extraction on one or more modalities of the multimedia content to extract one or more features of the multimedia content;

means for identifying one or more generic cues based on the one or more extracted features;

means for identifying a semantic based on a combination of the one or more generic cues; and

means for generating a model for the multimedia content based on the identified semantic.

18. A method of searching for multimedia content, comprising:

providing an interface for entering a search request, wherein the interface includes a field for entering a search term and a field for designating a modality corresponding to the search term;

receiving a search request from a client device via the interface, wherein the search request includes a search term and a corresponding modality;

searching a data structure of multimedia content models based on the identified search term and corresponding modality; and

returning results of searching the data structure to the client device.

19. The method of claim 18, wherein the modality is one of audio, video and text.

20. The method of claim 18, wherein the multimedia content models in the data structure include one or more searchable labels generated based on a semantic representation of the multimedia content.

21. The method of claim 20, wherein the semantic representation of the multimedia content is generated based on generic cues obtained from features extracted from the multimedia content.

22. The method of claim 18, wherein searching a data structure of multimedia content models based on the identified search term and corresponding modality includes comparing the search term and corresponding modality to searchable labels stored in the multimedia content models.

23. A computer program product in a computer readable medium for searching for multimedia content, comprising:

first instructions for providing an interface for entering a search request, wherein the interface includes a field for entering a search term and a field for designating a modality corresponding to the search term;

second instructions for receiving a search request from a client device via the interface, wherein the search request includes a search term and a corresponding modality;

third instructions for searching a data structure of multimedia content models based on the identified search term and corresponding modality; and

fourth instructions for returning results of searching the data structure to the client device.

24. The computer program product of claim 23, wherein the modality is one of audio, video and text.

25. The computer program product of claim 23, wherein the multimedia content models in the data structure include one or more searchable labels generated based on a semantic representation of the multimedia content.

26. The computer program product of claim 25, wherein the semantic representation of the multimedia content is generated based on generic cues obtained from features extracted from the multimedia content.

27. The computer program product of claim 23, wherein the third instructions for searching a data structure of multimedia content models based on the identified search term and corresponding modality include instructions for comparing the search term and corresponding modality to searchable labels stored in the multimedia content models.