METHOD AND APPARATUS FOR AUTOMATIC MULTIMEDIA NARRATIVE ENRICHMENT

• US 20080306925A1
• Filed: 06/07/2007
• Published: 12/11/2008
• Est. Priority Date: 06/07/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Smith, John R., Naphade, Milind R., Li, Chung-Sheng, Xie, Lexing, Campbell, Murray S., Hill, Matthew L., Ebadollahi, Shahram, Tesic, Jelena, Natsev, Apostol Ivanov, Landry, Brian Michael
• Granted Patent: US 7,818,329 B2
• Time in Patent Office: Days
• US Class Current: 707/4
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A method and apparatus is disclosed that receives an input narrative in a particular media format, and produces therefrom a multimedia enriched version of the input narrative. In one embodiment, a method is provided for enriching an input narrative with multimedia content, wherein the method includes the step of selectively segmenting the input narrative to form queries pertaining to different portions or elements of the input narrative. The method further comprises retrieving multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of the queries, and selecting a subset of the retrieved multimedia artifacts, wherein the subset includes artifacts comprising different types of multimedia content. The input narrative is then combined with respective multimedia artifacts of the subset, in order to provide an enriched and enhanced narrative.

Claims

1. A method for enriching an input narrative with multimedia content, said method comprising the steps of:

selectively segmenting said input narrative, to form queries pertaining to different segmented elements of said input narrative;

using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries;

selecting a subset of said retrieved multimedia artifacts, wherein said subset includes artifacts comprising different types of multimedia content; and

combining said input narrative with respective multimedia artifacts of said subset to provide an enriched narrative.

2. The method of claim 1, wherein:

said artifacts retrieved from said one or more multimedia repositories include artifacts comprising two or more different types of multimedia content.

3. The method of claim 1, wherein:

said input narrative includes content material of a type included in a group that comprises at least textual, video, audio, pictorial and graphic content material.

4. The method of claim 1, wherein:

said input narrative is segmented into elements, wherein at least one element is a type of element included in a group that comprises at least story units, episodes, paragraphs, combination of words, video shots, events, activities and concepts.

5. The method of claim 1, wherein:

said segmenting step comprises parsing said input narrative and extracting tokens therefrom, wherein each token comprises a query associated with one of said narrative elements.

6. The method of claim 5, wherein:

said tokens are sent to one or more multimedia repositories in order to retrieve artifacts pertaining to their respective associated queries.

7. The method of claim 1, wherein:

each of said queries is processed to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures.

8. The method of claim 1, wherein:

said selecting step comprises automatically selecting respective retrieved artifacts for said subset.

9. The method of claim 1, wherein:

retrieved artifacts that match a particular segmented element of the narrative are placed on a ranked list of matches for the particular segmented element.

10. The method of claim 9, wherein:

said selecting step comprises selecting the best match for the particular segmented element from the corresponding ranked list, and placing the best match into said subset.

11. The method of claim 1, wherein:

said selecting step includes a filtering procedure to ensure that no multimedia artifact in said subset is identical to any other artifact contained therein.

12. The method of claim 1, wherein said selecting step is performed on the basis of semantics associated with temporal ordering.

13. A computer program product in a computer readable medium for enriching an input narrative with multimedia content, said method comprising:

first instructions for selectively segmenting said input narrative, to form queries pertaining to different segmented elements of said input narrative;

second instructions for using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries;

third instructions for selecting a subset of said retrieved multimedia artifacts, wherein said subset includes artifacts comprising different types of multimedia content; and

fourth instructions for combining said input narrative with respective multimedia artifacts of said subset to provide an enriched narrative.

14. The computer program product of claim 13, wherein:

fifth instructions are provided for processing each of said queries to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures.

15. The computer program product of claim 13, wherein:

sixth instructions are provided for automatically selecting respective retrieved artifacts for said subset.

16. The computer program product of claim 13, wherein:

seventh instructions are provided for placing retrieved artifacts that match a particular segmented element of the narrative on a ranked list of matches for the particular segmented element, and the best match for the particular segmented element from the corresponding ranked list is selected and placed into said subset.

17. An apparatus for enriching an input narrative with multimedia content comprising:

a first device for selectively segmenting said input narrative, to form queries pertaining to different segmented elements of said input narrative;

a second device for using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries;

a third device for selecting a subset of said retrieved multimedia artifacts, wherein said subset includes artifacts comprising different types of multimedia content; and

a fourth device for combining said input narrative with respective multimedia artifacts of said subset to provide an enriched narrative.

18. The apparatus of claim 17, wherein:

means are provided for processing each of said queries to retrieve artifacts by using a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures.

19. The apparatus of claim 17, wherein:

means are provided for automatically selecting respective retrieved artifacts for said subset.

20. The apparatus of claim 17, wherein:

means are provided for placing retrieved artifacts that match a particular segmented element of the narrative on a ranked list of matches for the particular segmented element, and selecting the best match for the particular segmented element from the corresponding ranked list, and placing the best match into said subset.