Method and apparatus for automatic multimedia narrative enrichment

• US 7,818,329 B2
• Filed: 06/07/2007
• Issued: 10/19/2010
• Est. Priority Date: 06/07/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Landry, Brian Michael, Smith, John R., Naphade, Milind R., Li, Chung-Sheng, Xie, Lexing, Campbell, Murray S., Hill, Matthew L., Ebadollahi, Shahram, Tesic, Jelena, Natsev, Apostol Ivanov
• Primary Examiner(s): FILIPCZYK, MARCIN R
• Application Number:
  US11/759,654
• Publication Number:
  US 20080306925A1
• Time in Patent Office: 1,230 Days
• Field of Search: 707/1, 707/3, 707/6, 707/10, 707/755, 707/791, 707999003-999006, 707/999.102, 715/256
• US Class Current: 707/755
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A method and apparatus is disclosed that receives an input narrative in a particular media format, and produces therefrom a multimedia enriched version of the input narrative. In one embodiment, a method is provided for enriching an input narrative with multimedia content, wherein the method includes the step of selectively segmenting the input narrative to form queries pertaining to different portions or elements of the input narrative. The method further comprises retrieving multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of the queries, and selecting a subset of the retrieved multimedia artifacts, wherein the subset includes artifacts comprising different types of multimedia content. The input narrative is then combined with respective multimedia artifacts of the subset, in order to provide an enriched and enhanced narrative.

Claims

1. A method for enriching an input narrative with multimedia content, said method comprising the steps of:

receiving said input narrative, wherein said input narrative comprises a description of an event in the form of text;

selectively segmenting said input narrative, to form queries using text from the different text segments of said input narrative;

using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries; and

wherein each of said queries is processed to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures;

selecting a subset of said retrieved multimedia artifacts, wherein said retrieved artifacts of said subset include one or more nontextual artifacts that each does not include any of said segments of said input narrative; and

combining said input narrative with each of said retrieved multimedia artifacts of said subset in an arrangement specified by a user, in order to provide an enriched narrative.

2. The method of claim 1, wherein:

said retrieved multimedia artifacts are selected for said subset in accordance with pre-specified artifact selection criteria provided by a user.

3. The method of claim 2, wherein:

said pre-specified artifact selection criteria includes at least time and place information respectively associated with said received narrative, and further includes the types of multimedia artifacts to select for said subset.

4. The method of claim 1, wherein:

said specified arrangement comprises selectively following or preceding each of said input narrative segments with a corresponding retrieved multimedia artifact of said subset.

5. The method of claim 1, wherein:

said segmenting step comprises parsing said input narrative and extracting tokens therefrom, wherein each token comprises a query associated with one of said narrative elements.

6. The method of claim 5, wherein:

said tokens are sent to one or more multimedia repositories in order to retrieve artifacts pertaining to their respective associated queries.

7. The method of claim 1, wherein:

said selecting step comprises automatically selecting respective retrieved artifacts for said subset.

8. The method of claim 1, wherein:

retrieved artifacts that match a particular segmented element of the narrative are placed on a ranked list of matches for the particular segmented element.

9. The method of claim 8, wherein:

said selecting step comprises selecting the best match for the particular segmented element from the corresponding ranked list, and placing the best match into said subset.

10. The method of claim 1, wherein:

said selecting step includes a filtering procedure to ensure that no multimedia artifact in said subset is identical to any other artifact contained therein.

11. The method of claim 1, wherein said selecting step is performed on the basis of semantics associated with temporal ordering.

12. A computer program product in a computer readable medium for enriching an input narrative with multimedia content, said method comprising:

instructions for receiving said input narrative, wherein said input narrative comprises a description of an event in the form of text;

instructions for selectively segmenting said input narrative, to form queries using text from different text segments of said input narrative;

instructions for using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries; and

wherein each of said queries is processed to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures;

instructions for selecting a subset of said retrieved multimedia artifacts, wherein said retrieved artifacts of said subset include one or more nontextual artifacts that each does not include any of said segments of said input narrative; and

instructions for combining said input narrative with each of said retrieved multimedia artifacts of said subset in an arrangement specified by a user, in order to provide an enriched narrative.

13. The computer program product of claim 12, wherein:

additional instructions are provided for processing each of said queries to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures.

14. The computer program product of claim 12, wherein:

additional instructions are provided for automatically selecting respective retrieved artifacts for said subset.

15. The computer program product of claim 12, wherein:

additional instructions are provided for placing retrieved artifacts that match a particular segmented element of the narrative on a ranked list of matches for the particular segmented element, and the best match for the particular segmented element from the corresponding ranked list is selected and placed into said subset.

16. An apparatus for enriching an input narrative with multimedia content comprising:

a device for receiving said input narrative, wherein said input narrative comprises a description of an event in the form of text;

a device for selectively segmenting said input narrative, to form queries using text from different text segments of said input narrative;

a device for using said queries to retrieve multimedia artifacts from one or more multimedia repositories, wherein each retrieved multimedia artifact is associated with one of said queries; and

wherein each of said queries is processed to retrieve artifacts by means of a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures;

a device for selecting a subset of said retrieved multimedia artifacts, wherein said retrieved artifacts of said subset include one or more nontextual artifacts that each does not include any of said segments of said input narrative; and

a device for combining said input narrative with each of said retrieved multimedia artifacts of said subset in an arrangement specified by a user, in order to provide an enriched narrative.

17. The apparatus of claim 16, wherein:

means are provided for processing each of said queries to retrieve artifacts by using a procedure selected from a group of procedures that include at least content-based, text-based, speech-based and model-based retrieval procedures.

18. The apparatus of claim 16, wherein:

means are provided for automatically selecting respective retrieved artifacts for said subset.

19. The apparatus of claim 16, wherein:

means are provided for placing retrieved artifacts that match a particular segmented element of the narrative on a ranked list of matches for the particular segmented element, and selecting the best match for the particular segmented element from the corresponding ranked list, and placing the best match into said subset.