System and method for linking an audio stream with accompanying text material

• US 6,636,238 B1
• Filed: 04/20/1999
• Issued: 10/21/2003
• Est. Priority Date: 04/20/1999
• Status: Expired due to Term

• Current Assignee: Hulu LLC (The Walt Disney Company)
• Original Assignee: International Business Machines Corporation
• Inventors: Srinivasan, Savitha, Pass, Norman Jerome, Syeda-Mahood, Tanveer Fathima, Amir, Arnon, Petkovic, Dragutin, Ponceleon, Dulce Beatriz, Niblack, Carlton Wayne
• Primary Examiner(s): SAX, STEVEN PAUL
• Application Number:
  US09/295,076
• Time in Patent Office: 1,645 Days
• Field of Search: 345/730, 345/731, 345/732, 345/727, 345/728, 345/729, 345/709, 345/717, 345/719-726, 345/748, 345/749, 345/747, 345/742, 345/743, 345/802, 345/804-805, 345/808-809, 345/780, 707/10, 707/11, 707/104
• US Class Current: 715/730
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/447 Temporal browsing, e.g. timeline
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: G06F 16/685 using automatically derived transcript of audio data, e.g. lyrics speech recognition G10L15/00

Abstract

A system enables a user to query for key words and phrases a text document, such as a presentation slide file, and an associated audio stream, such as can be derived from an audio-video recording that is made of a presenter contemporaneously with the showing of the slides to an audience. A graphical user interface is presented in which query results for both the text document and the audio stream are displayed in a time-aligned format, to enable a user to easily and conveniently browse the text document and accompanying time-aligned audio stream based on the key words/phrases.

Claims

1. A computer implemented system for automatically associating audio from at least one audio source with text from at least one text document related to the audio, the system including a query system comprising:

means or accessing at least one text document;

means for accessing at least one audio stream generated contemporaneously with a display of at least a portion of the text document;

means for automatically associating at least portions of the audio with at least respective portions of the text document;

means for permitting a user to specify at least one query word or phrase using a computer input device;

means for presenting in summary form, on a computer output device, at least some occurrences if any of the word or phrase in the text document and the audio stream;

means for extracting at least one of;

text, and keywords, from the audio along with timing information representative of the temporal location of at least some of the text and keywords in the audio; and

means for extracting at least one of;

text, and keywords, from the text document along with position information representative of the position of at least some of the text and keywords in the text document.

2. The system of claim 1, wherein the means for presenting further presents timing information common to the text document and the audio stream.

3. The system of claim 2, wherein the means for presenting further presents at least one temporal relationship between occurrences of the word or phrase in the audio stream and occurrences of the word or phrase in the text document.

4. The system of claim 1, wherein the text document is a presentation file including pages of slides.

7. The system of claim 1, further comprising:

means for electronically storing the audio or a transcript thereof in a database;

means for electronically storing the text document in a database; and

means for receiving a user query, wherein the automatically associating means is executed before or after the query.

8. The system of claim 7, wherein the means for automatically associating includes means for associating at least a first portion of the text document with at least a first portion of the audio when both first portions include at least one key word in the user query.

9. The system of claim 7, wherein the associating means includes means for associating at least a first portion of the text document with at least a first portion of the audio when both first portions contain identical time stamps.

10. The system of claim 9, wherein the time stamps include at least one of:

discrete times, and discrete time periods.

11. The system of claim 1, wherein the text document includes at least one presentation slide.

12. The system of claim 1, wherein the means for presenting in summary form presents, based on relevance to the query word or phrase, at least summaries of the text document and the audio stream.

13. The system of claim 1, wherein the audio stream is from a recording having both audio and video, the recording being generated contemporaneously with a display of at least a portion of the text document.

14. The system of claim 13, wherein the means for presenting presents in summary form at least portions of the video.

5. A computer implemented system for automatically associating audio from at least one audio source with text from at least one text document related to the audio, the system including a query system comprising:

means for accessing at least one text document;

mean for accessing at least one audio stream generated contemporaneously with a display of at least a portion of the text document;

means for automatically associating at least portions of the audio with at least respective portions of the text document;

means for permitting a user to specify at least one query word or phrase using a computer input device;

means for presenting in summary form, on a computer output device, at least some occurrences if any of the word or phrase in the text document and the audio stream; and

means for determining, for at least portions of the text document, information representative of times when the portions were presented on a large screen display.

6. The system of claim 5, further comprising:

means for determining, for at least portions of the text document, information representative of times when the portions were removed from a large screen display.

15. A computer system, comprising:

a data store holding at least one audio stream and at least one text source, the audio stream being based on audio associated with the text source; and

a processor receiving a query for data in the audio stream or text source and in response enabling a user to access at least portions of tile audio stream and text source or symbols representative of one or more thereof simultaneously, wherein the processor extracts at least one of;

text, and keywords, from the audio stream along with timing information representative of temporal location of at least some of the text and keywords in the audio stream, the processor further extracting at least one of;

text, and keywords, from the text source along with position information representative of the position of at least some of the text and keywords in the text source.

16. The system of claim 15, wherein the audio stream represents audio generated contemporaneously with a display of the text source.

17. The system of claim 15, wherein the text source is at least one slide in a presentation.

18. The system of claim 15, wherein the processor associates at least portions of the audio stream with at least portions of the text source based on at least one of:

the objects containing common key words, and the objects having substantially the same time stamps.

19. The system of claim 18, wherein the processor associates at least portions of the audio stream with at least portions of the text source based on the objects having substantially the same time stamps.

20. The system of claim 19, wherein the time stamps include at least one of:

discrete times, and discrete time periods.

21. The system of claim 15, wherein the processor associates at least portions of the audio stream with at least portions of the text source based on the relevance of the portions to the query.

22. The system of claim 15, wherein the audio stream is from a recording having both audio and video, the recording being generated contemporaneously with a display of at least a portion of the text source.

23. The system of claim 22, wherein the processor presents in summary form at least portions of the video.

25. The computer program device of claim 19, wherein the method steps further comprise:

for at least portions of the text document, determining information representative of times when the portions were removed from a large screen display.

26. The computer program device of claim 25, wherein the method steps further comprise:

electronically storing the audio signal or a transcript thereof in a database;

electronically storing the text document in a database; and

receiving a user query, wherein the presenting step is undertaken before or after the query.

27. The computer program device of claim 26, wherein the presenting step is accomplished by associating at least a first portion of the text document with at least a first portion of the audio signal when both first portions include at least one key word in the user query.

28. The computer program device of claim 26, wherein the presenting step is accomplished by associating at least a first portion of the text document with at least a first portion of the audio signal when both first portions contain substantially identical time stamps.

29. The computer program device of claim 28, wherein the time stamps include at least one of:

discrete times, and discrete time periods.

24. A computer program product comprising:

a computer program storage device readable by a computer; and

a program on the program storage device and including program code elements embodying instructions executable by the computer for performing method steps for associating at least portions of a text document with at least portions of an audio signal or transcript of the audio signal, the audio signal having been generated contemporaneously with a large screen display of the text document, the method steps comprising;

simultaneously presenting, via a computer output device, at least symbols representative of the text document and audio signal to a user, such that the user can navigate between the text document and audio signal; and

for at least portions of the text document, determining information representative of times when the portions were presented on a large screen display;

extracting at least one of text and keywords, from the audio signal along with timing information representative of the temporal location of at least some of the text and keywords in the audio signal,

extracting at least one of text and keywords, from the text document along with position information representative of the position of at least some of the text and keywords in the text document,

and wherein linking the audio with the text document is accomplished by associating at least a first portion of the text document with at least a first portion of the audio when both first portions include at least one key word in a user query.

30. The computer program product of claim 24, wherein the symbols representative of the text document and audio signal are presented in ranked pairs.

31. The computer program product of claim 24, wherein the audio signal is from a recording having both audio and video, the recording being generated contemporaneously with a display of at least a portion of the text document.

32. The computer program product of claim 31, wherein the method steps include presenting, in summary form, at least portions of the video.

39. The computer-implemented method of claim 24, comprising presenting symbols representative of the text document and audio signal in ranked pairs.

33. A computer-implemented method for associating audio from at least one audio source with at least one text document relating to the audio, the text document having been presented contemporaneously with the generation of the audio, comprising:

linking the audio with the text document;

associating at least portions of the audio with at least respective portions of the text document such that associated portions can be presented simultaneously on a computer output device; and

for at least portions of the text document, determining information representative of times when the portions were presented on a large screen display;

extracting at least one of text and keywords, from the audio signal along with timing information representative of the temporal location of at least some of the text and keywords in the audio signal,

extracting at least one of text and keywords, from the text document along with position information representative of the position of at least some of the text and keywords in the text document,

and wherein linking the audio with the text document is accomplished by associating at least a first portion of the text document with at least a first portion of the audio when both first portions include at least one key word in a user query.

34. The method of claim 33, further comprising:

for at least portions of the text document, determining information representative of times when the portions were removed from a large screen display.

35. The method of claim 33, further comprising:

electronically storing the audio or a transcript thereof in a database;

electronically storing the text document in a database; and

receiving a user query, wherein the linking step is undertaken before or after the query.

36. The method of claim 35, wherein the linking step is accomplished by associating at least a first portion of the text document with at least a first portion of the audio when both first portions contain identical time stamps.

37. The method of claim 36, wherein the time stamps include at least one of:

discrete times, and discrete time periods.

38. The method of claim 35, wherein the text document includes at least one presentation slide.

40. The computer-implemented method of claim 33, wherein the audio is from a recording having both audio and video, the recording being generated contemporaneously with a display of at least a portion of the text document, and further wherein the method steps include presenting, in summary form, at least portions of the video.