Multimedia visualization and integration environment

• US 8,739,040 B2
• Filed: 11/04/2003
• Issued: 05/27/2014
• Est. Priority Date: 12/22/1997
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): DUONG, HIEN LUONGVAN
• Application Number:
  US10/701,966
• Publication Number:
  US 20040103372A1
• Time in Patent Office: 3,857 Days
• Field of Search: 715/764, 715/701, 715/702, 715/716, 715/787, 715/723, 715/835, 715/719, 715/724, 715/725, 725/61, 725/51, 725/52
• US Class Current: 715/723
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/34 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Embodiments of the present invention provide techniques for retrieving and displaying multimedia information. According to an embodiment of the present invention, a graphical user interface (GUI) is provided that displays multimedia information that may be stored in a multimedia document. According to the teachings of the present invention, the GUI enables a user to navigate through multimedia information stored in a multimedia document. The GUI provides both a focused and a contextual view of the contents of the multimedia document.

Claims

1. A computer-implemented method of displaying recorded information of a multimedia document, including audio information and video information, the method comprising:

displaying, in a first display area, a textual representation of the audio information associated with the multimedia document;

displaying, in a second display area, a pictorial representation of the video information associated with the multimedia document, the pictorial representation including one or more video keyframes, wherein the textual representation and the pictorial representation are temporally synchronized with each other along a timeline;

receiving a search term after displaying the textual and pictorial representations;

determining a plurality of locations in the textual representation that include the search term, each location having an associated time along the timeline;

identifying a first location and a second location from the plurality of locations;

determining that a first time associated with the first location and a second time associated with the second location satisfy a proximity criteria;

generating a first range encompassing the first and the second locations, wherein the first range comprises a portion of the multimedia document occurring between a first start time and a first end time along the timeline;

visually emphasizing the first range; and

visually distinguishing every occurrence of the search term within the textual representation from a remainder of the textual representation.

2. The method of claim 1 wherein:

if the recorded information comprises at least one of audio information or closed-caption text information;

the textual representation of the audio information comprises at least one of text information obtained from transcribing the audio information or text information obtained from the closed-caption text information; and

the one or more video keyframes are extracted from the video information.

3. The method of claim 1 further comprising:

displaying an identifier for each location from the plurality of locations.

4. The method of claim 1 further comprising:

generating a second range including one or more locations from the plurality of locations, wherein the second range comprises a portion of the multimedia document occurring between a second start time and a second end time along the timeline;

determining a time interval between the first range and the second range; and

combining the first range and the second range to form a third range if the time interval is less than a threshold value, wherein the third range comprises a portion of the multimedia document occurring between the first start time and the second end time along the timeline.

5. A computer-implemented method of displaying information of a multimedia document, including audio information and video information, the method comprising:

displaying a first thumbnail comprising text information obtained from transcribing the audio information associated with the multimedia document;

displaying a second thumbnail comprising a plurality of video keyframes extracted from the video information associated with the multimedia document, wherein the first thumbnail and the second thumbnail are temporally synchronized with each other along a timeline;

receiving from a user information identifying a user-provided criterion after displaying the first and the second thumbnails, the user-provided criterion comprising a topic of interest;

analyzing the text information to determine one or more portions of the audio information displayed in the first thumbnail, each of the one or more portions of the audio information has a start time and an end time along the timeline, and relates to the topic of interest;

analyzing the plurality of video keyframes to identify one or more video keyframes in the displayed second thumbnail that are relevant to the topic of interest, each video keyframe having an associated time along the timeline;

identifying a first section within the first thumbnail, the first section including a first subset of the one or more portions of the audio information that satisfy a proximity constraint, wherein the first section comprises a portion of the multimedia document occurring between a first start time and a first end time along the timeline;

visually indicating the first section; and

visually distinguishing each of the one or more portions of the audio information that relate to the topic of interest displayed in the first thumbnail from a remainder of the first thumbnail.

6. The method of claim 5 further comprising:

identifying a second section within the first thumbnail, the second section including a second subset of the one or more portions of the audio information that satisfy the proximity constraint, wherein the second section comprises a portion of the multimedia document occurring between a second start time and a second end time along the timeline;

determining that a time interval between the first section and the second section is less than a threshold value; and

combining the first section and the second section to form a third section, wherein the third section comprises a portion of the multimedia document occurring between the first start time and the second end time along the timeline.

7. A non-transitory computer-readable storage medium for storing computer code for displaying recorded information of a multimedia document comprising:

code for displaying a textual representation of audio information associated with the multimedia document in a first display area of a display and a pictorial representation of video information associated with the multimedia document in a second display area of the display, the pictorial representation including one or more video keyframes, wherein the textual representation and the pictorial representation are temporally synchronized with each other along a timeline;

code for receiving, from a user, information identifying a user-provided criterion after displaying the textual and the pictorial representations, the user-provided criterion comprising a search term;

code for determining one or more locations in the textual representation of the audio information using the user-provided criterion, wherein each location in the textual representation of the audio information includes an occurrence of the search term;

code for determining that a first location and a second location in the textual representation of the audio information satisfy a proximity criteria;

code for visually identifying a first section in the textual representation of the audio information, wherein the first section includes the first location and the second location and comprises a portion of the multimedia document occurring between an associated first start time and a first end time along the timeline; and

code for visually distinguishing every occurrence of the search term within the textual representation from a remainder of the textual representation.

8. The non-transitory computer-readable storage medium of claim 7 further comprising:

code for determining a second section in the textual representation of the audio information, wherein the second section includes (i) a third location and a fourth location in the textual representation of the audio information that satisfy the proximity criteria and (ii) comprises a portion of the multimedia document occurring between a second start time and a second end time along the timeline;

code for determining that a time interval between the first section and the second section is less than a threshold value; and

code for combining the first section and the second section to form a third section, wherein the third section comprises a portion of the multimedia document occurring between the first start time and the second end time along the timeline.

9. The non-transitory computer-readable storage medium of claim 8 further comprising:

code for visually distinguishing the first, the second, the third, and the fourth locations from a remainder of the textual representation of the audio information.

10. The non-transitory computer-readable storage medium of claim 8 further comprising:

code for displaying an identifier for each of the first, the second, the third, and the fourth locations.

11. The non-transitory computer-readable storage medium of claim 7 wherein:

the recorded information includes closed-caption text information;

the textual representation of the audio information comprises at least one of;

text information obtained from transcribing the audio information or text information obtained from the closed-caption text information; and

the one or more video keyframes are extracted from the video information.

12. A non-transitory computer-readable storage medium for storing computer code for displaying information of a multimedia document, including audio information and video information comprising:

code for displaying a first thumbnail comprising text information obtained from transcribing the audio information associated with the multimedia document;

code for displaying a second thumbnail comprising a plurality of video keyframes extracted from the video information associated with the multimedia document, wherein the first thumbnail and the second thumbnail are temporally synchronized with each other along a timeline;

code for receiving information from a user identifying a user-provided criterion, the user-provided criterion comprising a topic of interest;

code for analyzing the text information using the user-provided criterion to determine a plurality of locations within the text information, each location having a start time and an end time along the timeline and related to the topic of interest;

code for determining a first section within the text information, the first section including a first subset of locations from the plurality of locations that satisfy a proximity criteria, wherein the first section comprises a portion of the multimedia document occurring between a first start time and a first end time along the timeline;

code for visually indicating the first section; and

code for visually distinguishing each of the plurality of locations related to the topic of interest within the text information from a remainder of the text information.

13. The non-transitory computer-readable storage medium of claim 12 further comprising:

code for determining a second section within the text information, the second section including a second subset of locations from the plurality of locations that satisfy the proximity criteria, wherein the second section comprises a portion of the multimedia document occurring between a second start time and a second end time along the timeline;

code for determining that a time interval between the first section and the second section is less than a threshold value;

code for combining the first and the second sections to generate a third section, wherein the third section comprises a portion of the multimedia document occurring between the first start time and the second end time along the timeline; and

code for visually indicating the third section.

14. A method comprising:

receiving, by a computer system, multimedia information of a multimedia document, including audio information and video information;

displaying, by the computer system, in a first area, a textual representation of the audio information associated with the multimedia document, wherein the textual representation includes text that is generated by transcribing the audio information;

displaying, by the computer system, in a second area, a pictorial representation of the video information associated with the multimedia document, wherein the pictorial representation comprises a plurality of video frames extracted from the video information, wherein the textual representation and the pictorial representation are temporally synchronized with each other along a timeline;

receiving, by the computer system, a search term after displaying the textual and the pictorial representations;

determining, by the computer system, a plurality of locations within the textual representation that include the search term;

determining, by the computer system, a first section within the textual representation, the first section including one or more locations from the plurality of locations that satisfy a proximity constraint and wherein the first section comprises a portion of the multimedia document occurring between a first start time and a first end time along the timeline;

visually emphasizing the first section; and

visually distinguishing every occurrence of the search term within the textual representation from a remainder of the textual representation.

15. The method of claim 14 further comprising:

determining a second section within the textual representation, the second section including one or more locations from the plurality of locations that satisfy the proximity constraint and wherein the second section comprises a portion of the multimedia document occurring between a second start time and a second end time along the timeline;

determining that a time interval between the first section and the second section is less than a predetermined value; and

combining the first and the second sections to generate a third section, wherein the third section comprises a portion of the multimedia document occurring between the first start time and the second end time along the timeline.