Multimedia visualization and integration environment

• US 20040103372A1
• Filed: 11/04/2003
• Published: 05/27/2004
• Est. Priority Date: 12/22/1997
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey
• Granted Patent: US 8,739,040 B2
• Time in Patent Office: Days
• US Class Current: 715/513
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/34 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Embodiments of the present invention provide techniques for retrieving and displaying multimedia information. According to an embodiment of the present invention, a graphical user interface (GUI) is provided that displays multimedia information that may be stored in a multimedia document. According to the teachings of the present invention, the GUI enables a user to navigate through multimedia information stored in a multimedia document. The GUI provides both a focused and a contextual view of the contents of the multimedia document.

Claims

1. A computer-implemented method of displaying recorded information, the recorded information comprising information of at least a first type and a second type, the method comprising:

displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

receiving information identifying a criterion;

determining one or more locations in the information of the first type that satisfy the criterion; and

highlighting, based upon the one or more locations in the information of the first type, one or more sections of at least one of the first representation and the second representation.

2. The method of claim 1 wherein highlighting the one or more sections comprises:

determining, based upon the one or more locations in the information of the first type that satisfy the criterion, one or more portions of the information of the first type, each portion having a start time and an end time; and

determining the one or more sections to be highlighted based upon the start and end times associated with the one or more portions.

3. The method of claim 2 wherein determining the one or more portions of the information of the first type comprises:

determining a time associated with each location in the one or more locations in the information of the first type; and

determining the one or more portions of the information of the first type based upon the times associated with the one or more locations.

4. The method of claim 3 wherein:

the one or more locations comprise a first location and a second location; and

determining the one or more portions of the information of the first type based upon the times associated with the one or more locations comprises;

determining a time interval between the first location and the second location based upon the times associated with the first location and the second location; and

determining if the time interval exceeds a first threshold value.

5. The method of claim 1 wherein highlighting the one or more sections comprises:

determining one or more segments of the information of the first type based upon the one or more locations in the information of the first type that satisfy the criterion, each segment having a start time and an end time;

determining one or more portions of the information of the first type based upon the start and end times associated with the one or more segments, each portion having a start time and end time; and

determining the one or more sections to be highlighted based upon the start and end times associated with the one or more portions.

6. The method of claim 5 wherein:

the one or more segments comprise a first segment and a second segment; and

determining the one or more portions of the information of the first type comprises;

determining a time interval between the first segment and the second segment; and

combining the first segment and the second segment to form a first portion in the one or more portions if the time interval is less than a threshold value, the first portion having a start time equal to the start time of the first segment and an end time equal to the end time of the second segment.

7. The method of claim 1 wherein:

the criterion is a word; and

determining the one or more locations in the information of the first type that satisfy the criterion comprises determining one or more occurrences of the word in the information of the first type.

8. The method of claim 1 wherein:

the criterion is a topic of interest; and

determining the one or more locations in the information of the first type that satisfy the criterion comprises determining one or more locations in the information of the first type that are relevant to the topic of interest.

9. The method of claim 1 wherein:

the information of the first type is at least one of audio information and closed-caption text information;

the information of the second type is video information;

the first representation comprises at least one of text information obtained from transcribing the audio information and text information obtained from the closed-caption text information; and

the second representation comprises one or more video keyframes extracted from the video information.

10. The method of claim 1 further comprising:

displaying an identifier for each highlighted section in the one or more sections.

11. A computer-implemented method of displaying audio information and video information, the method comprising:

displaying a first thumbnail comprising text information obtained from transcribing the audio information;

displaying a second thumbnail comprising one or more video keyframes extracted from the video information;

receiving information identifying a criterion;

determining one or more locations in the audio information that satisfy the criterion;

determining one or more portions of the audio information based upon the one or more locations, each portion of the audio information having a start time and an end time; and

highlighting one or more sections of at least one of the first thumbnail and the second thumbnail based upon the start and end times associated with the one or more portions of the audio information.

12. A computer-implemented method of displaying recorded information, the recorded information comprising information of at least a first type and a second type, the method comprising:

displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

receiving information identifying a first time and a second time; and

highlighting a section of the first representation corresponding to a portion of information of the first type occurring between the first time and the second time.

13. A computer-implemented method of displaying recorded information, the recorded information comprising information of at least a first type and a second type, the method comprising:

displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

receiving information identifying a first range and a second range, the first range having a first start time and a first end time and the second range having a second start time and a second end time;

determining a time interval between the first range and the second range;

combining the first range and the second range to form a third range if the time interval is less than a threshold value, the third range having a start time equal to the start time of the first range and an end time equal to the end time of the second range; and

highlighting a section of the first representation corresponding to a portion of information of the first type occurring between the start time and the end time of the third range.

14. The method of claim 13 wherein determining the time interval between the first range and the second range comprises determining the time interval based upon the first end time and the second start time.

15. The method of claim 13 wherein combining the first range and the second range to form the third range comprises:

determining a time span of the first range, wherein the time span is a time interval between the first start time and the first end time; and

combining the first range and the second range to form the third range only if the time span of the first range is less than a second threshold value.

16. A computer-implemented method of performing operations on recorded information, the method comprising:

displaying a representation of the recorded information, wherein a section of the displayed representation of the recorded information is highlighted;

receiving a signal indicative of an operation to be performed;

determining a portion of the recorded information corresponding to the highlighted section; and

performing the operation on the portion of the recorded information corresponding to the highlighted section.

17. The method of claim 16 wherein the recorded information comprises audio information.

18. The method of claim 16 wherein the recorded information comprises video information.

19. The method of claim 16 wherein performing the operation on the portion of the recorded information corresponding to the highlighted section comprises outputting the portion of the recorded information via an output device.

20. The method of claim 16 wherein performing the operation on the portion of the recorded information corresponding to the highlighted section comprises storing the portion of the recorded information in a memory location.

21. The method of claim 16 wherein performing the operation on the portion of the recorded information corresponding to the highlighted section comprises printing a representation of the portion of the recorded information on a paper medium.

22. The method of claim 16 wherein performing the operation on the portion of the recorded information corresponding to the highlighted section comprises communicating the portion of the recorded information to a recipient.

23. The method of claim 16 wherein performing the operation on the portion of the recorded information corresponding to the highlighted section comprises providing the portion of the recorded information to an application.

24. A computer program product stored on a computer-readable medium for displaying recorded information comprising information of at least a first type and a second type, the computer program product comprising:

code for displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

code for receiving information identifying a criterion;

code for determining one or more locations in the information of the first type that satisfy the criterion; and

code for highlighting, based upon the one or more locations in the information of the first type, one or more sections of at least one of the first representation and the second representation.

25. The computer program product of claim 24 wherein the code for highlighting the one or more sections comprises:

code for determining, based upon the one or more locations in the information of the first type that satisfy the criterion, one or more portions of the information of the first type, each portion having a start time and an end time; and

code for determining the one or more sections to be highlighted based upon the start and end times associated with the one or more portions.

26. The computer program product of claim 25 wherein the code for determining the one or more portions of the information of the first type comprises:

code for determining a time associated with each location in the one or more locations in the information of the first type; and

code for determining the one or more portions of the information of the first type based upon the times associated with the one or more locations.

27. The computer program product of claim 26 wherein:

the one or more locations comprise a first location and a second location; and

the code for determining the one or more portions of the information of the first type based upon the times associated with the one or more locations comprises;

code for determining a time interval between the first location and the second location based upon the times associated with the first location and the second location; and

code for determining if the time interval exceeds a first threshold value.

28. The computer program product of claim 24 wherein the code for highlighting the one or more sections comprises:

code for determining one or more segments of the information of the first type based upon the one or more locations in the information of the first type that satisfy the criterion, each segment having a start time and an end time;

code for determining one or more portions of the information of the first type based upon the start and end times associated with the one or more segments, each portion having a start time and end time; and

code for determining the one or more sections to be highlighted based upon the start and end times associated with the one or more portions.

29. The computer program product of claim 28 wherein:

the one or more segments comprise a first segment and a second segment; and

code for determining the one or more portions of the information of the first type comprises;

code for determining a time interval between the first segment and the second segment; and

code for combining the first segment and the second segment to form a first portion in the one or more portions if the time interval is less than a threshold value, the first portion having a start time equal to the start time of the first segment and an end time equal to the end time of the second segment.

30. The computer program product of claim 24 wherein:

the criterion is a word; and

the code for determining the one or more locations in the information of the first type that satisfy the criterion comprises code for determining one or more occurrences of the word in the information of the first type.

31. The computer program product of claim 24 wherein:

the criterion is a topic of interest; and

the code for determining the one or more locations in the information of the first type that satisfy the criterion comprises code for determining one or more locations in the information of the first type that are relevant to the topic of interest.

32. The computer program product of claim 24 wherein:

the information of the first type is at least one of audio information and closed-caption text information;

the information of the second type is video information;

the first representation comprises at least one of text information obtained from transcribing the audio information and text information obtained from the closed-caption text information; and

the second representation comprises one or more video keyframes extracted from the video information.

33. The computer program product of claim 24 further comprising:

code for displaying an identifier for each highlighted section in the one or more sections.

34. A computer program product stored on a computer-readable medium for displaying audio information and video information, the computer program product comprising:

code for displaying a first thumbnail comprising text information obtained from transcribing the audio information;

code for displaying a second thumbnail comprising one or more video keyframes extracted from the video information;

code for receiving information identifying a criterion;

code for determining one or more locations in the audio information that satisfy the criterion;

code for determining one or more portions of the audio information based upon the one or more locations, each portion of the audio information having a start time and an end time; and

code for highlighting one or more sections of at least one of the first thumbnail and the second thumbnail based upon the start and end times associated with the one or more portions of the audio information.

35. A computer program product stored on a computer-readable medium for displaying recorded information comprising information of at least a first type and a second type, the computer program product comprising:

code for displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

code for receiving information identifying a first time and a second time; and

code for highlighting a section of the first representation corresponding to a portion of information of the first type occurring between the first time and the second time.

36. A computer program product stored on a computer-readable medium for displaying recorded information comprising information of at least a first type and a second type, the computer program product comprising:

code for displaying a representation of the recorded information, the displayed representation of the recorded information comprising a first representation and a second representation, wherein the first representation is a representation of information of the first type and the second representation is a representation of information of the second type;

code for receiving information identifying a first range and a second range, the first range having a first start time and a first end time and the second range having a second start time and a second end time;

code for determining a time interval between the first range and the second range;

code for combining the first range and the second range to form a third range if the time interval is less than a threshold value, the third range having a start time equal to the start time of the first range and an end time equal to the end time of the second range; and

code for highlighting a section of the first representation corresponding to a portion of information of the first type occurring between the start time and the end time of the third range.

37. The computer program product of claim 36 wherein the code for determining the time interval between the first range and the second range comprises code for determining the time interval based upon the first end time and the second start time.

38. The computer program product of claim 36 wherein the code for combining the first range and the second range to form the third range comprises:

code for determining a time span of the first range, wherein the time span is a time interval between the first start time and the first end time; and

code for combining the first range and the second range to form the third range only if the time span of the first range is less than a second threshold value.

39. A computer program product stored on a computer-readable medium for performing operations on recorded information, the computer program product comprising:

code for displaying a representation of the recorded information, wherein a section of the displayed representation of the recorded information is highlighted;

code for receiving a signal indicative of an operation to be performed;

code for determining a portion of the recorded information corresponding to the highlighted section; and

code for performing the operation on the portion of the recorded information corresponding to the highlighted section.

40. The computer program product of claim 39 wherein the recorded information comprises audio information.

41. The computer program product of claim 39 wherein the recorded information comprises video information.

42. The computer program product of claim 39 wherein the code for performing the operation on the portion of the recorded information corresponding to the highlighted section comprises code for outputting the portion of the recorded information via an output device.

43. The computer program product of claim 39 wherein the code for performing the operation on the portion of the recorded information corresponding to the highlighted section comprises code for storing the portion of the recorded information in a memory location.

44. The computer program product of claim 39 wherein the code for performing the operation on the portion of the recorded information corresponding to the highlighted section comprises code for printing a representation of the portion of the recorded information on a paper medium.

45. The computer program product of claim 39 wherein the code for performing the operation on the portion of the recorded information corresponding to the highlighted section comprises code for communicating the portion of the recorded information to a recipient.

46. The computer program product of claim 39 wherein the code for performing the operation on the portion of the recorded information corresponding to the highlighted section comprises code for providing the portion of the recorded information to an application.