Paper-based interface for multimedia information

• US 7,263,659 B2
• Filed: 11/19/2001
• Issued: 08/28/2007
• Est. Priority Date: 09/09/1998
• Status: Expired due to Fees

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Hull, Jonathan J.
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US10/001,895
• Publication Number:
  US 20050229107A1
• Time in Patent Office: 2,108 Days
• Field of Search: 345/700, 345/716, 345/717, 345/719, 345/727, 345/764, 345/835, 345/838, 345/839, 345/855, 358/1.1, 358/1.15, 358/1.18, 358/530, 358/540, 705/55, 715/479, 715/500, 715/500.1, 715/501.1, 715/513, 715/700, 715/716, 715/717, 715/719, 715/727, 715/764, 715/835, 715/838, 715/839, 715/855, 715/517, 715/526
• US Class Current: 715/716
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

The present invention provides techniques for generating a printable representation of multimedia information that can be printed on a paper medium to provide a paper-based interface for the multimedia information. The paper-based interface provided by the present invention provides a more readable and comprehensible representation of the multimedia information.

Claims

1. A computer-implemented method of generating a printable representation for an electronically stored multimedia document storing multimedia information, the multimedia information comprising information of at least a first type and information of a second type, the method comprising:

receiving input specifying a concept of interest;

analyzing the multimedia information in the multimedia document based on the input to identify information relevant to the concept of interest, the information relevant to the concept of interest comprising information of the first type and information of the second type;

accessing layout information specifying a first layout feature associated with how the information of the first type is to arranged when printed on a paper medium and specifying a second layout feature associated with how the information of the second type is to be arranged when printed on the paper medium; and

generating the printable representation for the multimedia information stored in the multimedia document based upon the layout information, the printable representation for the multimedia document comprising a printable representation for information of the first type and a printable representation for information of the second type, the printable representation of the multimedia document capable of being printed on a paper medium.

2. The method of claim 1 wherein generating the printable representation for the multimedia information stored in the multimedia document comprises:

dividing the multimedia information stored in the multimedia document into a plurality of segments, each segment comprising multimedia information of a pre-configured time length;

from the plurality of segments, selecting a set of segments to be included in the printable representation for the multimedia document, the set of segments including one or more segments from the plurality of segments; and

generating the printable representation for the multimedia document based upon the layout information and multimedia information corresponding to the selected set of segments.

3. The method of claim 2 wherein selecting the set of segments to be included in the printable representation comprises:

identifying a selection criterion; and

selecting a segment from the plurality of segments to be included in the set of segments if the multimedia information corresponding to the segment satisfies the selection criterion.

4. The method of claim 3 wherein:

selecting a segment from the plurality of segments to be included in the set of segments comprises selecting the segment if the multimedia information corresponding to the segment comprises information relevant to the topic of interest.

5. The method of claim 2 wherein generating the printable representation for the multimedia document based upon the layout information and multimedia information corresponding to the selected set of segments comprises:

determining a set of pages on which the multimedia information corresponding to the selected set of segments is to be printed, the set of pages comprising one or more pages; and

creating printable representations for the set of pages, wherein the printable representation for each page includes the printable representation for information of the first type and the printable representation for information of the second type corresponding to segments to be printed on the page.

6. The method of claim 5 wherein the printable representation for each page in the set of pages includes information identifying one or more segments from the set of segments whose information is included in the printable representation for the page.

7. The method of claim 1 wherein:

the information of the first type corresponds to video information; and

the printable representation for information of the first type comprises video keyframes selected from the video information.

8. The method of claim 7 wherein:

the information of the second type corresponds to closed-caption text information; and

the printable representation for information of the second type comprises text information selected from the closed-caption text information.

9. The method of claim 7 wherein:

the information of the second type corresponds to audio information; and

the printable representation for information of the second type comprises text information selected from a text transcript of the audio information.

10. The method of claim 1 wherein the multimedia information stored by the multimedia document includes information of a third type and information of a fourth type and the printable representation for the multimedia document comprises a printable representation for information of the third type and a printable representation for information of the fourth type, wherein the information of the first type corresponds to video information, the information of the second type corresponds to closed-caption text information, information of the third type corresponds to slides information, and information of the fourth type corresponds to whiteboard information.

11. The method of claim 1 further comprising:

printing the printable representation for the multimedia document on a paper medium to generate a multimedia paper document comprising one or more printed pages.

12. The method of claim 11 wherein:

a time span is associated with at least one page of the multimedia paper document, the time span characterized by a first time and a second time; and

the at least one page of the multimedia paper document is imprinted with a portion of the printable representation for the multimedia document corresponding to multimedia information occurring between the first time and the second time associated with the at least one page.

13. The method of claim 12 wherein:

the first time and the second time associated with the at least one page of the one or more pages is printed on the at least one page.

14. The method of claim 12 wherein:

a portion of the printable representation for information of the first type corresponding to the time span associated with the at least one page is printed on a first section of the at least one page, and a portion of the printable representation for information of the second type corresponding to the time span associated with the at least one page is printed on a second section of the at least one page.

15. The method of claim 14 wherein the at least one page of the one or more printed pages is imprinted with identifiers for temporally correlating the printable representation for information of the first type printed on the page with the printable representation for information of the second type printed on the at least one page.

16. The method of claim 14 wherein:

the information of the first type corresponds to video information; and

the portion of the printable representation for information of the first type printed on the at least one page of the multimedia paper document comprises video keyframes selected from the video information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

17. The method of claim 16 wherein:

the information of the second type corresponds to closed-caption text information; and

the portion of the printable representation for information of the second type printed on the at least one page comprises text information selected from the closed-caption text information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

18. The method of claim 16 wherein:

the information of the second type corresponds to audio information; and

the portion of the printable representation for information of the second type printed on the at least one page comprises text information selected from a transcript of the audio information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

19. The method of claim 11 wherein information imprinted on at least one page of the multimedia paper document includes information identifying the multimedia document, information identifying a time length of the multimedia document, and information identifying a time when the multimedia information in the multimedia document was recorded.

20. The method of claim 11 wherein:

the multimedia information stored by the multimedia document includes information of a third type and information of a fourth type; and

at least one page of the multimedia paper document is imprinted with a portion of the printable representation for the multimedia document corresponding to a time span associated with the at least one page, the time span characterized by a first time and a second time, the portion of the printable representation for the multimedia document imprinted on the at least one page comprising a portion of the printable representation for information of the first type corresponding to the time span associated with the at least one page, a portion of the printable representation for information of the second type corresponding to the time span associated with the at least one page, a portion of the printable representation for information of the third type corresponding to the time span associated with the at least one page, and a portion of the printable representation for information of the fourth type corresponding to the time span associated with the at least one page; and

the information of the first type corresponds to video information, the information of the second type corresponds to closed-caption text information, the information of the third type corresponds to slides information, and the information of the fourth type corresponds to whiteboard information.

21. A computer-implemented method of generating a paper document for a electronically stored multimedia document, the multimedia document storing multimedia information that includes video information, the method comprising:

receiving a time span associated with a page of the paper document, the time span characterized by a first time for the page and a second time for the page;

printing one or more video keyframes in a first area of the page of the paper document, wherein the one or more video keyframes are extracted from the video information occurring during the time span associated with the page; and

printing text information in a second area of the page of the paper document, wherein the text information printed in the second area is extracted from the multimedia information occurring during the time span associated with the page.

22. The method of claim 21 further comprising:

printing the first time and the second time on the page.

23. The method of claim 21 wherein the multimedia information stored by the multimedia document includes information identifying a plurality of slides, the method further comprising:

printing one or more slides from the plurality of slides on the page of the paper document, wherein the one or more slides printed on the page include slides having a time stamp occurring during the time span associated with the page.

24. The method of claim 21 wherein the multimedia information stored by the multimedia document includes information identifying a plurality of whiteboard images, the method further comprising:

printing one or more whiteboard images from the plurality of whiteboard images on the page of the paper document, wherein the one or more whiteboard images printed on the page include images having a time stamp occurring during the time span associated with the page.

25. The method of claim 21 further comprising:

printing information identifying the multimedia document on the page of the paper document;

printing information identifying a time length of the multimedia document on the page; and

printing information identifying a time when the multimedia information stored by the multimedia document was recorded on the page.

26. The method of claim 21 further comprising:

printing a first identifier on the page of the paper document in a location proximal to a first video frame printed on the page; and

printing the same first identifier on the page in a location proximal to text information printed on the page that occurs substantially at the same time in the multimedia document as the first video frame.

27. The method of claim 21 wherein:

the multimedia information stored by the multimedia document includes closed-caption text information; and

the text information printed in the second area of the page of the paper document is extracted from the closed-caption text information occurring during the time span associated with the page.

28. The method of claim 27 further comprising:

determining a time period during the time span associated with the page of the paper document when the closed-caption information is empty;

determining a video feature from the video information occurring during the time period; and

printing text identifying the video feature on the page.

29. The method of claim 27 wherein the multimedia information stored by the multimedia document includes audio information and the method further comprises:

determining a time period during the time span associated with the page when the closed-caption information is empty;

determining an audio feature from the audio information occurring during the time period; and

printing text identifying the audio feature on the page.

30. The method of claim 21 wherein:

the multimedia information stored by the multimedia document includes audio information; and

the text information printed in the second area of the page of the paper document is extracted from a transcript of the audio information occurring during the time span associated with the page.

31. A system for generating a printable representation for an electronically stored multimedia document storing multimedia information, the multimedia information comprising information of at least a first type and information of a second type, the system comprising:

a data processor, configured to;

receive input specifying a concept of interest;

analyze the multimedia information in the multimedia document based on the input to identify information relevant to the concept of interest, the information relevant to the concept interest comprising information of the first type and the information of the second type;

access layout information specifying a first layout feature associated with how the information of the first type is to be arranged when printed on a paper medium and specifying a second layout feature associated with how the information ofthe second type is to be arranged when printed on the paper medium; and

generate the printable representation for the multimedia information stored in the multimedia document based upon the layout information, the printable representation for the multimedia document comprising a printable representation for information of the first type and a printable representation for information of the second type, the printable representation of the multimedia document capable of being printed on a paper medium; and

communicate the printable representation of the multimedia document to an output device; and

an output device capable of printing the printable representation of the multimedia document on a paper medium.

32. The system of claim 31 wherein in order to generate the printable representation for the multimedia information stored in the multimedia document, the data processor is configured to:

divide the multimedia information stored in the multimedia document into a plurality of segments, each segment comprising multimedia information of a pre-configured time length;

from the plurality of segments, select a set of segments to be included in the printable representation for the multimedia document, the set of segments comprising one or more segments from the plurality of segments; and

generate the printable representation for the multimedia document based upon the layout information and multimedia information corresponding to the selected set of segments.

33. The system of claim 32 wherein in order to select the set of segments to be included in the printable representation, the data processor is configured to:

receive a selection criterion; and

select a segment from the plurality of segments to be included in the set of segments if the multimedia information corresponding to the segment satisfies the selection criterion.

34. The system of claim 33 wherein:

in order to select a segment from the plurality of segments to be included in the set of segments, the data processor is configured to select the segment if the multimedia information corresponding to the segment comprises information relevant to the topic of interest.

35. The system of claim 32 wherein in order to generate the printable representation for the multimedia document based upon the layout information and multimedia information corresponding to the selected set of segments, the data processor is configured to:

determine a set of pages on which the multimedia information corresponding to the selected set of segments is to be printed, the set of pages comprising one or more pages; and

create printable representations for the set of pages, wherein the printable representation for each page includes the printable representation for information of the first type and the printable representation for information of the second type corresponding to segments to be printed on the page.

36. The system of claim 35 wherein the printable representation for each page in the set of pages includes information identifying one or more segments from the set of segments whose information is included in the printable representation for the page.

37. The system of claim 31 wherein:

the information of the first type corresponds to video information; and

the printable representation for information of the first type comprises video keyframes selected from the video information.

38. The system of claim 37 wherein:

the information of the second type corresponds to closed-caption text information; and

the printable representation for information of the second type comprises text information selected from the closed-caption text information.

39. The system of claim 37 wherein:

the information of the second type corresponds to audio information; and

the printable representation for information of the second type comprises text information selected from a text transcript of the audio information.

40. The system of claim 31 wherein the multimedia information stored by the multimedia document includes information of a third type and information of a fourth type and the printable representation for the multimedia document comprises a printable representation for information of the third type and a printable representation for information of the fourth type, wherein the information of the first type corresponds to video information, the information of the second type corresponds to closed-caption text information, information of the third type corresponds to slides information, and information of the fourth type corresponds to whiteboard information.

41. The system of claim 31 wherein the output device is configured to print the printable representation for the multimedia document on a paper medium to generate a multimedia paper document comprising one or more printed pages.

42. The system of claim 41 wherein:

a time span is associated with at least one page of the multimedia paper document, the time span characterized by a first time and a second time; and

the at least one page of the multimedia paper document is imprinted with a portion of the printable representation for the multimedia document corresponding to multimedia information occurring between the first time and the second time associated with the at least one page.

43. The system of claim 42 wherein:

the first time and the second time associated with the at least one page are printed on the at least one page.

44. The system of claim 42 wherein:

a portion of the printable representation for information of the first type corresponding to the time span associated with the at least one page is printed on a first section of the at least one page, and a portion of the printable representation for information of the second type corresponding to the time span associated with the at least one page is printed on a second section of the at least one page.

45. The system of claim 44 wherein the at least one page is imprinted with identifiers for temporally correlating the printable representation for information of the first type printed on the at least one page with the printable representation for information of the second type printed on the at least one page.

46. The system of claim 44 wherein:

the information of the first type corresponds to video information; and

the portion of the printable representation for information of the first type printed on the at least one page of the multimedia paper document comprises video keyframes selected from the video information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

47. The system of claim 46 wherein:

the information of the second type corresponds to closed-caption text information; and

the portion of the printable representation for information of the second type printed on the at least one page comprises text information selected from the closed-caption text information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

48. The system of claim 46 wherein:

the information of the second type corresponds to audio information; and

the portion of the printable representation for information of the second type printed on the at least one page comprises text information selected from a transcript of the audio information occurring in the multimedia information stored by the multimedia document during the time span associated with the at least one page.

49. The system of claim 41 wherein information imprinted on at least one page of the multimedia paper document includes information identifying the multimedia document, information identifying a time length of the multimedia document, and information identifying a time when the multimedia information in the multimedia document was recorded.

50. The system of claim 41 wherein:

the multimedia information stored by the multimedia document includes information of a third type and information of a fourth type; and

at least one page of the multimedia paper document is imprinted with a portion of the printable representation for the multimedia document corresponding to a time span associated with the at least one page, the time span characterized by a first time and a second time, the portion of the printable representation for the multimedia document imprinted on the at least one page comprising a portion of the printable representation for information of the first type corresponding to the time span associated with the at least one page, a portion of the printable representation for information of the second type corresponding to the time span associated with the at least one page, a portion of the printable representation for information of the third type corresponding to the time span associated with the at least one page, and a portion of the printable representation for information of the fourth type corresponding to the time span associated with the at least one page; and

the information of the first type corresponds to video information, the information of the second type corresponds to closed-caption text information, the information of the third type corresponds to slides information, and the information of the fourth type corresponds to whiteboard information.

51. A system for generating a printed page, the system comprising:

an input module, configured to receive a printable representation of multimedia information stored by a multimedia document, the printable representation including one or more video keyframes extracted from the multimedia information occurring during a time span associated with a page of a paper document, the time span characterized by a first time for the page and a second time for the page, the printable representation including text information extracted from the multimedia information occurring during the time span; and

an output module, configured to;

print the one or more video keyframes in a first area of the page of the paper document; and

print the text information in a second area of the page of the paper document.

52. The system of claim 51 wherein the output module is configured to print the first time and the second time of the time span on the page.

53. The system of claim 51 wherein:

the printable representation received by the input module includes one or more slides extracted from multimedia information occurring during the time span; and

the output module is configured to print the one or more slides on the page.

54. The system of claim 51 wherein:

the printable representation received by the input module includes one or more whiteboard images extracted from multimedia information occurring during the time span; and

the output module is configured to print the one or more whiteboard images on the page.

55. The system of claim 51 wherein the output module is configured to:

print information identifying the multimedia document on the page;

print information identifying a time length of the multimedia document on the page; and

print information identifying a time when the multimedia information stored by the multimedia document was recorded on the page.

56. The system of claim 51 wherein the output module is configured to:

print a first identifier on the page in a location proximal to a first video frame printed on the page; and

print the same first identifier on the page in a location proximal to text information printed on the page that occurs substantially at the same time in the multimedia information as the first video frame.

57. The system of claim 51 wherein the text information included in the printable representation received by the input module is extracted from closed-caption text information included in the multimedia information and occurring during the time span.

58. The system of claim 57 wherein:

the printable representation received by the input module includes text related to a video feature extracted from video information included in the multimedia information during the time span when the closed-caption text information is empty; and

the output module is configured to print the text related to the video feature on the page.

59. The system of claim 57 wherein:

the printable representation received by the input module includes text related to an audio feature extracted from audio information included in the multimedia information during the time span when the closed-caption text information is empty; and

the output module is configured to print the text related to the audio feature on the page.

60. The system of claim 51 wherein the text information printed in the second area of the page by the output module is extracted from a transcript of audio information included in the multimedia information and occurring during the time span.

61. A computer program product stored on a computer-readable storage medium for generating a printable representation for an electronically stored multimedia document storing multimedia information, the multimedia information comprising information of at least a first type and information of a second type, the computer program product comprising:

code for receiving inputs specifying a concept of interest;

code for analyzing the multimedia information in the multimedia document based on the input to identify information relevant to the concept of interest, the information relevant to the concept of interest comprising information of the first type and information of the second type;

code for accessing layout information specifying a first layout feature associated with how the information of the first type is to be arranged when printed on a paper medium and specifying a second layout feature associated with how the information of the second type is to be arranged when printed on the paper medium; and

code for generating the printable representation for the multimedia information stored in the multimedia document based upon the layout information, the printable representation for the multimedia document comprising a printable representation for information of the first type and a printable representation for information of the second type, the printable representation of the multimedia document capable of being printed on a paper medium.

62. The computer program product of claim 61 wherein the code for generating the printable representation for the multimedia information stored in the multimedia document comprises:

code for dividing the multimedia information stored in the multimedia document into a plurality of segments, each segment comprising multimedia information of a pre-configured time length;

code for selecting a set of segments from the plurality of segments to be included in the printable representation for the multimedia document, the set of segments including one or more segments from the plurality of segments; and

code for generating the printable representation for the multimedia document based upon the layout information and multimedia information corresponding to the selected set of segments.

63. The computer program product of claim 61 wherein:

the information of the first type corresponds to video information; and

the printable representation for information of the first type comprises video keyframes selected from the video information.

64. The computer program product of claim 63 wherein:

the information of the second type corresponds to closed-caption text information; and

the printable representation for information of the second type comprises text information selected from the closed-caption text information.

65. The computer program product of claim 63 wherein:

the information of the second type corresponds to audio information; and

the printable representation for information of the second type comprises text information selected from a text transcript of the audio information.

66. The computer program product of claim 61 further comprising:

code for printing the printable representation for the multimedia document on a paper medium to generate a multimedia paper document comprising one or more printed pages.

67. The computer program product of claim 66 wherein:

a time span is associated with at least one page of the multimedia paper document, the time span characterized by a first time and a second time; and

the at least one page of the multimedia paper document is imprinted with a portion of the printable representation for the multimedia document corresponding to multimedia information occurring between the first time and the second time associated with the at least one page.

68. A computer program product stored on a computer-readable storage medium for generating a paper document for a electronically stored multimedia document, the multimedia document storing multimedia information that includes video information, the computer program product comprising:

code for receiving a time span associated with a page of the paper document, the time span characterized by a first time for the page and a second time for the page;

code for printing one or more video keyframes in a first area of the page of the paper document, wherein the one or more video keyframes are extracted from the video information occurring during the time span associated with the page; and

code for printing text information in a second area of the page of the paper document, wherein the text information printed in the second area is extracted from the multimedia information occurring during the time span associated with the page.

69. The computer program product of claim 68 wherein the multimedia information stored by the multimedia document includes information identifying a plurality of slides, the computer program product further comprising:

code for printing one or more slides from the plurality of slides on the page of the paper document, wherein the one or more slides printed on the page include slides having a time stamp occurring during the time span associated with the page.

70. The computer program product of claim 68 wherein the multimedia information stored by the multimedia document includes information identifying a plurality of whiteboard images, the computer program product further comprising:

code for printing one or more whiteboard images from the plurality of whiteboard images on the page of the paper document, wherein the one or more whiteboard images printed on the page include images having a time stamp occurring during the time span associated with the page.

71. The computer program product of claim 68 wherein:

the multimedia information stored by the multimedia document includes closed-caption text information; and

the text information printed in the second area of the page of the paper document is extracted from the closed-caption text information occurring during the time span associated with the page.

72. The computer program product of claim 68 wherein:

the multimedia information stored by the multimedia document includes audio information; and

the text information printed in the second area of the page of the paper document is extracted from a transcript of the audio information occurring during the time span associated with the page.