Methods for authoring and interacting with multimedia representations of documents

• US 8,812,969 B2
• Filed: 03/21/2007
• Issued: 08/19/2014
• Est. Priority Date: 03/21/2007
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hart, Peter E., Erol, Berna, Berkner, Kathrin, Hull, Jonathan J.
• Primary Examiner(s): Ng, Amy
• Assistant Examiner(s): Gutierrez, Andres E
• Application Number:
  US11/689,394
• Publication Number:
  US 20080235585A1
• Time in Patent Office: 2,708 Days
• Field of Search: 715/730, 715/731, 715/732, 715/765
• US Class Current: 715/765
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor

Abstract

A method, apparatus and article of manufacture for assisting a user with authoring a multimedia representation of documents. In one embodiment, the method comprises displaying one or more visual, audible, or audiovisual document elements from an input document in graphical user interface; and selecting one or more elements and for inclusion in a multimedia representation.

Claims

1. A method comprising:

extracting a plurality of visual, audible, and audiovisual document elements from content contained within an input document, wherein the input document comprises a single file;

displaying at least one visual, audible, or audiovisual document elements from the extracted visual, audible, and audiovisual document elements of the input document in a graphical user interface;

automatically selecting two or more of the extracted visual, audible, or audiovisual document elements for inclusion in a multimedia thumbnail representation of the input document based on a time attribute and an information attribute associated with each of the extracted visual, audible, and audiovisual document elements to maximize a total of information content provided by the selected visual, audible, or audiovisual document elements in a given amount of time, wherein the automatically selecting comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting one or more visual, audible, or audiovisual document elements based on each time attribute and each information attribute, wherein each time attribute is calculated, using one or more characteristics of a device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, and wherein each information attribute is calculated, using one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein automatically selecting two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises;

selecting audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

filling the one or more time intervals of the audible channel with the selected audible document elements; and

automatically generating the multimedia representation based on the automatically selected one or more visual, audible, or audiovisual document elements.

2. The method of claim 1 wherein displaying the graphical user interface comprises:

displaying the selected document elements in a context of a page of the input document; and

further comprising receiving user input from one or more user interface elements in the graphical user interface, the user input specifying at least one change in selection of the document elements for inclusion in the multimedia representation.

3. The method of claim 1 wherein displaying the document elements in the graphical user interface comprises:

displaying the document elements in a dialog box; and

displaying one or more user interface elements by which a user can specify a change in selection of the document elements for inclusion in the multimedia representation.

4. The method of claim 1 further comprising:

receiving user input through the graphical user interface from a user to specify one or more operations that include at least one of the following;

adding a new document element, removing an existing document element, defining a break point, grouping two or more of the document elements, and specifying a navigation path.

5. The method of claim 1 further comprising:

receiving user input through the user graphical interface to specify a presentation action associated with each of the document elements, wherein the presentation action includes at least one of the following zooming, panning over, synthesizing to speech, and synchronizing with other document elements.

6. The method of claim 1 wherein displaying a graphical user interface further comprises:

receiving user input through the graphical user interface to specify a sequence number for each document element to indicate a presentation sequence.

7. The method of claim 1 further comprising:

highlighting the document elements with different visual indicia to indicate a selection status of the document elements.

8. The method of claim 1 further comprising:

receiving user input through the graphical user interface to specify one or more parameters that constrain a visual setting or an audio setting of the multimedia representation.

9. The method of claim 8 wherein the parameters include at least one of the following:

presentation duration, display resolution, synthesized voice, gender of synthesized voice, background music, target device profiling and text-to-speech engine.

10. The method of claim 1 further comprising:

storing a user profile that includes a viewing preference of a user; and

utilizing the user profile in creating the multimedia presentation for the user.

37. The method of claim 1, wherein the automatically selecting two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in a given amount of time, further comprises:

selecting visual and audiovisual document elements that maximize an amount of information contained in the combination of selected visual and audiovisual document elements when presented in a video channel of the multimedia representation in the given amount of time;

filling the video channel of the multimedia representation with the selected visual and audiovisual document elements in an order in which the selected visual and audiovisual document elements occur in the input document, wherein audio data associated with the selected audiovisual elements are utilized to fill an audio channel of the multimedia representation and are synchronized with a presentation of the corresponding audiovisual document elements in the multimedia representation;

automatically generating the multimedia representation from the filled video channel and audible channel for the multimedia representation.

11. A method comprising:

extracting a plurality of visual, audible, and audiovisual document elements from content contained within an input document, wherein the input document comprises a single file;

automatically selecting two or more visual, audible, or audiovisual document elements from the extracted visual, audible, and audiovisual document elements of the input document based on a time attribute and an information attribute associated with each of the extracted visual, audible, and audiovisual document elements to maximize a total of information content provided by the selected visual, audible, and audiovisual document elements in a given amount of time when included in a multimedia representation of the input document, wherein the automatically selecting comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting at least one visual, audible, or audiovisual document element based on each time attribute and each information attribute, wherein each time attribute is calculated, using one or more characteristics of a device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, and wherein each information attribute is calculated, using one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein automatically selecting two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises;

selecting audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

filling the one or more time intervals of the audible channel with the selected audible document elements; and

automatically generating the multimedia representation based on the automatically selected one or more visual, audible, or audiovisual document elements; and

displaying an interactive user interface to playback a combination of the multimedia representation and the input document under user control.

12. The method of claim 11 further comprising:

linking at least one of the document elements to a region on a page of the input document; and

under the user control, switching between presentation of the multimedia representation and the input document by utilizing the linking.

13. The method of claim 11 further comprising:

providing a manual navigation control for a user to view the input document.

14. The method of claim 11 further comprising:

providing an automatic navigation control for a user to view the multimedia representation.

15. The method of claim 11 wherein providing an interactive user interface further comprises:

embedding a play indicator in at least a page of the input document; and

activating the play indicator to play back the multimedia representation from a corresponding document element linked to the page.

16. The method of claim 11 wherein providing an interactive user interface further comprises:

playing back the multimedia representation of a portion of a document page when the portion of the document page is selected in a document viewing interface; and

automatically resizing the selected portion of the page in the multimedia representation.

17. The method defined in claim 11 further comprising adding a voice annotation to the multimedia representation.

18. The method defined in claim 11 further comprising adding a tag to the multimedia representation.

19. The method defined in claim 11 wherein displaying the multimedia representation in response to a user activating a user control on a document viewer interface for viewing the input document.

20. An apparatus comprising:

a memory to store an input document; and

a processor coupled with the memory to execute a

a multimedia thumbnail generator to extract a plurality of visual, audible, and audiovisual document elements from content contained within the input document and automatically generate a multimedia representation from one or more visual, audible, or audiovisual document elements automatically selected from extracted document elements from the input document based on one or more optimizing constraints to maximize a total of information content that is provided by the selected document element in a given amount of time, wherein the one or more optimizing constraints comprises one or more characteristics of a device and one or more application constraints, wherein the automatic selection comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting one or more visual, audible, or audiovisual document elements based on each time attribute and each information attribute, wherein each time attribute is calculated, using one or more characteristics of a device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, wherein each information attribute is calculated, using the one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein the automatic selection of two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises the processor configured to;

select audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

fill the one or more time intervals of the audible channel with the selected audible document elements, and

a graphical user interface, coupled to the multimedia thumbnail generator, to present the automatically selected visual, audible, and audiovisual document elements and to receive a user input to edit the multimedia representation.

21. The apparatus of claim 20, wherein the graphical user interface includes:

a document page display view to display the document elements of the multimedia representation in a context of a page of the input document and to receive the user input.

22. The apparatus of claim 20, wherein the graphical user interface includes:

one or more dialog boxes to display the document elements of the multimedia representation and to receive the user input.

23. The apparatus of claim 20, wherein the graphical user interface uses different visual indicia to indicate a different selection status of each document element.

24. The apparatus of claim 20, wherein the graphical user interface provides a menu of presentation actions associated with each document element for a user to select one of the presentation actions for the associated document element.

25. The apparatus of claim 20, wherein the graphical user interface provides a printing interface for a user to specify one or more parameters that constrain printing of the multimedia representation to a target.

26. The apparatus of claim 25, wherein the printing interface includes one or more of the following:

a content selection section, an audio setting section, and an imported content section.

27. An apparatus comprising:

a memory to store an input document; and

a processor coupled with the memory to execute a

a multimedia thumbnail generator to analyze the input document and extract a plurality of visual, audible, and audiovisual document elements from the input document and automatically generate a multimedia representation from one or more visual, audible, or audiovisual document elements automatically selected from extracted document elements from the input document based on one or more optimizing constraints and a user input, wherein the one or more optimizing constraints are utilized to maximize a total of information content provided by the selected document elements in a given amount of time, and comprise one or more characteristics of a device displaying a graphical user interface and one or more application constraints, wherein the automatic selection comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting one or more visual, audible, or audiovisual document elements based on each time attribute and each information attribute, wherein each time attribute is calculated, using one or more characteristics of a device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, and wherein each information attribute is calculated, using the one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein the automatic selection of two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises the processor configured to;

select audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

fill the one or more time intervals of the audible channel with the selected audible document elements, and

a graphical user interface, coupled to the multimedia thumbnail generator, to present the multimedia representation and to switch between the presentation of the multimedia representation and presentation of the input document under user control.

28. The apparatus of claim 27 further comprising:

a document viewing interface for a user to view the input document using manual control.

29. The apparatus of claim 27 further comprising:

an automatic viewing interface to automatically resize and navigate the multimedia representation.

30. The apparatus of claim 27 further comprising:

a stop indicator associated with at least one of the document elements to allow a user to stop the presentation of the multimedia representation and to switch to the presentation of the input document.

31. The apparatus of claim 27 further comprising:

at least one of the following indicators;

play indicator, pause indicator, fast forward indicator, fast backward indicator, skipping indicator embedded in at least a page of the input document to allow a user to control the multimedia representation from a corresponding document element linked to the page.

32. A non-transitory computer readable storage medium with instructions thereon which, when executed by a system, cause the system to perform a method comprising:

extracting a plurality of visual, audible, and audiovisual document elements from content within an input document, wherein the input document comprises a single file;

automatically selecting two or more visual, audible, or audiovisual document elements from the extracted visual, audible, and audiovisual document elements of the input document for inclusion into a multimedia representation of the input document based on one or more optimizing constraints to maximize a total of information content provided by the selected two or more visual, audible, or audiovisual document elements in a given amount of time, wherein the one or more optimizing constraints comprises one or more characteristics of a device displaying a graphical user interface and one or more application constraints, wherein the automatically selecting comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting one or more visual, audible, or audiovisual document elements based on each time attribute and each information attribute, wherein each time attribute is calculated, using the one or more characteristics of the device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, and wherein each information attribute is calculated, using the one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein automatically selecting two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises;

selecting audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

filling the one or more time intervals of the audible channel with the selected audible document elements;

automatically generating the multimedia representation based on the automatically selected one or more visual, audible, and audiovisual document elements;

providing a graphical user interface to view the automatically selected visual, audible, and audiovisual document elements and to edit the multimedia representation.

33. The non-transitory computer readable storage medium of claim 32 wherein providing a user interface further comprises:

providing the graphical user interface for a user to specify one or more parameters that constrain a visual setting or an audio setting of the multimedia representation.

34. A non-transitory computer readable storage medium with instructions thereon which, when executed by a system, cause the system to perform a method comprising:

extracting a plurality of visual, audible, and audiovisual document elements from content contained within an input document, wherein the input document comprises a single file;

automatically selecting two or more visual, audible, or audiovisual document elements from the extracted visual, audible, and audiovisual document elements of the input document for inclusion into a multimedia representation of the input document based on one or more optimizing constraints to maximize a total of information content provided by the selected two or more visual, audible, or audiovisual document elements in a given amount of time, wherein the one or more optimizing constraints comprise one or more characteristics of a device displaying a graphical user interface and one or more application constraints, wherein the automatic selection comprises calculating a time attribute and an information attribute for each of the extracted visual, audible, and audiovisual document elements and selecting one or more visual, audible, or audiovisual document elements based on each time attribute and each information attribute, wherein each time attribute is calculated, using the one or more characteristics of the device that is to display the multimedia representation, to determine a minimum presentation time for a user to comprehend each of the extracted visual, audible, and audiovisual document elements, and wherein each information attribute is calculated, using the one or more application constraints, to determine an amount of information content that is contained in each of the extracted visual, audible, and audiovisual document elements, and wherein automatically selecting two or more of the extracted visual, audible, or audiovisual document elements to maximize the total of information content provided by the selected visual, audible, or audiovisual document elements in the given amount of time, further comprises;

selecting audible document elements that maximize an amount of information contained in the selected audible document elements when presented in one or more time intervals of an audible channel of the multimedia representation that have not been filled with audio data associated with the selected audiovisual elements, and

filling the one or more time intervals of the audible channel with the selected audible document elements;

automatically generating the multimedia representation based on the automatically selected one or more visual, audible, or audiovisual document elements;

providing an interactive user interface to playback a combination of the multimedia representation and the input document under user control.

35. The non-transitory computer readable storage medium of claim 34 wherein providing an interactive user interface further comprises:

embedding a play indicator in at least a page of the input document; and

activating the play indicator to play back the multimedia representation from a corresponding document element linked to the page.

36. The non-transitory computer readable storage medium of claim 34 wherein providing an interactive user interface further comprises:

playing back the multimedia representation of a portion of a document page when the portion of the document page is selected in a document viewing interface; and

automatically resizing the selected portion of the page in the multimedia representation.