System and methods for portable device for mixed media system

• US 7,669,148 B2
• Filed: 07/31/2006
• Issued: 02/23/2010
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Hart, Peter E., Hull, Jonathan J.
• Primary Examiner(s): Lee; Ting
• Assistant Examiner(s): ZAHR, ASHRAF A
• Application Number:
  US11/461,064
• Publication Number:
  US 20070052997A1
• Time in Patent Office: 1,303 Days
• Field of Search: 715/864, 715/241, 715/277, 715/760, 715209-210
• US Class Current: 715/864
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32101 Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
• CPC Class Codes: H04N 2201/0081 Image reader H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0082 Image hardcopy reproducer H04N2201/0091 - H04N2201/0094 take precedence
• CPC Class Codes: H04N 2201/0084 Digital still camera
• CPC Class Codes: H04N 2201/3205 of identification information, e.g. name or ID code H04N2201/3209 takes precedence
• CPC Class Codes: H04N 2201/3216 of a job size, e.g. a number of images, pages or copies, size of file, length of message
• CPC Class Codes: H04N 2201/3274 Storage or retrieval of prestored additional information
• CPC Class Codes: H04N 2201/3278 Transmission

Abstract

A Mixed Media Reality (MMR) system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system facilitates business methods that take advantage of the combination of a portable electronic device, such as a portable video player or a portable audio player, and a paper document for the portable electronic device to provide information.

Claims

1. A method of providing information related to a portion of a printed document, the method comprising:

storing as a data structure a mixed media reality document associated with the printed document in a portable device, the data structure comprising a first representation of at least a portion of the printed document for pattern matching a first location associated with the printed document, at least one media, and a hotspot that links the at least one media to both the first representation and a selectable portion of a second representation of the printed document, the selectable portion corresponding to a second location in the second representation that is indicated by a bounding box;

displaying on the portable device the selectable portion of the second representation of the printed document and the bounding box corresponding to the selectable portion; and

in response to a user selection of the selectable portion of the second representation of the printed document on the portable device;

determining the hotspot corresponding to the selected selectable portion of the second representation of the printed document; and

displaying the at least one media that is linked by the determined hotspot with the selected selectable portion of the second representation of the printed document.

2. The method of claim 1 wherein the portable device includes an audio player and wherein displaying the at least one media includes playing audio of the at least one media that is linked with the selected selectable portion of the second representation of the printed document.

3. The method of claim 1 wherein displaying the at least one media includes displaying text of the at least one media that is linked with the selected selectable portion of the image second representation of the printed document.

4. The method of claim 1 wherein the portable device includes a video player and wherein displaying the at least one media includes playing video of the at least one media that is linked with the selected selectable portion of the second representation of the printed document.

5. The method of claim 1 further comprising:

receiving a user audio command;

converting the user audio command into a signal indicative of the user selection of a selectable portion of the second representation of the printed document.

6. A portable information device for providing information related to a portion of a printed document, the system comprising:

a communication mechanism for receiving as a data structure a mixed media reality document associated with the printed document, the data structure comprising a first representation of at least a portion of the printed document for pattern matching a first location associated with the printed document, at least one media, and a hotspot that links the at least one media to both the first representation and a selectable portion of a second representation of the printed document, the selectable portion corresponding to a second location in the second representation that is indicated by a bounding box;

a memory for storing the mixed media reality document;

a display device for displaying the selectable portion of the second representation of the printed document and the bounding box corresponding to the selectable portion; and

a mechanism for determining the hotspot corresponding to the selected selectable portion of the second representation of the printed document and for providing, to the display device for display, the at least one media that is linked by the determined hotspot with the selected selectable portion of the second representation of the printed document, wherein the determining and displaying is in response to a user selection of the selectable portion of the second representation of the printed document.

7. The portable information device of claim 6 wherein the mechanism includes an audio player that plays audio associated with the at least one media that is linked with the selected selectable portion of the second representation of the printed document.

8. The portable information device of claim 6 wherein the mechanism displays text associated with the at least one media that is linked with the selected selectable portion of the second representation of the printed document.

9. The portable information device of claim 6 wherein the mechanism includes a video player that plays video associated with the at least one media that is linked with the selected selectable portion of the second representation of the printed document.

10. A method of providing information related to a portion of a printed document, the method comprising:

storing as a data structure a mixed media reality document associated with the printed document in a portable device, the data structure comprising a first representation of at least a portion of the printed document for pattern matching of a first location associated with the printed document, at least one media, and a hotspot that links the at least one media to both the first representation and a selectable portion of a second representation of the printed document, the selectable portion corresponding to a second location in the second representation that is indicated by a bounding box;

providing hot spot location information corresponding to the hotspot to a capture device;

in response to receiving a user selection of the selectable portion of the second representation of the printed document at the capture device;

determining the hotspot corresponding to the selected selectable portion of the second representation of the printed document;

retrieving the at least one media that is linked by the determined hotspot with the selected selectable portion of the second representation of the printed document; and

providing to the capture device the retrieved at least one media that is linked with the selected selectable portion of the second representation of the printed document.

11. The method of claim 10 wherein receiving a user selection of the selected selectable portion is via a wireless communication mechanism.

12. The method of claim 10 wherein receiving a user selection of the selected selectable portion is via an infrared communication mechanism.

13. The method of claim 10 wherein receiving a user selection of the selected selectable portion is via an inaudible audio communication mechanism.

14. The method of claim 10 wherein the portable device and the capture device are formed as an integrated device.

15. The method of claim 10 further comprising:

receiving user selection of printed documents that the user subscribes to; and

downloading information and hotspot information related to the printed documents that the user subscribes to.

16. A mixed media information device for providing information related to a portion of a printed document, the device comprising:

a first communication mechanism for receiving as a data structure a mixed media reality document associated with a printed document, the data structure comprising a first representation of at least a portion of the printed document for pattern matching of a location associated with the printed document, at least one media, and a hotspot that links the at least one media to a selectable portion of a second representation of the printed document, the selectable portion corresponding to a second location in the second representation that is indicated by a bounding box;

a memory for storing the mixed media reality document;

a second communication mechanism for providing hotspot location information corresponding to the hotspot to a capture device

a third communication mechanism for receiving a user selection of the selectable portion of the second representation of the printed document at the capture device;

a mechanism for determining the hotspot corresponding to the selected selectable portion of the second representation of the printed document and for retrieving the at least one media that is linked by the determined hotspot with the selected selectable portion of the second representation of the printed document in response to receiving the user selection of the selectable portion of the second representation of the printed document; and

a fourth communication mechanism for providing to the capture device the retrieved at least one media that is linked with the selected selectable portion of the second representation of the printed document.

17. The mixed media information device of claim 16 wherein the fourth communication mechanism provides the retrieved at least one media to the capture device for communication to the user.

18. The mixed media information device of claim 16 wherein the first communication mechanism receives the mixed media reality document associated with the printed document in response to user selection of the printed document that user has access to.

19. The mixed media information device of claim 16 wherein the fourth communication mechanism includes an audio player that plays audio corresponding to the at least one media that is linked with the selected selectable portion.

20. The mixed media information device of claim 16 wherein the fourth communication mechanism displays text corresponding to the at least one media that is linked with the selected selectable portion.

21. The mixed media information device of claim 16 wherein the fourth communication mechanism includes a video player that plays video corresponding to the at least one media that is linked with the selected selectable portion.