Integration and use of mixed media documents

• US 8,195,659 B2
• Filed: 07/31/2006
• Issued: 06/05/2012
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Graham, Jamey, Hart, Peter E., Nudd, Geoffrey H., Weyl, Stephen
• Primary Examiner(s): Badawi, Sherief
• Application Number:
  US11/461,126
• Publication Number:
  US 20070046983A1
• Time in Patent Office: 2,136 Days
• Field of Search: 707/5, 707/3, 707/1, 707/707, 707/765, 707/758, 707/769, 707/602, 707/736, 382/124, 705/37, 715/209, 715/738
• US Class Current: 707/736
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/32133 on the same paper sheet, e.g. a facsimile page header
• CPC Class Codes: H04N 2201/3226 of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• CPC Class Codes: H04N 2201/3249 data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3269 of machine readable codes or marks, e.g. bar codes or glyphs

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one embodiment, an MMR document is retrieved based on recognition of a paper document. Responsive to the comparison of the paper document and the virtual multimedia document, an action is performed. For example, the media of the matching MMR document can be displayed, or an action associated with the matching MMR document can be performed.

Claims

1. A system for accessing mixed media reality document data structures, the system comprising:

a mixed media processor for receiving, from a device, an image patch of a commercially printed document that includes a hotspot designation location, the image patch comprising at least a portion of an entire page of the commercially printed document that includes the hotspot designation location, for extracting features from the image patch including generating a histogram to discriminate between text and non-text, for determining a mixed media reality document data structure that corresponds to the image patch based on extracted features and the histogram, for capturing an x-y location of a hotspot in response to receiving a print command by identifying a begin mark designating a beginning point of the hotspot designation location and an end mark designating an ending point of the hotspot designation location, the begin mark and the end mark being represented by a visually imperceptible font, for instructing the device to highlight the hotspot based on the determining of the mixed media reality document data structure and for retrieving the mixed media reality document data structure, the mixed media reality document data structure comprising;

a representation of the portion of the commercially printed document corresponding to the image patch;

a supplemental media associated with the image patch; and

the hotspot that is sent for display along with the representation, the hotspot comprising a pointer to the supplemental media, such that the supplemental media is sent in response to receipt of an indication that a user selected the hotspot.

2. The system of claim 1, further comprising a communication mechanism coupled to the mixed media processor.

3. The system of claim 2, wherein the communication mechanism is one from the group of a network, a wired coupling, an infrared link, a Bluetooth link, a wireless communication link, a cellular communication link, a data communication link, a mobile data service, a Global System for Mobile Communications, and a public switched telephone network.

4. The system of claim 1, wherein the commercially printed document is a paper document.

5. The system of claim 1, wherein the mixed media processor further performs an action associated with the retrieved supplemental media responsive to receiving instructions from the device, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

6. The system of claim 2, further comprising the device coupled to the communication mechanism capable of receiving one of digital data and an image representing a portion of the commercially printed document.

7. The system of claim 6, wherein the device is one from the group of a cellular camera phone, a Personal Digital Assistant device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera, and a video card.

8. The system of claim 1, further comprising:

a capture mechanism for acquiring a media file and associating the media file with the mixed media reality document data structure.

9. The system of claim 8, further comprising:

an electronic medium for storing the media file associated with the mixed media reality document data structure.

10. The system of claim 1, further comprising a storage repository for a plurality of mixed media document data structures, the repository coupled to the mixed media processor.

11. The system of claim 1, further comprising an output device for outputting the supplemental media associated with the mixed media reality document data structure.

12. The system of claim 1, wherein the mixed media reality document data structure further comprises an electronic representation of the commercially printed document.

13. The system of claim 12, wherein the mixed media processor further displays the electronic representation of the commercially printed document.

14. The system of claim 1, wherein the mixed media reality document data structure further comprises an associated action.

15. The system of claim 14, further comprising performing the associated action.

29. The system of claim 1, wherein the commercially printed document is a tourist travel book.

30. The system of claim 29, wherein the supplemental media is an article associated with a portion of the tourist travel book corresponding to the image patch.

31. The system of claim 29, wherein the supplemental media is a media file associated with a portion of the tourist travel book corresponding to the image patch.

32. The system of claim 29, wherein the supplemental media is an updated version of the tourist travel book.

33. The system of claim 29, wherein the supplemental media is an advertisement associated with a portion of the tourist travel book corresponding to the image patch.

34. The system of claim 1, wherein the commercially printed document is an article about a publicly-traded company.

35. The system of claim 34, wherein the supplemental media is a user'"'"'s stock holding information related to the publicly-traded company.

36. The system of claim 34, wherein retrieving supplemental media associated with the publicly-traded company results in an option to purchase stock associated with the publicly traded company.

37. The system of claim 1, wherein the image patch of the commercially printed document is associated with an advertisement within the commercially printed document.

38. The system of claim 37, wherein retrieving the supplemental media associated with the advertisement results in a payment to a user associated with the supplemental media.

39. The system of claim 37, wherein retrieving the supplemental media associated with the advertisement requires user payment for the supplemental media.

40. The system of claim 1, wherein the commercially printed document is a conference guide.

41. The system of claim 40, wherein the supplemental media is a media file captured during a conference associated with the conference guide.

16. An electronically-implemented method for accessing mixed media reality document data structures, the method comprising:

receiving, from a device, an image patch of a commercially printed document that includes a hotspot designation location, the image patch comprising at least a portion of an entire page of the commercially printed document that includes the hotspot designation location;

extracting features from the image patch including generating a histogram to discriminate between text and non-text;

determining a mixed media reality document data structure that corresponds to the image patch based on extracted features and the histogram;

capturing an x-y location of a hotspot in response to receiving a print command by identifying a begin mark designating a beginning point of the hotspot designation location and an end mark designating an ending point of the hotspot designation location, the begin mark and the end mark being represented by a visually imperceptible font;

instructing the device to highlight the hotspot based on the determining of the mixed media reality document data structure;

retrieving the mixed media reality document data structure that corresponds to the image patch, the mixed media reality document data structure comprising;

a representation of the portion of the commercially printed document corresponding to the image patch; and

the hotspot that is sent for display along with the representation, the hotspot comprising a pointer to a supplemental media; and

responsive to receiving an indication that a user selected the hotspot, sending the supplemental media associated with the hotspot.

17. The method of claim 16, wherein the mixed media reality document data structure includes a plurality of media files, and further comprising:

receiving a user selection of one of the media files; and

displaying the user-selected media file.

18. The method of claim 16, wherein the mixed media reality document data structure further comprises an electronic representation of the commercially printed document.

19. The method of claim 18, further comprising displaying the electronic representation of the commercially printed document.

20. The method of claim 16, wherein the mixed media reality document data structure further comprises an associated action.

21. The method of claim 20, further comprising performing the associated action.

22. The method of claim 16, wherein the mixed media reality document data structure further comprises a plurality of associated actions, and further comprising:

receiving a user selection of one of the associated actions; and

performing the user-selected associated action.

23. The method of claim 16, further comprising:

storing a media file; and

associating the media file with the mixed media reality document data structure.

24. The method of claim 16, wherein the hotspot indicates a plurality of locations in the commercially printed document and a plurality of media files associated with the plurality of locations, and further comprising:

receiving a user selection of one of the locations; and

displaying the media file associated with the user-selected location.

25. The method of claim 16, wherein the representation is one from the group of a representation of the entire document, a representation of patch of text, a representation of single word, a representation of a section of an image, and any other representation of a matchable portion of the commercially printed document.

26. The method of claim 16, wherein receiving the image patch of the commercially printed document comprises one from the group of receiving a scanned a patch of text, receiving a scanned patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, receiving a scanned machine-readable identifier, receiving input data that identifies the commercially printed document, receiving an email with the commercially printed document attached, receiving the commercially printed document by file transfer, receiving a scanned portion of an image, and receiving the commercially printed document that was input.

27. The method of claim 16, wherein comparing the commercially printed document and the mixed media reality document data structure further comprises performing pattern matching between the commercially printed document and the mixed media reality document data structure.

28. The method of claim 16, further comprising performing an action associated with the retrieved supplemental media, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.

42. A computer program product comprising a machine-readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive, from a device, an image patch of a commercially printed document that includes a hotspot designation location, the image patch comprising at least a portion of an entire page of the commercially printed document that includes the hotspot designation location;

extract features from the image patch including generating a histogram to discriminate between text and non-text;

determine a mixed media reality document data structure that corresponds to the image patch based on extracted features and the histogram;

capture an x-y location of a hotspot in response to receiving a print command by identifying a begin mark designating a beginning point of the hotspot designation location and an end mark designating an ending point of the hotspot designation location, the begin mark and the end mark being represented by a visually imperceptible font;

instruct the device to highlight the hotspot based on the determining of the mixed media reality document data structure;

retrieve the mixed media reality document data structure that corresponds to the image patch, the mixed media reality document data structure comprising;

a representation of the portion of the commercially printed document corresponding to the image patch; and

the hotspot that is sent for display along with the representation, the hotspot comprising a pointer to a supplemental media; and

responsive to receipt of an indication that a user selected the hotspot, sending the supplemental media associated with the hotspot.