TRIGGERING ACTIONS WITH CAPTURED INPUT IN A MIXED MEDIA ENVIRONMENT

• US 20070046982A1
• Filed: 07/31/2006
• Published: 03/01/2007
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Erol, Berna, Lee, Dar-Shyang, Hart, Peter, Hull, Jonathan
• Granted Patent: US 7,991,778 B2
• Time in Patent Office: Days
• US Class Current: 358/1.150
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition
• CPC Class Codes: H04N 1/32133 on the same paper sheet, e.g. a facsimile page header
• CPC Class Codes: H04N 2201/3226 of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• CPC Class Codes: H04N 2201/3249 data relating to a linked page or object, e.g. hyperlink
• CPC Class Codes: H04N 2201/3269 of machine readable codes or marks, e.g. bar codes or glyphs

Abstract

A Mixed Media Reality (MMR) system and associated techniques provide mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). The MMR system includes an action processor and methods, and an MMR document with having an associated action. The MMR document specifies different actions for different MMR documents, and creates any number of MMR documents for a particular location on any media, thereby allowing the MMR architecture to serve as a universal trigger or initiator for additional processing. The action processor receives the output of the MMR recognition process which yields an MMR document including at least one action. The action processor executes that action which includes various commands to the MMR system or other systems coupled to the MMR system. The MMR system architecture can perform action such as retrieving the electronic form of the document to the capture device, retrieving the specification for the action, inserting data to a MMR document, transferring data between documents, purchasing items, authoring actions or reviewing historical information about actions.

Claims

1. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a capture device for capturing a first representation of the first type of medium;

a mixed media processor for recognizing a mixed media document, having a second representation of the first type of medium, an index and an action, by comparing the second representation to the first representation; and

a media storage for a storing the mixed media document.

2. The system of claim 1 further comprising an action processing unit coupled the mixed media processor and adapted to perform the action of the recognized mixed media document.

3. The system of claim 1 wherein the capture device further comprises an action processing unit coupled the mixed media processor and adapted to perform the action of the recognized mixed media document.

4. The system of claim 1 wherein the mixed media processor further comprises an action processing unit coupled adapted to perform the action of the recognized mixed media document.

5. The system of claim 1 wherein the media storage further comprises an action processing unit coupled the mixed media processor and adapted to perform the action of the recognized mixed media document.

6. The system of claim 1 wherein the media storage is a database that stores mixed media documents and actions, and includes a mapping table and an index table.

7. The system of claim 1 wherein the media storage includes a first database for storing mixed media documents and actions accessible to a particular user, and a second database accessible to a plurality of users including the particular user.

8. The system of claim 2 wherein the action processing unit further comprises a retrieve module for retrieving data linked to a location associated with the first representation from the mixed media document, the retrieve module coupled to the media storage for accessing the mixed media document.

9. The system of claim 8 wherein the data includes menus that specify subsequent steps to be performed on the capture device.

10. The system of claim 8 wherein the data is static data for display on the capture device.

11. The system of claim 8 wherein the data is stored at a remote location and the retrieve module performs the steps necessary to retrieve the data from the remote location.

12. The system of claim 2 wherein the action processing unit further comprises an insert module for adding data to a location in the recognized mixed media document, the insert module adapted to receive data from the capture device and adapted to store the data in the media storage.

13. The system of claim 2 wherein the action processing unit further comprises an insert module for adding data from a location associated with the first representation into a file, the insert module adapted to receive data from the media storage and adapted to store the data in the file.

14. The system of claim 2 wherein the action processing unit further comprises a transfer module for transferring data linked to a location in the mixed media document from a source to a destination, the transfer module adapted to obtain the data from the source and send the data to the destination.

15. The system of claim 2 wherein the action processing unit further comprises a purchase module for purchasing merchandise linked to a location in the mixed media document, the purchase module adapted to specify a purchase item, send an order to a seller and complete the purchase.

16. The system of claim 2 wherein the action processing unit further comprises an authoring module for creating a new action and adding the new action to the mixed media document associated with the first representation, the authoring module adapted to access and modify the mixed media document.

17. The system of claim 2 wherein the action is creation of a second mixed media document.

18. The system of claim 1 further comprising a historical action record module for recording actions executed by the mixed media processor, the historical action record module adapted for communication with the mixed media processor to receive information about action that have been executed, and adapted for storing information in the media storage.

19. The system of claim 18, wherein the information is one from the group of a user that executed the action, a time when the action was executed, a MMR system component used to perform the action, and a context in which the action was executed.

20. The system of claim 18, wherein the historical action record module is adapted to retrieve historical actions performed in response to input of a location in the mixed media document and display the retrieved historical actions on the capture device, the historical action record module coupled to the media storage and the capture device.

21. The system of claim 1 further comprising a historical analysis module for analyzing actions executed by the mixed media processor to discover patterns, the historical analysis module adapted for communication with the mixed media processor to receive information about actions that have been executed, and adapted for storing patterns in the media storage.

22. The system of claim 2 wherein the action processing unit further comprises an action execution queue ordering performance of actions, the action processing unit adapted to receive actions from the mixed media processor and provide them to the action processing unit.

23. A method for performing an action in a mixed media environment having a first type of medium and a second type of medium, the method comprising:

acquiring an input representation;

comparing the input representation to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media document including an action;

identifying a matching stored representation that matches the input representation and the mixed media document associated with the matching stored representation; and

performing the action of the mixed media document associated with the matching stored representation.

24. The method of claim 23 wherein the action is retrieving static data for display on the capture device.

25. The method of claim 23 wherein the action is retrieving data stored at a remote location.

26. The method of claim 23 wherein the action is inserting data to a location in the mixed media document associated with the captured representation.

27. The method of claim 23 wherein the action is inserting data from a location in the recognized mixed media document into a file.

28. The method of claim 23 wherein the action is transferring data linked to a location in the mixed media document from a source to a destination.

29. The method of claim 23 wherein the action is purchasing merchandise linked to a location in the mixed media document, purchasing including specifying a purchase item, sending an order to a seller and completing the purchase.

30. The method of claim 23 wherein the action is authoring a new action and adding the new action to the mixed media document associated with the input representation.

32. The method of claim 23 further comprising retrieving historical actions performed in response to input of a location in the mixed media document and displaying the retrieved historical actions on the capture device.

33. The method of claim 23 further comprising analyzing actions performed to discover patterns.

46. The method of claim 23 further comprising retrieving historical actions performed in response to input of a location in the mixed media document and displaying the retrieved historical actions on the capture device.

31. The method of claim 23 wherein the action is creation of a second mixed media document.

31-1. (cancel)

34. A machine-readable medium encoded with instructions, that when executed by one or more processors, cause the processor to carry out a process for organizing and accessing information in a mixed media document system, the process comprising:

acquiring an input representation;

comparing the input representation to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media document including an action;

identifying a matching stored representation that matches the input representation and the mixed media document associated with the matching stored representation; and

performing the action of the mixed media document associated with the matching stored representation.

35. The machine-readable medium of claim 34 wherein the action is retrieving static data for display on the capture device.

36. The machine-readable medium of claim 34 wherein the action is retrieving data stored at a remote location.

37. The machine-readable medium of claim 34 wherein the action is inserting data to a location in the mixed media document associated with the captured representation.

38. The machine-readable medium of claim 34 wherein the action is inserting data from a location in the recognized mixed media document into a file.

39. The machine-readable medium of claim 34 wherein the action is transferring data linked to a location in the mixed media document from a source to a destination.

40. The machine-readable medium of claim 34 wherein the action is purchasing merchandise linked to a location in the mixed media document, purchasing including specifying a purchase item, sending an order to a seller and completing the purchase.

41. The machine-readable medium of claim 34 wherein the action is authoring a new action and adding the new action to the mixed media document associated with the input representation.

42. The machine-readable medium of claim 34 wherein the action is creation of a second mixed media document.

43. The machine-readable medium of claim 34, the process further comprising recording information about actions performed, and wherein the information is one from the group of a user that performed the action, a time when the action was performed, a MMR system component used to perform the action, and a context in which the action was performed.

44. The machine-readable medium of claim 34, the process further comprising retrieving historical actions performed in response to input of a location in the mixed media document and displaying the retrieved historical actions on the capture device.

45. The machine-readable medium of claim 34, the process further comprising analyzing actions performed to discover patterns.