TRIGGERING APPLICATIONS FOR DISTRIBUTED ACTION EXECUTION AND USE OF MIXED MEDIA RECOGNITION AS A CONTROL INPUT

• US 20070050341A1
• Filed: 07/31/2006
• Published: 03/01/2007
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Erol, Berna, Lee, Dar-Shyang, Hart, Peter, Hull, Jonathan
• Granted Patent: US 7,920,759 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium. In one embodiment of the present invention, the MMR system includes an action processor and method, and MMR documents with an associated action. The MMR document structure specifies different actions for different MMR documents combined with the ability to create any number of MMR documents for a particular location on any media, which allows the MMR architecture to serve as a universal trigger or initiator for additional processing. The action processor receives the output of the MMR recognition process which yields an MMR document including at least one action. The action processor executes that action which includes various commands to the MMR system or other systems coupled to the MMR system. The MMR system executes the action using other system and the MMR software is distributed among the MMR system and these other systems. The MMR system architecture is advantageous because an action can be executed by pointing the capture device at a rendered document, and the action is performed. Example actions include retrieving the electronic form of the rendered document, retrieving the specification for the action, inserting data to a MMR document, transferring data between documents, purchasing items, authoring actions or reviewing historical information about actions. In another embodiment, the MMR system, in particular MMR recognition and the input/output capabilities of the capture device, are used as a control mechanism for a third party system.

Claims

1. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a capture device for capturing a first representation of a document in the first type of medium;

a media storage for a storing a plurality of mixed media documents, each mixed media document having an index, an action and a second representation of a corresponding document in the first type of medium;

a recognition unit for identifying a mixed media document from the plurality of mixed media documents that has a second representation that matches the first representation; and

a server coupled for communication with the capture device, the media storage and the recognition unit, the server including an action processor for receiving an action of the identified mixed media document, and performing the action of the identified mixed media document.

2. The system of claim 1 further comprising:

a second action processor coupled to the server; and

wherein the action is the initiation of a plurality of second actions, a first set of at least one of said plurality of second actions executed by the action processor of the server, a second set of at least one of said plurality of second actions executed by the second action processor.

3. The system of claim 2 wherein the action is a uniqueness measurement for a graphic and the plurality of second actions are performing mixed media recognition of the graphic against mixed media document stored in the media storage using a retrieve command.

4. The system of claim 1 further comprising:

a second action processor for performing actions and coupled to the server;

wherein the identified document includes a first action and a second action; and

wherein the server includes a controller to distribute the second action for execution on the second action processor, the controller adapted to send the second action processor the second action and receive a result.

5. The system of claim 4, wherein the second action processor is external to the system.

6. The system of claim 5, wherein the second action processor is included in one from the group of a user computer, a media server, a service provider, a set-top box, an office portal and a third party server.

7. The system of claim 4, wherein the second action processor is included in the media storage.

8. The system of claim 4, wherein the second action processor is included in the capture device.

9. The system of claim 1, further comprising:

a second capture device for capturing a third representation of a document in the first type of medium;

a second recognition unit for identifying a mixed media document that has the second representation that matches the third representation, the second unit coupled to the second capture device;

a second action processor for performing actions and coupled to the server; and

wherein the media storage includes a shared space to which the first action processor or the second action processor are adapted to store information for collaborative annotation of the mixed media document.

10. The system of claim 1, wherein:

the server is capable of being coupled to an output device for loading a media file from the server to the output device;

the server is capable of sending signals to control the output device; and

the server adapted to process an action to control the output of a loaded media file by the output device.

11. The system of claim 10, wherein the output device is one from the group of an MP3 player and a video player.

12. The system of claim 1, wherein the server also includes a module for playing, saving, and forwarding voicemails.

13. A first system for controlling a second system, the first system providing a mixed media environment having a first type of medium and a second type of medium, the first system comprising:

a capture device for capturing a first representation of a document in the first type of medium;

a media storage for a storing a plurality of mixed media documents, each mixed media document having an index, an action and a second representation of a corresponding document in the first type of medium;

a recognition unit for identifying a mixed media document from the plurality of mixed media documents that has a second representation that matches the first representation; and

a server coupled for communication with the capture device, the media storage, the recognition unit and the second system, the server adapted to control the second system in response to an action in a mixed media document.

14. The first system of claim 13, wherein the second system includes a display device coupled to a set-top box, the set-top box coupled to a cable network;

and wherein the server is coupled to the cable network to send signals to the set-top box.

15. The first system of claim 14, wherein the server is adapted to execute actions including controlling the set-top box coupled to input commands for cable services and controls.

16. The first system of claim 14, wherein the server is adapted to execute actions including retrieving information related to an image captured by the capture device from the display device.

17. The first system of claim 14, wherein the server is adapted to execute actions to purchase an item related to an image captured by the capture device from the display device.

18. The first system of claim 13, wherein:

the second system is a computer having a display and being coupled to the server by a network;

the server is adapted to cooperate with the second system to perform the action; and

the capturing of a representation of a document by the capture device and recognition of the representation by the recognition unit causes the server to perform the action specified by identified mixed media document.

19. The first system of claim 18, wherein the action performed by cooperation of the server and the second system is one from the group of:

initiating video replay on the computer;

displaying video clips that show changes in the display of the computer while the representation was visible;

displaying traces of web pages that were viewed while the representation was shown on the display of the computer;

displaying a history of how a form was completed; and

retrieving a source file and a printed representation of past and future versions of the document.

20. A method for performing an action in a mixed media environment having a first type of medium and a second type of medium, the method comprising:

acquiring an input representation;

comparing the input representation to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media document having a first action and a second action;

identifying a matching stored representation that matches the input representation and the mixed media document associated with the matching stored representation; and

performing the first action of the mixed media document associated with the matching stored representation; and

performing the second action of the mixed media document associated with the matching stored representation.

21. The method of claim 20 wherein the first action and the second action perform mixed media recognition of a graphic against mixed media documents stored in the media storage using a retrieve command.

22. The method of claim 20 wherein a second action processor that is part of an external system executes the step of performing the second action.

23. The method of claim 20 wherein a second action processor executes the step of performing the second action and is included in one from the group of a user computer, a media server, a service provider, a set-top box, an office portal and a third party server.

24. The method of claim 20 wherein a second action processor that is part of media storage executes the step of performing the second action.

25. The method of claim 20 wherein a second action processor that is part of a capture device executes the step of performing the second action.

26. The method of claim 20 wherein the first action is loading a media file from a server to an output device, and wherein the second action is sending a signal to control an output of the loaded media file by the output device

27. A method for controlling a system using an action in a mixed media environment having a first type of medium and a second type of medium in response to recognition of a representation, the method comprising:

acquiring an input representation;

comparing the input representation to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media document including an action;

identifying a matching stored representation that matches the input representation and the mixed media document associated with the matching stored representation; and

controlling the system by performing the action of the mixed media document associated with the matching stored representation.

28. The method of claim 27, wherein the system includes a display device coupled to a set-top box, the set-top box coupled to a cable network;

and wherein performing the action generates control signals that are sent to the set-top box.

29. The method of claim 28, wherein the generated control signals input commands for cable services and controls.

30. The method of claim 28, wherein the generated control signals input commands for retrieving information related to an image captured by the capture device from the display device.

31. The method of claim 28, wherein the generated control signals input commands to purchase an item related to an image captured by the capture device from the display device.

32. The method of claim 27, wherein the system is a computer having a display, and wherein acquiring of the input representation by a capture device and identification of the matching stored representation causes performance of the action specified by identified mixed media document.

33. The method of claim 27, wherein the action is one from the group of:

initiating video replay on the computer;

displaying video clips that show changes in the display of the computer while the representation was visible;

displaying traces of web pages that were viewed while the representation was shown on the display of the computer;

displaying a history of how a form was completed; and

retrieving a source file and a printed representation of past and future versions of the document.