Triggering applications for distributed action execution and use of mixed media recognition as a control input

• US 7,920,759 B2
• Filed: 07/31/2006
• Issued: 04/05/2011
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Primary Examiner(s): Chen; Wenpeng
• Application Number:
  US11/461,049
• Publication Number:
  US 20070050341A1
• Time in Patent Office: 1,709 Days
• Field of Search: 382/305, 358/3.28
• US Class Current: 382/305
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types. In one embodiment of the present invention, the MMR system includes an action processor and method, and MMR documents with an associated action. The MMR document structure specifies different actions for different MMR documents combined with the ability to create any number of MMR documents for a particular location on any media, which allows the MMR architecture to serve as a universal trigger or initiator for additional processing. In another embodiment, the MMR system, in particular MMR recognition and the input/output capabilities of the capture device, are used as a control mechanism for a third party system.

Claims

1. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a capture device for capturing an image of at least a portion of a document in the first type of medium;

a media storage for storing a plurality of mixed media reality document data structures, each mixed media reality document data structure comprising an action, a representation of a corresponding document in the first type of medium, a location of data within the representation that matches the image and an index linking the action with the representation;

a recognition unit for identifying a mixed media reality document data structure from the plurality of mixed media reality document data structures that has a representation that matches the image and for outputting the location of data within the representation that matches the image; and

a server coupled for communication with the capture device, the media storage and the recognition unit, the server including an action processor for receiving an action of the identified mixed media reality document data structure, and performing the action of the identified mixed media reality document data structure based on the location of data within the representation that matches the image.

2. The system of claim 1 further comprising:

a second action processor coupled to the server; and

wherein the action is the initiation of a plurality of second actions, a first set of at least one of said plurality of second actions executed by the action processor of the server, a second set of at least one of said plurality of second actions executed by the second action processor.

3. The system of claim 2 wherein the action is a uniqueness measurement for a graphic and the plurality of second actions are performing mixed media recognition of the graphic against a mixed media reality document data structure stored in the media storage using a retrieve command.

4. The system of claim 1 further comprising:

a second action processor for performing actions, the second action processor coupled to the server;

wherein the identified document includes a first action and a second action; and

wherein the server includes a controller to distribute the second action for execution on the second action processor, the controller adapted to send the second action processor the second action and receive a result.

5. The system of claim 4, wherein the second action processor is external to the system.

6. The system of claim 5, wherein the second action processor is included in one from the group of a user computer, a media server, a service provider, a set-top box, an office portal and a third party server.

7. The system of claim 4, wherein the second action processor is included in the media storage.

8. The system of claim 4, wherein the second action processor is included in the capture device.

9. The system of claim 1, wherein:

the server is capable of being coupled to an output device for loading a media file from the server to the output device;

the server is capable of sending signals to control the output device; and

the server adapted to process an action to control the output of a loaded media file by the output device.

10. The system of claim 9, wherein the output device is one from the group of an MP3 player and a video player.

11. The system of claim 1, wherein the server also includes a module for playing, saving, and forwarding voicemails.

12. A first system for controlling a second system, the first system providing a mixed media environment having a first type of medium and a second type of medium, the first system comprising:

a capture device for capturing an image of at least a portion of a document in the first type of medium;

a media storage for storing a plurality of mixed media reality document data structures, each mixed media reality document data structure comprising an action, a representation of a corresponding document in the first type of medium, a location of data within the representation that matches the image and an index linking the action with the representation;

a recognition unit for identifying a mixed media reality document data structure from the plurality of mixed media reality document data structures that has a representation that matches the image and for outputting the location of data within the representation that matches the image; and

a server coupled for communication with the capture device, the media storage, the recognition unit and the second system, the server adapted to control the second system in response to an action in a mixed media document based on the location of data within the representation that matches the image.

13. A method for performing an action in a mixed media environment having a first type of medium and a second type of medium, the method comprising:

using a computer to perform the steps of;

acquiring an input image;

comparing the input image to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media reality document data structure comprising a first action, a second action, a location of data within the representation that matches the input image and an index linking the first action and second action with the associated stored representation;

identifying a matching stored representation that matches the input image and the mixed media reality document data structure associated with the matching stored representation;

outputting the location of data within the representation that matches the input image;

performing the first action of the mixed media reality document data structure associated with the matching stored representation based on the location of data within the representation that matches the input image; and

performing the second action of the mixed media reality document data structure associated with the matching stored representation.

14. The method of claim 13 wherein the first action and the second action perform mixed media recognition of a graphic against stored mixed media reality document data structures using a retrieve command.

15. The method of claim 13 wherein a second action processor that is part of an external system executes the step of performing the second action.

16. The method of claim 13 wherein a second action processor executes the step of performing the second action and is included in one from the group of a user computer, a media server, a service provider, a set-top box, an office portal and a third party server.

17. The method of claim 13 wherein a second action processor that is part of media storage executes the step of performing the second action.

18. The method of claim 13 wherein a second action processor that is part of a capture device executes the step of performing the second action.

19. The method of claim 13 wherein the first action is loading a media file from a server to an output device, and wherein the second action is sending a signal to control an output of the loaded media file by the output device.

20. A method for controlling a system using an action in a mixed media environment having a first type of medium and a second type of medium in response to recognition of a representation, the method comprising:

acquiring an input image;

comparing the input image to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media reality document data structure comprising an action, a location of data within the representation that matches the input image and an index linking the action with the associated stored representation;

identifying a matching stored representation that matches the input image and the mixed media reality document data structure associated with the matching stored representation;

outputting the location of data within the representation that matches the input image; and

controlling the system by performing the action of the mixed media reality document data structure associated with the matching stored representation based on the location of data within the representation that matches the input image.

21. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a recognition unit for identifying a mixed media reality document data structure from a plurality of mixed media reality document data structures that has a representation that matches an image of at least a portion of a document in the first type of medium that is captured by a capture device and for outputting a location of data within the representation that matches the image;

a media storage for storing a plurality of mixed media reality document data structures, each mixed media reality document data structure comprising an action, the representation of a corresponding document in the first type of medium, the location of data within the representation that matches the image and an index linking the action with the representation; and

a server coupled for communication with the capture device, the media storage and the recognition unit, the server including an action processor for receiving an action of the identified mixed media reality document data structure, and performing the action of the identified mixed media reality document data structure based on the location of data within the representation that matches the image.