Triggering applications based on a captured text in a mixed media environment

• US 7,672,543 B2
• Filed: 07/31/2006
• Issued: 03/02/2010
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Primary Examiner(s): Ahmed; Samir A.
• Assistant Examiner(s): Bayat; Ali
• Application Number:
  US11/461,032
• Publication Number:
  US 20070050360A1
• Time in Patent Office: 1,310 Days
• Field of Search: 382/177, 382/176, 382/175, 382/164, 382/190, 382/159, 382/181, 382/276, 382/305, 382/100, 358/1.15, 358/403, 358/404, 707/1, 707/100, 707/102
• US Class Current: 382/305
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/148 Segmentation of character regions
• CPC Class Codes: G06V 30/158 using character size, text spacings or pitch estimation
• CPC Class Codes: G06V 30/18086 by performing operations within image blocks or by using histograms

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an action processor and method, and MMR documents with an associated action. The MMR document structure is particularly advantageous because the ability to specify different actions for different MMR documents, combined with the ability to create any number of MMR documents for a particular location on any media, allows the MMR architecture to serve as a universal trigger or initiator for additional processing. In other words, addition processing or actions can be triggered or initiated based on MMR recognition. The action processor receives the output of the MMR recognition process which yields an MMR document including at least one action. The action processor executes that action which includes various commands to the MMR system or other systems coupled to the MMR system. The MMR system architecture is advantageous because an action can be executed by pointing the capture device at a block of text, and the action is performed. Example actions include retrieving the text in electronic form to the capture device, retrieving the specification for the action, inserting data to a MMR document, transferring data between documents, purchasing items, authoring actions or reviewing historical information about actions. The MMR system includes a variety of user applications (one or more actions) initiated by the MMR recognition of a text patch such as information retrieval for a travel guide book, stock listings or advertisements; information capture such as recording content from a conference, recording and storing multimedia associated with the document, capturing information for a calendar and on the fly authoring; purchasing media files for storage on any part of an MMR document.

Claims

1. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a capture device for capturing a first electronic representation of a document in the first type of medium, wherein the first type of medium is a physical printed medium;

a media storage for a storing a plurality of mixed media documents, each mixed media document comprising;

an action;

a second electronic representation corresponding to the document in the physical printed medium; and

an index linking the action to the second electronic representation;

a recognition unit for identifying a mixed media document from the plurality of mixed media documents that has the second electronic representation that matches the first electronic representation; and

an action processing unit coupled to the recognition unit to receive an action of the identified mixed media document, and adapted to perform the action of the identified mixed media document.

2. The system of claim 1 wherein the first and second electronic representations are text patches.

3. The system of claim 1 wherein the action processing unit is coupled to the media storage and is adapted to retrieve data indexed by the identified mixed media document.

4. The system of claim 3 wherein the action of the identified mixed media document is retrieving information corresponding to a travel guide book, and wherein the action processing unit is adapted to retrieve a multimedia file from the media storage and provide it to the capture device for display.

5. The system of claim 3 wherein:

the action of the identified mixed media document is retrieving information about a stock, and

the action processing unit is adapted to retrieve information specific to a user about a stock using the identified mixed media document from the media storage and provide the information to the capture device for display; and

the media storage includes a shared database of up-to-date information about a plurality of stocks.

6. The system of claim 3 wherein:

the first electronic representation is of a printed advertisement;

the action of the identified mixed media document is retrieving information related to the printed advertisement;

the action processing unit is adapted to retrieve the information related to the printed advertisement using the identified mixed media document from the media storage and to provide the information to the capture device for display; and

the media storage includes the information related to the printed advertisement.

7. The system of claim 3 wherein:

the first electronic representation is of a portion of a first document having content related to a location on a page of a second document;

the action of the identified mixed media document is retrieving the page of the second document;

the action processing unit is adapted to retrieve the page of the second document using the identified mixed media document from a location indexed by a reference in the identified mixed media document, and adapted to provide the information to the capture device for display.

8. The system of claim 3 wherein:

the capture device includes a location detection mechanism for determining a physical location of the capture device, and the physical location of the capture device is sent to the action processing unit;

the action of the identified mixed media document is retrieving information based on the physical location of the capture device and identification by the recognition unit.

9. The system of claim 1 wherein the action processing unit is coupled to the media storage and is adapted to insert data into the identified mixed media document.

10. The system of claim 9 wherein:

the identified mixed media document is a conference guide;

the action of the identified mixed media document is inserting a file into the identified mixed media document; and

the action processing unit is adapted to update the identified mixed media document to include the file, and adapted to receive the from the capture device and store the file in the media storage.

11. The system of claim 9 wherein:

the action of the identified mixed media document is tagging multimedia; and

the action processing unit is adapted to receive a multimedia file obtained by the capture device, to select a text phrase from a document, to attach the selected text phrase to the multimedia file, and to insert the multimedia file and the selected text phrase into the identified mixed media document, the action processing unit coupled for communication with the capture device.

12. The system of claim 9 wherein:

the action of the identified mixed media document is transferring a document from the media storage to the capture device; and

the action processing unit is adapted to identify and transfer an electronic version of the identified mixed media document, the action processing unit coupled for communication with the capture device.

13. The system of claim 9 wherein:

the action of the identified mixed media document is selecting a date from a mixed media document and adding the date to a calendar manager application; and

the action processing unit is adapted to capture a date using the capture device, to use the captured date and the identified mixed media document add the date to a calendar manager application, the action processing unit adapted to interface with the calendar manager application, and coupled for communication with the capture device.

14. The system of claim 9 wherein:

the action of the identified mixed media document is capturing a phrase from a document and adding the phrase to a second document being drafted using an application program; and

the action processing unit is adapted to receive phrases from the capture device and adapted to input the phrases to the application program, the action processing coupled for communication with the capture device and a computing device running the application program.

15. The system of claim 1 wherein the action processing unit is coupled to the media storage and a server, and is adapted to purchase items using the identified mixed media document.

16. The system of claim 15 wherein:

the action of the identified mixed media document is using the identified mixed media document to specify a media file and purchasing the specified media file; and

the action processing unit is adapted to receive the identified mixed media document to specify the media file, to present to a user an option to purchase the specified media file, and to purchase the specified media file and transferring the specified media file to the system, the action processing unit coupled for communication with the capture device to present the option and receive input and coupled for communication with the server to purchase the specified media file and transfer the specified media file to the system.

17. The system of claim 15 wherein:

the action of the identified mixed media document is recognition to use the mixed media document to order merchandise and to automatically add an entry about an expected completion of that transaction into a user'"'"'s calendar; and

the action processing unit is adapted to recognize use of the mixed media to order merchandise, and add the entry about an expected completion of that transaction into the user'"'"'s calendar, the action processing unit interfacing with a calendar application, and a server for receiving order information.

18. The system of claim 15 wherein:

the action of the identified mixed media document is identity verification and a purchase transaction;

the recognition unit extracts personal information encoded in the first electronic representation; and

the action processing unit is adapted to receive the personal information from the recognition unit and use the personal information to verify a user'"'"'s identity, the action processing unit coupled to the server to perform verification and perform a transaction for purchasing an item.

19. The system of claim 15 wherein:

the action of the identified mixed media document is a payment; and

the action processing unit is adapted to identify a source for the payment and a recipient of the payment from identified mixed media document, the action processing unit adapted to communicate with the server to initiate, authorize, and confirm the payment.