TRIGGERING APPLICATIONS BASED ON A CAPTURED TEXT IN A MIXED MEDIA ENVIRONMENT

• US 20070050360A1
• Filed: 07/31/2006
• Published: 03/01/2007
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Erol, Berna, Lee, Dar-Shyang, Hart, Peter, Hull, Jonathan
• Granted Patent: US 7,672,543 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/148 Segmentation of character regions
• CPC Class Codes: G06V 30/158 using character size, text spacings or pitch estimation
• CPC Class Codes: G06V 30/18086 by performing operations within image blocks or by using histograms

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an action processor and method, and MMR documents with an associated action. The MMR document structure is particularly advantageous because the ability to specify different actions for different MMR documents, combined with the ability to create any number of MMR documents for a particular location on any media, allows the MMR architecture to serve as a universal trigger or initiator for additional processing. In other words, addition processing or actions can be triggered or initiated based on MMR recognition. The action processor receives the output of the MMR recognition process which yields an MMR document including at least one action. The action processor executes that action which includes various commands to the MMR system or other systems coupled to the MMR system. The MMR system architecture is advantageous because an action can be executed by pointing the capture device at a block of text, and the action is performed. Example actions include retrieving the text in electronic form to the capture device, retrieving the specification for the action, inserting data to a MMR document, transferring data between documents, purchasing items, authoring actions or reviewing historical information about actions. The MMR system includes a variety of user applications (one or more actions) initiated by the MMR recognition of a text patch such as information retrieval for a travel guide book, stock listings or advertisements; information capture such as recording content from a conference, recording and storing multimedia associated with the document, capturing information for a calendar and on the fly authoring; purchasing media files for storage on any part of an MMR document.

Claims

1. A system for performing an action in a mixed media environment having a first type of medium and a second type of medium, the system comprising:

a capture device for capturing a first representation of a document in the first type of medium;

a media storage for a storing a plurality of mixed media documents, each mixed media document having an index, an action and a second representation of a corresponding document in the first type of medium;

a recognition unit for identifying a mixed media document from the plurality of mixed media documents that has a second representation that matches the first representation; and

an action processing unit coupled to the recognition unit to receive an action of the identified mixed media document, and adapted to perform the action of the identified mixed media document.

2. The system of claim 1 wherein the first and second representations are text patches.

3. The system of claim 1 wherein the action processing unit is coupled to the media storage and is adapted to retrieve data indexed by the identified mixed media document.

4. The system of claim 3 wherein the action of the identified mixed media document is retrieving information corresponding to a travel guide book, and wherein the action processing unit is adapted to retrieve a multimedia file from the media storage and provide it to the capture device for display.

5. The system of claim 3 wherein:

the action of the identified mixed media document is retrieving information about a stock, and

the action processing unit is adapted to retrieve information specific to a user about a stock using the identified mixed media document from the media storage provide it to the capture device for display; and

the media storage includes a shared database of up-to-date information about a plurality of stocks.

6. The system of claim 3 wherein:

the first representation is a printed advertisement;

the action of the identified mixed media document is retrieving information related to the printed advertisement;

the action processing unit is adapted to retrieve the information related to the printed advertisement using the identified mixed media document from the media storage and to provide the information to the capture device for display; and

the media storage includes information related to the printed advertisement.

7. The system of claim 3 wherein:

the first representation is of a portion of a first document having content related to a location on a page of a second document;

the action of the identified mixed media document is retrieving the page of the second document;

the action processing unit is adapted to retrieve the page of the second document using the identified mixed media document from a location indexed by a reference in the identified mixed media document, and adapted to provide the information to the capture device for display.

8. The system of claim 3 wherein:

the capture device includes a location detection mechanism for determining a physical location of the capture device, and the physical location of the capture device is sent to the action processing unit;

the action of the identified mixed media document is retrieving information based on the physical location of the capture device and identification by the recognition unit.

9. The system of claim 1 wherein the action processing unit is coupled to the media storage and is adapted to insert data into the identified mixed media document.

10. The system of claim 9 wherein:

the identified mixed media document that is a conference guide;

the action of the identified mixed media document is inserting a file into the identified mixed media document; and

the action processing unit is adapted to update the identified mixed media document to include the file, and adapted to receive the from the capture device and store the file in the media storage.

11. The system of claim 9 wherein:

the action of the identified mixed media document is tagging multimedia; and

the action processing unit is adapted to receive a multimedia file obtained by the capture device, to select a text phrase from a document, to attach the selected text phrase to the multimedia file, and to insert the multimedia file and the selected text phrase into the identified mixed media document, the action processing unit coupled for communication with the capture device.

12. The system of claim 9 wherein:

the action of the identified mixed media document is transferring a document from the media storage to the capture device; and

the action processing unit is adapted to identify and transfer an electronic version of the identified mixed media document, the action processing unit coupled for communication with the capture device.

13. The system of claim 9 wherein:

the action of the identified mixed media document is selecting a date from a mixed media document and adding the date to a calendar manager application; and

the action processing unit is adapted to capture a date using the capture device, to use the captured date and the identified mixed media document add the date to a calendar manager application, the action processing unit adapted to interface with the calendar manager application, and coupled for communication with the capture device.

14. The system of claim 9 wherein:

the action of the identified mixed media document is capturing a phrase from a document and adding the phrase to a second document being drafted using an application program; and

the action processing unit is adapted to receive phrases from the capture device and adapted to input the phrases to the application program, the action processing coupled for communication with the capture device and a computing device running the application program.

15. The system of claim 1 wherein the action processing unit is coupled to the media storage and a server, and is adapted to purchase items using the identified mixed media document.

16. The system of claim 15 wherein:

the action of the identified mixed media document is using the identified mixed media document to specify a media file and purchasing the specified media file; and

the action processing unit is adapted to receive the identified mixed media document to specify the media file, to present to a user an option to purchase the specified media file, and to purchase the specified media file and transferring the specified media file to the system, the action processing unit coupled for communication with the capture device to present the option and receive input and coupled for communication with the server to purchase the specified media file and transfer the specified media file to the system.

17. The system of claim 15 wherein:

the action of the identified mixed media document is recognition of use the mixed media document to order merchandise and automatically adding an entry about an expected completion of that transaction into a user'"'"'s calendar; and

the action processing unit is adapted to recognize use of the mixed media to order merchandise, and add the entry about an expected completion of that transaction into the user'"'"'s calendar, the action processing unit interfacing with a calendar application, and a server for receiving order information.

18. The system of claim 15 wherein:

the action of the identified mixed media document is identity verification and a purchase transaction;

the recognition unit extracts personal information encoded in the first representation; and

the action processing unit is adapted to receive the personal information from the recognition unit and use the personal information to verify a user'"'"'s identity, the action processing unit coupled to the server to perform verification and perform a transaction for purchasing an item.

19. The system of claim 15 wherein:

the action of the identified mixed media document is a payment; and

the action processing unit is adapted to identify a source for the payment and a recipient of the payment from identified mixed media document, the action processing unit adapted to communicate with the server to initiate, authorize, and confirm the payment.

21. The method of claim 1 wherein the first and second representations are text patches.

22. The method of claim 1 wherein performing the action is retrieving information corresponding to a travel guide book.

23. The method of claim 1 wherein performing the action is retrieving information about a stock specific to a user and accessing a shared database of up-to-date information about a plurality of stocks.

24. The method of claim 1 wherein performing the action is retrieving information related to the printed advertisement and sending the information to the capture device for display.

25. The method of claim 1 wherein performing the action is retrieving a second document and providing the second document to the capture device for display, and wherein a first document has content related to a location on a page of the second document and was recognized by the recognition unit.

26. The method of claim 1 wherein performing the action is retrieving information based on the physical location of the capture device and an identification by the recognition unit of the identified mixed media document.

27. The method of claim 1 wherein performing the action is inserting a file into the identified mixed media document that includes a conference guide.

28. The method of claim 1 wherein performing the action includes:

inserting a receiving a multimedia file obtained by the capture device;

selecting a text phrase from a document;

attaching the selected text phrase to the multimedia file; and

inserting the multimedia file and the selected text phrase into the identified mixed media document.

29. The method of claim 1 wherein performing the action is transferring an electronic version of the identified mixed media document from the media storage to the capture device.

30. The method of claim 1 wherein performing the action is receiving a date from a mixed media document from the captured device and adding the date to a calendar manager application.

31. The method of claim 1 wherein performing the action is capturing a phrase from a document and adding the phrase to a second document being drafted using an application program

32. The method of claim 1 wherein performing the action is purchasing items using data from the identified mixed media document.

33. The method of claim 1 wherein performing the action is using the identified mixed media document to specify a media file and purchasing the specified media file.

34. The method of claim 1 wherein performing the action is recognition of the mixed media document to order merchandise and automatically adding an entry about an expected completion of that transaction into a user'"'"'s calendar.

35. The method of claim 1 wherein:

comparing the input representation to a plurality of stored representations includes extracting personal information encoded in the first representation; and

performing the action is identity verification and a purchase transaction using the extracted personal information.

36. The method of claim 1 wherein performing the action is making an payment including identifying a source for the payment and a recipient of the payment from identified mixed media document, initiating a payment process, authorizing payment, and confirming the payment was make and received.

20. A method for performing an action in a mixed media environment having a first type of medium and a second type of medium, the method comprising:

acquiring an input representation;

comparing the input representation to a plurality of stored representations, each of the plurality of stored representations being associated with a mixed media document including an action;

identifying a matching stored representation that matches the input representation and the mixed media document associated with the matching stored representation; and

performing the action of the mixed media document associated with the matching stored representation.