Authoring Tools Using A Mixed Media Environment

• US 20070047781A1
• Filed: 07/31/2006
• Published: 03/01/2007
• Est. Priority Date: 08/23/2005
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Erol, Berna, Hart, Peter, Hull, Jonathan
• Granted Patent: US 7,639,387 B2
• Time in Patent Office: Days
• US Class Current: 382/124
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides a method to associate related documents. The method acquires two associated documents and associates the related portions together. When the method acquires a portion of one of the document, it identifies the portion acquired, and provides the related portion of the other document.

Claims

1. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

generating a representation of an image of a first media type and an action of a second media type;

indexing the generated representation of the image to a mixed media document,

the generated representation being associated with the action;

acquiring an input representation;

comparing the input representation to the generated representation; and

performing the action if the input representation is similar to the generated representation.

2. The method of claim 1, wherein the generating the representation of the image of the first media type comprises automatically capturing an image layout of a printed document.

3. The method of claim 2, wherein the automatically capturing is done by one from a group of a printed document capture module, an operating system, a print driver, a scanner, a file transfer mechanism.

4. The method of claim 1, wherein indexing the generated representation comprises association of the action with the generated representation.

5. The method of claim 1, wherein acquiring an input representation comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies a document, receiving email with a document attached, receiving a document by file transfer, scanning a portion of an image, and inputting a document.

6. The method of claim 1, wherein comparing the input representation to the generated representation further comprises performing pattern matching between the input representation and the generated representation.

7. The method of claim 6, wherein the pattern matching is performed by comparing the document fingerprint of the input representation to the document fingerprint of the generated representation.

8. The method of claim 7, wherein comparing the document fingerprint comprises extracting features from a captured image, composing descriptors from those features, and looking up a document and patch that contain a percentage of those descriptors.

9. The method of claim 1, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.

10. The method of claim 1, further comprising:

determining that the image is unique.

11. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

generating a representation of a document of a first media type;

generating data of a second media type;

indexing the generated representation of the document and the data to a mixed media document;

acquiring an input representation;

comparing the input representation to the generated representation; and

performing an action specified by the mixed media document if the input representation is similar to the generated representation.

12. The method of claim 11, wherein the generating the data of the second media type comprises retrieving contents from the Internet.

13. The method of claim 11, wherein acquiring an input representation comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies a document, receiving email with a document attached, receiving a document by file transfer, scanning a portion of an image, and inputting a document.

14. The method of claim 11, wherein comparing the input representation to the generated representation further comprises performing pattern matching between the input representation and the generated representation.

15. The method of claim 11, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.

16. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

generating a first representation of a first document of a first media type;

indexing the generated first representation to a first mixed media document;

acquiring an input representation;

comparing the input representation to the generated first representation; and

performing an action specified by the first mixed media document if the input representation is similar to the generated first representation.

17. The method of claim 16, wherein generating the first representation further comprises:

generating a second representation of a second document of the first media type, the second document being relevant to the first document, wherein indexing the generated first representation further comprises indexing the generated second representation to a second mixed media document, the indexing associating a portion of the first representation with a portion of the second representation.

18. The method of claim 17, wherein the indexing also associating the portion of the second representation with the portion of the first representation.

19. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

creating a first document;

generating a representation of a second document of a first media type;

indexing the generated representation to a mixed media document;

acquiring a input representation;

modifying the first document based on the input representation and the second document.

20. The method of claim 19, further comprising:

parsing the second document for entity names.

21. The method of claim 20, wherein modifying the first document further comprising:

labeling the input representation using the entity names.

22. A method for providing a mixed media environment for interacting with documents of a first media type and a second media type, the method comprising:

generating a representation of a document of a first media type;

indexing the generated representation to a mixed media document;

acquiring a input representation;

comparing the input representation to the generated representation; and

creating a second document based on the comparison and the first document.