System And Methods For Creation And Use Of A Mixed Media Environment

• US 20100166309A1
• Filed: 03/08/2010
• Published: 07/01/2010
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Graham, Jamey, Piersol, Kurt, Hart, Peter E., Hull, Jonathan J., Erol, Berna, Lee, Dar-Shyang
• Granted Patent: US 9,063,953 B2
• Time in Patent Office: Days
• US Class Current: 382/176
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an MMR user, a MMR computer, a user printer that produces a printed document, a networked media server, an office portal, a service provider server, an electronic display that is electrically connected to a set-top box, a document scanner, a network, a capture device, a cellular infrastructure, wireless fidelity (Wi-Fi) technology, Bluetooth® technology, infrared (IR) technology, wired technology, and a geo location mechanism. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system for creating a mixed media reality document data structure, the system comprising:

a capture device that captures an image patch of a printed document;

a mixed media processor communicatively coupled to the capture device to receive the image patch and to compare the captured image patch to a previously generated mixed media reality document data structure to relate the printed document with the mixed media reality document data structure, the mixed media reality document data structure comprising;

a representation of the printed document comprising a matchable portion for use in pattern matching to a corresponding first location within the captured image patch;

an electronic representation of the printed document; and

an index correlating the matchable portion of the representation with a second location in the electronic document; and

a media storage communicatively coupled to the mixed media processor that stores the mixed media reality document data structure.

2. The system of claim 1, wherein the mixed media processor is adapted to relate the image patch to the mixed media reality document data structure and perform an action in response to a relationship between the image patch and the mixed media reality document data structure.

3. The system of claim 2, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

4. The system of claim 1, wherein the media storage stores a plurality of mixed media reality document data structures each having a representation of a printed document and an index, the media storage coupled to the mixed media processor.

5. The system of claim 1, wherein the image patch is a portion of the printed document.

6. The system of claim 1, wherein the image patch comprises a text finger print of the printed document having at least two lines that is usable for pattern matching.

7. The system of claim 1, wherein the image patch uniquely identifies a location in the printed document.

8. The system of claim 7, wherein the image patch comprises a text finger print of the printed document having a portion of a first line and a portion of an adjacent second line.

9. The system of claim 2, wherein the action comprises text to be presented or one or more commands to be executed, and wherein the index is a link between the text to be presented or one or more commands to be executed and the electronic representation of the printed document.

10. A computer-implemented method for creating a mixed media reality document data structure for interacting with printed documents and electronic documents, the method comprising:

creating the mixed media reality document data structure, comprising;

a representation of a printed document comprising a matchable portion for use in pattern matching to a corresponding first location within a captured image patch of the printed document, wherein the image patch comprises a portion less than an entire page of the document;

an electronic representation of the printed document;

and an index correlating the matchable portion of the first representation with a second location in the electronic document;

wherein creating the mixed media reality document data structure further comprises;

generating the representation of the printed document;

indexing the generated representation of the printed document;

capturing the image patch from the printed document;

comparing the image patch to the generated representation; and

performing an action if the image patch is similar to the generated representation.

11. The method of claim 10, wherein the generating the representation of the printed document comprises automatically capturing a text layout of the printed document.

12. The method of claim 11, wherein the automatically capturing is done by one from a group of a printed document capture module, an operating system, a print driver, a scanner, a file transfer mechanism.

13. The method of claim 10, wherein the generated representation is one from the group of a representation of the entire printed document document, a representation of a patch of text, a representation of a single word, a representation of a section of an image, and any other representation of a matchable portion of the printed document.

14. The method of claim 10, wherein indexing the generated representation includes association of one from the group of the index, an electronic representation of the document, a second media type, and an action, with the generated representation.

15. The method of claim 10, wherein indexing the generated representation includes one from the group of specifying x-y locations of characters of the printed document, specifying x-y locations of words of the printed document, specifying x-y locations of an image or a portion of an image in a printed document, performing an OCR operation and specifying x-y locations of characters and/or words, performing feature extraction and specifying x-y locations of characters and/or words.

16. The method of claim 10, wherein capturing the image patch comprises one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies the document, receiving email with the document attached, receiving the document by file transfer, scanning a portion of an image, and inputting the document.

17. The method of claim 10, wherein comparing the image patch to the generated representation further comprises performing pattern matching between the image patch and the generated representation.

18. The method of claim 17, wherein the pattern matching is performed by comparing a document fingerprint of the image patch to a document fingerprint of the generated first representation.

19. The method of claim 18, wherein comparing the document fingerprint includes extracting features from a captured image, composing descriptors from those features, and looking up a document and patch that contain a percentage of those descriptors.

20. The method of claim 10, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.