System and methods for creation and use of a mixed media environment

• US 9,063,953 B2
• Filed: 03/08/2010
• Issued: 06/23/2015
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Graham, Jamey, Hart, Peter E., Lee, Dar-Shyang, Piersol, Kurt
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): BROMELL, ALEXANDRIA Y
• Application Number:
  US12/719,437
• Publication Number:
  US 20100166309A1
• Time in Patent Office: 1,933 Days
• Field of Search: 707/707, 707/728, 707/736, 707/793, 707/915, 707/743, 382/201
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system includes an MMR user, a MMR computer, a user printer that produces a printed document, a networked media server, an office portal, a service provider server, an electronic display that is electrically connected to a set-top box, a document scanner, a network, a capture device, a cellular infrastructure, wireless fidelity (Wi-Fi) technology, Bluetooth® technology, infrared (IR) technology, wired technology, and a geo location mechanism. The MMR system of the present invention provides mechanisms for forming a mixed media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system of the present invention facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system comprising:

one or more mixed media processors configured to;

receive an image patch of at least a first portion of a media; and

compare features of the first image patch to a previously generated mixed media document data structure to associate the media with the mixed media document data structure;

a mixed media document data structure comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch, and the matchable portion including a geometric relationship between a first portion of a first line of text and a second portion of a second line of text, wherein the geometric relationship is determined based on at least one from the group of specifying x-y locations of characters of the media, specifying x-y locations of words of the media, specifying x-y locations of an image or a portion of an image in the media, performing an optical character recognition operation and specifying x-y locations of at least one of characters and words, and

performing feature extraction and specifying x-y locations of at least one of features, characters, and words; and

an index correlating the matchable portion of the representation with an identifier for information associated with the media; and

a media storage communicatively coupled to the one or more mixed media processors that stores the mixed media document data structure.

2. The system of claim 1, wherein the one or more mixed media processors are adapted to relate the received image patch to the mixed media document data structure and perform an action in response to a relationship between the received image patch and the mixed media document data structure.

3. The system of claim 2, wherein the action is one from the group of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

4. The system of claim 2, wherein the action comprises text to be presented or one or more commands to be executed, and wherein the index is a link between the text to be presented or one or more commands to be executed and the electronic document.

5. The system of claim 1, wherein the media storage stores a plurality of mixed media document data structures each having a corresponding representation of a media and a corresponding index.

6. The system of claim 1, wherein the received image patch depicts a portion of the media.

7. The system of claim 1, wherein the representation comprises a text fingerprint of the media having at least two lines that are usable for pattern matching.

8. The system of claim 1, wherein the received image patch uniquely identifies a location in the media.

9. The system of claim 1, wherein the representation comprises a text fingerprint of the media having the first portion of the first line and the second portion of the second line, wherein the second line is adjacent to the first line.

10. A computer-implemented method for creating a mixed media document data structure, the method comprising:

receiving a media;

receiving an image patch of at least a first portion of the media;

comparing features of the received image patch to a previously generated mixed media document data structure to determine a match; and

responsive to the comparing resulting no matches, creating a mixed media document data structure, comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch, the matchable portion including a geometric relationship between a first portion of a first line of text and a second portion of a second line of text, wherein the geometric relationship is determined based on at least one from the group of specifying x-y locations of characters of the media, specifying x-y locations of words of the media, specifying x-y locations of an image or a portion of an image in the media, performing an optical character recognition operation and specifying x-y locations of characters and/or words, and

performing feature extraction and specifying x-y locations of features, characters, and/or words; and

an index correlating the matchable portion of the representation with an identifier for information associated with the media.

11. The method of claim 10, further comprising generating the representation of the media by automatically capturing a text layout of the media.

12. The method of claim 11, wherein the automatically capturing is done by one from a group of a media capture module, an operating system, a print driver, a scanner, and a file transfer mechanism.

13. The method of claim 10, wherein the representation is one from the group of a representation of the entire media, a representation of a patch of text, a representation of a single word, a representation of a section of an image, and a representation of any matchable portion of the media.

14. The method of claim 10, further comprising indexing the representation, wherein indexing the representation includes association of one from the group of the index, the electronic document and a second media type with the representation.

15. The method of claim 10, further comprising representing the x-y locations of words as a set of horizontally and vertically adjacent word-pairs.

16. The method of claim 10, wherein the received image patch is received by one from the group of scanning a patch of text, scanning a patch of text and preprocessing to determine the likelihood that an intended feature description is extracted correctly, scanning a machine-readable identifier, inputting data that identifies the media, receiving email with the media attached, receiving the media by file transfer, scanning a portion of an image, and inputting the media.

17. The method of claim 10, wherein comparing the received image patch to the previously generated mixed media document data structure further comprises performing pattern matching between the received image patch and a representation of a media in the previously generated mixed media document data structure.

18. The method of claim 17, wherein the pattern matching is performed by comparing a document fingerprint of the received image patch to a document fingerprint of the representation of the media in the previously generated mixed media document data structure.

19. The method of claim 18, wherein comparing the document fingerprint includes extracting features from the received image patch, composing descriptors from those features, and looking up a media and a patch that contains a percentage of those descriptors.

20. The method of claim 10, further comprising performing an action if features of the received image patch are similar to a representation of a media in the previously generated mixed media document data structure, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, and input of a command to another system.

21. A computer program product comprising a machine-readable medium including a non-transitory computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a media;

receive an image patch of at least a first portion of the media;

compare features the received image patch to a previously generated mixed media document data structure; and

responsive to the comparing resulting no matches, create a mixed media document data structure, comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch, and the matchable portion including a geometric relationship between a first portion of a first line of text and a second portion of a second line of text, wherein the geometric relationship is determined based on one at least one from the group of specifying x-y locations of characters of the media, specifying x-y locations of words of the media, specifying x-y locations of an image or a portion of an image in the media, performing an optical character recognition operation and specifying x-y locations of characters and/or words, and

performing feature extraction and specifying x-y locations of characters, features, and/or words; and

an index correlating the matchable portion of the representation with an identifier for information associated with the media.