System and Methods for Creation and Use of a Mixed Media Environment

• US 20150379043A1
• Filed: 06/22/2015
• Published: 12/31/2015
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Graham, Jamey, Hart, Peter E., Lee, Dar-Shyang, Piersol, Kurt
• Granted Patent: US 10,073,859 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). The MMR system provides mechanisms for forming a media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system comprising:

one or more media processors configured to;

receive an image patch of at least a first portion of a media; and

compare features of the first image patch to a previously generated media data structure to associate the media with the media data structure, the media data structure comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch;

a second media; and

an index correlating the matchable portion of the representation with the second media; and

a media storage communicatively coupled to the one or more media processors that stores the media data structure.

2. The system of claim 1, wherein the one or more mixed media processors are adapted to relate the received image patch to the media data structure and perform an action in response to a relationship between the received image patch and the media data structure.

3. The system of claim 2, wherein the action is one from the group of writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, input of a command to another system, retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

4. The system of claim 2, wherein the action is performed using the second media.

5. The system of claim 1, wherein the matchable portion includes a geometric relationship between a first portion of a first line of text and a second portion of a second line of text, wherein the geometric relationship is determined based on at least one from the group of specifying x-y locations of characters of the media, specifying x-y locations of words of the media, specifying x-y locations of an image or a portion of an image in the media, performing an optical character recognition operation and specifying x-y locations of at least one of characters and words, and performing feature extraction and specifying x-y locations of at least one of features, characters, and words.

6. The system of claim 1, wherein the representation comprises a text fingerprint of the media having at least two lines that are usable for pattern matching.

7. The system of claim 1, wherein the received image patch uniquely identifies a location in the media.

8. A computer-implemented method for creating a media data structure, the method comprising:

receiving a media;

receiving an image patch of at least a first portion of the media;

comparing features of the received image patch to a previously generated media data structure to determine a match; and

responsive to the comparing resulting no matches, creating the media data structure, comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch;

a second media; and

an index correlating the matchable portion of the representation with the second media.

9. The method of claim 8, further comprising:

relating the received image patch to the media data structure; and

performing an action in response to a relationship between the received image patch and the media data structure.

10. The method of claim 9, wherein the action is performed using the second media.

11. The method of claim 8, wherein the representation is one from the group of a representation of the entire media, a representation of a patch of text, a representation of a single word, a representation of a section of an image, and a representation of any matchable portion of the media.

12. The method of claim 8, wherein the matchable portion includes a geometric relationship between a first portion of a first line of text and a second portion of a second line of text, wherein the geometric relationship is determined based on at least one from the group of specifying x-y locations of characters of the media, specifying x-y locations of words of the media, specifying x-y locations of an image or a portion of an image in the media, performing an optical character recognition operation and specifying x-y locations of at least one of characters and words, and performing feature extraction and specifying x-y locations of at least one of features, characters, and words.

13. The method of claim 8, wherein comparing the received image patch to the previously generated media data structure further comprises performing pattern matching between the received image patch and a representation of a media in the previously generated media data structure.

14. The method of claim 13, wherein the pattern matching is performed by comparing a document fingerprint of the received image patch to a document fingerprint of the representation of the media in the previously generated media document data structure.

15. The method of claim 14, wherein comparing the document fingerprint includes extracting features from the received image patch, composing descriptors from those features, and looking up a media and a patch that contains a percentage of those descriptors.

16. The method of claim 8, further comprising performing an action if features of the received image patch are similar to a representation of a media in the previously generated media data structure, wherein the action is one from the group of retrieving information, writing information to a location, performing a search for information, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, purchase of a product, input of a command to another system, retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

17. A computer program product comprising a machine-readable medium including a non-transitory computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a media;

receive an image patch of at least a first portion of the media;

compare features the received image patch to a previously generated media data structure; and

responsive to the comparing resulting no matches, create a media data structure, comprising;

a representation of the media comprising a matchable portion corresponding to a first location within the media, the matchable portion for matching to the received image patch;

a second media; and

an index correlating the matchable portion of the representation with the second media.

18. The computer program product of claim 17, wherein to compare the received image patch to the previously generated media data structure, the computer readable program causes the computer to perform pattern matching between the received image patch and a representation of a media in the previously generated media data structure.

19. The computer program product of claim 17, wherein the computer readable program causes the computer to relate the received image patch to the media data structure and perform an action in response to a relationship between the received image patch and the media data structure.

20. The computer program product of claim 19, wherein the action is performed using the second media.