System and methods for creation and use of a mixed media environment

• US 10,073,859 B2
• Filed: 06/22/2015
• Issued: 09/11/2018
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Graham, Jamey, Hart, Peter E., Lee, Dar-Shyang, Piersol, Kurt
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Bromell, Alexandria
• Application Number:
  US14/746,663
• Publication Number:
  US 20150379043A1
• Time in Patent Office: 1,177 Days
• Field of Search: 707743, 707915, 382201
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 40/12 Use of codes for handling textual entities
• CPC Class Codes: G06F 40/123 Storage facilities
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). The MMR system provides mechanisms for forming a media document that includes media of at least two types, such as printed paper as a first medium and a digital photograph, digital movie, digital audio file, or web link as a second medium. Furthermore, the MMR system facilitates business methods that take advantage of the combination of a portable electronic device, such as a cellular camera phone, and a paper document.

Claims

1. A system comprising:

one or more media processors configured to;

receive an image patch of at least a first portion of a first media;

convert the image patch into an object pair to capture a two-dimensional geometric relationship between objects in the image patch, the two-dimensional geometric relationship specifying one or more of a direction, an angle, and a distance between the objects; and

compare the object pair to previously generated object pairs in a media data structure to associate the first media with the media data structure, the media data structure comprising;

a representation of the first media comprising a matchable object pair corresponding to a first location within the first media, the matchable object pair for matching to the object pair;

a second media; and

an index correlating the matchable object pair of the representation with the second media, the index recording a document, a page, and a second location within the page at which the object pair appears; and

a media storage communicatively coupled to the one or more media processors that stores the media data structure.

2. The system of claim 1, wherein the one or more media processors are configured to perform an action in response to an association between the first media and the media data structure.

3. The system of claim 2, wherein the action is one from the group of writing information to a location, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, completing a purchase of a product, sending a command to another system, retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

4. The system of claim 2, wherein the action is performed using the second media.

5. The system of claim 1, wherein the matchable object pair includes the two-dimensional geometric relationship between objects in the first media, wherein the two-dimensional geometric relationship is determined based on at least one from the group of:

specifying x-y locations of characters of the first media, specifying x-y locations of words of the first media, specifying x-y locations of an image or a portion of an image in the first media, performing an optical character recognition operation and specifying x-y locations of at least one of characters and words, and performing feature extraction and specifying x-y locations of at least one of features, characters, and words.

6. The system of claim 1, wherein the representation comprises a text fingerprint of the first media having at least two lines that are usable for pattern matching.

7. The system of claim 1, wherein the image patch uniquely identifies a third location in the first media.

8. A computer-implemented method for creating a media data structure, the method comprising:

receiving a first media;

receiving an image patch of at least a first portion of the first media;

converting the image patch into an object pair to capture a two-dimensional geometric relationship between objects in the image patch, the two-dimensional geometric relationship specifying one or more of a direction, an angle, and a distance between the objects;

comparing the object pair to previously generated object pairs in the media data structure to determine a match; and

responsive to the comparing resulting no matches, creating the media data structure, comprising;

a representation of the first media comprising a matchable object pair corresponding to a first location within the first media, the matchable object pair for matching to the object pair;

a second media; and

an index correlating the matchable object pair of the representation with the second media, the index recording a document, a page, and a second location within the page at which the object pair appears.

9. The method of claim 8, further comprising

performing an action in response to an association between the first media and the media data structure.

10. The method of claim 9, wherein the action is performed using the second media.

11. The method of claim 8, wherein the representation is one from the group of a representation of the entire first media, a representation of a patch of text, a representation of a single word, a representation of a section of an image, and a representation of any matchable object pair of the first media.

12. The method of claim 8, wherein the matchable object pair includes the two-dimensional geometric relationship between objects in the first media, wherein the two-dimensional geometric relationship is determined based on at least one from the group of:

specifying x-y locations of characters of the first media, specifying x-y locations of words of the first media, specifying x-y locations of an image or a portion of an image in the first media, performing an optical character recognition operation and specifying x-y locations of at least one of characters and words, and performing feature extraction and specifying x-y locations of at least one of features, characters, and words.

13. The method of claim 8, further comprising performing pattern matching between the image patch and the representation of the first media in the media data structure.

14. The method of claim 13, wherein the pattern matching is performed by comparing a document fingerprint of the image patch to a document fingerprint of the representation of the first media in the media data structure.

15. The method of claim 14, wherein comparing the document fingerprint includes extracting features from the image patch, composing descriptors from those features, and identifying the representation of the first media when the document fingerprint of the representation of the first media contains a percentage of the descriptors.

16. The method of claim 8, further comprising performing an action if the object pair of the image patch is similar to the previously generated object pairs in the media data structure, wherein the action is one from the group of retrieving information, writing information to a location, displaying information on a client device, conducting an interactive dialog, deferring the action for later execution, modifying a document, completing a purchase of a product, sending a command to another system, retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, or presenting information.

17. A computer program product comprising a machine-readable medium including a non-transitory computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a first media;

receive an image patch of at least a first portion of the first media;

convert the image patch into an object pair to capture a two-dimensional geometric relationship between objects in the image patch, the two-dimensional geometric relationship specifying one or more of a direction, an angle, and a distance between the objects;

compare the object pair to previously generated object pairs in a media data structure; and

responsive to the comparing resulting no matches, create a media data structure, comprising;

a representation of the first media comprising a matchable object pair corresponding to a first location within the first media, the matchable object pair for matching to the object pair;

a second media; and

an index correlating the matchable object pair of the representation with the second media, the index recording a document, a page, and a second location within the page at which the object pair appears.

18. The computer program product of claim 17, wherein the computer readable program causes the computer to perform an action in response to an association between the first media and the media data structure.

19. The computer program product of claim 18, wherein the action is performed using the second media.