Method and system for image matching in a mixed media environment

• US 8,600,989 B2
• Filed: 07/31/2006
• Issued: 12/03/2013
• Est. Priority Date: 10/01/2004
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Hart, Peter E., Lee, Dar-Shyang, Piersol, Kurt
• Primary Examiner(s): Wilson, Kimberly
• Assistant Examiner(s): Liu, Hexing
• Application Number:
  US11/461,279
• Publication Number:
  US 20060262352A1
• Time in Patent Office: 2,682 Days
• Field of Search: 707/708, 707/726
• US Class Current: 707/736
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/142 using hand-held instruments; Constructional details of the instruments
• CPC Class Codes: G06V 30/268 Lexical context
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/418 Document matching, e.g. of document images

Abstract

A Mixed Media Reality (MMR) system and associated techniques are disclosed. The MMR system provides mechanisms for forming a mixed media document that includes media of at least two types (e.g., printed paper as a first medium and digital content and/or web link as a second medium). In one particular embodiment, the MMR system provides for image matching portions of a document.

Claims

1. A computer-implemented method of image matching, the method comprising:

receiving a first image of at least part of a first media type;

distinguishing between a first text patch and an image patch in the first image;

determining a quality of the first text patch;

determining that the quality of the first text patch is unreliably processable for extracting the plurality of features from the first text patch;

generating instructions for modifying a focus of an optics device based on the quality of the first text patch;

receiving a second image of at least part of the first media type;

determining the quality of a second text patch in the second image;

validating the quality of the second text patch as being reliably processable for extracting a plurality of features from the second text patch;

extracting from the second text patch the plurality of features including a vertical grouping of a first word boundary with one or more second word boundaries present above or below overlapping the first word boundary in the second text patch;

extracting from the second text patch a geometric relationship between the plurality of features, the geometric relationship including one or more angles between the first word boundary and the one or more second word boundaries in the plurality of features;

generating a query for identifying a document corresponding to the second text patch using the plurality of features and the geometric relationship;

applying the query against a collection of documents in a database; and

in response to positively applying the query, returning an identification comprising a matching document and a location of the second text patch within the portion of the matching document.

2. The computer-implemented method of claim 1, wherein the location of the second text patch within the portion of the matching document is specified by x and y coordinates.

3. The computer-implemented method of claim 1, further comprising:

returning a confidence value associated with the identification.

4. The computer-implemented method of claim 1, further comprising:

associating the identification with a second media type.

5. The computer-implemented method of claim 4, further comprising:

providing the second media type to a capture device.

6. The computer-implemented method of claim 4, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

7. The computer-implemented method of claim 1, wherein the first media type is a paper document.

8. The computer-implemented method of claim 1, further comprising modifying the quality of the second image to generate a modified image;

and wherein extracting is performed on the modified image.

9. A system for image matching, comprising:

one or more processors;

a memory;

a quality assessment module stored on the memory and executable by the one or more processors, the quality assessment module operable to receive a first image of at least part of a first media type, to distinguish between a first text patch and an image patch in the first image, to determine a quality of the first text patch, to determine that the quality of the first text patch is unreliably processable for extracting a plurality of features from the first text patch, generate instructions for modifying a focus of an optics device based on the quality of the first text patch, receive a second image of at least part of the first media type, determine the quality of a second text patch in the second image and validate the quality of the second text patch as being reliably processable for extracting a plurality of features from the second text patch;

a feature extraction module stored on the memory and executable by the one or more processors, the feature extraction module coupled to the quality assessment module and operable to extract from the second text patch the plurality of features including a vertical grouping of a first word boundary with one or more second word boundaries present above or below overlapping the first word boundary in the second text patch and a geometric relationship between the plurality of features, wherein the geometric relationship includes at least one or more angles between the first word boundary and the one or more second boundaries in the plurality of features;

a classification module stored on the memory and executable by the one or more processors, the classification module coupled to the feature extraction module and operable to receive the plurality of features and the geometric relationship, to generate and send a query for identifying a document corresponding to the second text patch, and to receive an identification comprising a matching document and a location of the second text patch within the portion of the matching document; and

a database operable to store a collection of documents, to receive the query from the classification module, to apply the query to the collection of documents, and to create and return the identification to the classification module.

10. The system of claim 9, wherein the image processing module is operable to modify the quality of the image for image recognition by the classification module.

11. The system of claim 9, wherein a capture device comprises the feature extraction module.

12. The system of claim 9, further comprising:

wherein the quality assessment module is operable to assess content of the first image, and wherein the assessment is dependent on at least one of a need and a capability of the system.

13. The system of claim 12, wherein the assessment is dependent on whether the content of the first image contains text.

14. The system of claim 12, wherein the assessment is dependent on a sharpness of text in the first image.

15. The system of claim 12, wherein the assessment is dependent on whether the first image contains a non-document image.

16. The system of claim 12, wherein the assessment is dependent on a functionality of the feature extraction module.

17. The system of claim 12, wherein an operation of a capture device is adjustable dependent on the assessment.

18. The system of claim 9, further comprising:

a position tracking module operable to track movement of a capture device over the first media type.

19. The system of claim 9, wherein the first media type is a paper document.

20. The system of claim 9, further comprising:

a control structure operable to control information and command flow through the system.

21. The system of claim 9, wherein the identification is associated with a second media type.

22. The system of claim 21, wherein the second media type is provided to a capture device.

23. The system of claim 21, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

24. The system of claim 22, wherein the capture device comprises one selected from the group consisting of a cellular camera phone, a Personal Digital Assistant (PDA) device, a digital camera, a barcode reader, a radio frequency identification (RFID) reader, a computer peripheral, a web camera, and a video card.

25. A computer-implemented method of associating a first media type with a second media type, the method comprising:

receiving a first image of at least part of the first media type;

distinguishing between a first text patch and an image patch in the first image;

determining a quality of the first text patch;

determining that the quality of the text patch is unreliably processable for extracting the plurality of features from the first text patch;

generating instructions for modifying a focus of an optics device associated with a capture device based on the quality of the first text patch;

receiving a second image of at least part of the first media type;

determining the quality of a second text patch in the second image;

validating the quality of the second text patch as being reliably processable for extracting a plurality of features from the text patch;

extracting from the second text patch the plurality of features including a vertical grouping of a first word boundary with one or more second word boundaries present above or below overlapping the first word boundary in the second text patch;

extracting from the second text patch at least one geometric relationship between the plurality of features, the geometric relationship including one or more angles between the first word boundary and the one or more second word boundaries in the plurality of features;

converting the plurality of features and the at least one geometric relationship into a symbolic representation in response to the validation;

generating a query for identifying a document corresponding to the second text patch using the symbolic representation;

applying the query against a collection of documents in a database;

in response to positively matching the query, returning an identification comprising a matching document and a location of the second text patch within the portion of the matching document; and

providing the identification to the capture device, the identification comprising the second media type.

26. The computer-implemented method of claim 25, wherein the first media type is a paper document.

27. The computer-implemented method of claim 25, wherein the second media type comprises at least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.

28. The computer-implemented method of claim 25, further comprising:

associating a confidence value with the identification.

29. The computer-implemented method of claim 25, further comprising:

performing an action dependent on the identification, wherein the action comprises at least one selected from the group consisting of retrieving information, placing an order, retrieving a video, retrieving a sound, storing information, creating a new document, printing a document or image, displaying a document or image, searching information, and presenting information.

30. The computer-implemented method of claim 25, further comprising:

maintaining a collection of documents comprising the at least one document.

31. A computer program product comprising a non-transitory computer readable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive a first image, the first image being of at least part of a first media type;

distinguish between a first text patch and an image patch in the image;

determine a quality of the first text patch;

determine that the quality of the first text patch is unreliably processable for extracting the plurality of features from the first text patch;

generate instructions for modifying a focus of an optics device based on the quality of the first text patch;

receive a second image of at least part of the first media type;

determine the quality of a second text patch in the second image;

validate the quality of the second text patch as being reliably processable for extracting a plurality of features from the second text patch;

extract from the second text patch the plurality of features including a vertical grouping of a first word boundary with one or more second word boundaries present above or below overlapping the first word boundary in the second text parch;

extract from the second text patch at least one geometric relationship between the plurality of features, the at least one geometric relationship including one or more angles between the first word boundary and the one or more second word boundaries in the plurality of features;

generate a query for identifying a document corresponding to the second text patch using the plurality of features and the geometric relationship;

apply the query against a collection of documents stored in a database; and

in response to positively applying the query, return an identification comprising a matching document and a location of the second text patch within the portion of the matching document.

32. The computer-readable medium of claim 31, further comprising instructions to:

assess a quality of the first image; and

send a command to a capture device dependent on the quality assessment.

33. The computer-readable medium of claim 31, further comprising instructions to:

communicate a second media type to a capture device in response to determining the location where the second text patch is located.

34. The computer-readable medium of claim 31, wherein the first media type is a paper document.

35. The computer-readable medium of claim 33, wherein the second media type comprises least one selected from the group consisting of a data structure, a command, text, audio, video, an image, a digital photograph, web link text, an application file, updated information, and services.