Automatic adaption of an image recognition system to image capture devices

• US 8,510,283 B2
• Filed: 09/15/2008
• Issued: 08/13/2013
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Primary Examiner(s): Vy, Hung T
• Assistant Examiner(s): Pham, Tuan A
• Application Number:
  US12/210,519
• Publication Number:
  US 20090074300A1
• Time in Patent Office: 1,793 Days
• Field of Search: None
• US Class Current: 707/706
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A MMR system for newspaper publishing includes a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The image registration unit includes an indexing unit for generating images adapted to the environment and capabilities of the image capture device. The indexing unit also automatically adapts the configuration of the plurality of recognition units and index tables based upon image queries applied to the plurality of recognition and index tables. The plurality of recognition units and index tables are configured based on content they reference, recognition algorithm used or other factors.

Claims

1. A computer-implemented method for recognizing images, the method comprising:

receiving, by a computer, an image query;

determining, by the computer, a recognizability score based at least in part on the image query;

determining, by the computer, a first recognition algorithm based at least in part on the recognizability score;

extracting, by the computer, image features from the image query;

performing, by the computer, a first recognition by comparing the extracted image features to a first set of image data using the first recognition algorithm to identify a first set of recognition results that are substantially pristine images, each recognition result including at least a portion of a matching document and a location of the image query in the portion of the matching document;

performing, by the computer, a second recognition by comparing the extracted image features to a second set of image data to identify a second set of recognition results, the second set of image data being the first set of image data that are modified by a first level of blur;

performing, by the computer, a third recognition by comparing the extracted image features to a third set of image data to identify a third set of recognition results, the third set of image data being the first set of image data modified by a noise factor; and

combining, by the computer, the first set of recognition results from performing the first recognition, the second set of recognition results from performing the second recognition and the third set of recognition results from performing the third recognition by merging and sorting the first set of recognition results, the second set of recognition results and the third set of recognition results based on a confidence value of each recognition result.

2. The method of claim 1 wherein performing the second recognition is only performed if performing the first recognition does not return any recognition results.

3. The method of claim 1 further comprising providing a user that sent the image with information capable of being rendered on a mobile device based on the first set of recognition results, the second set of recognition results and the third set of recognition results.

4. The method of claim 1, wherein the recognizability score is determined using a prediction algorithm on the image query.

5. The method of claim 1 wherein performing the second recognition uses the first recognition algorithm, and further comprising:

performing the third recognition using a second recognition algorithm different than the first recognition algorithm to identify the third set of recognition results; and

combining the first set of recognition results, the second set of recognition results and the third set of recognition results.

6. A system for recognizing images, the system comprising:

a processor;

a dispatcher stored on a memory and executable by the processor, the dispatcher for receiving an image query, for determining a recognizability score based at least in part on the image query and for determining a first recognition algorithm based at least in part on the recognizability score;

a first recognition unit stored on the memory and executable by the processor, the first recognition unit coupled to the dispatcher, the first recognition unit for extracting image features from the image query, for performing a first recognition by comparing the extracted image features to a first set of image data using the first recognition algorithm to identify a first set of recognition results that are substantially pristine images, each recognition result including at least a portion of a matching document and a location of the image query in the portion of the matching document;

a second recognition unit stored on the memory and executable by the processor, the second recognition unit coupled to the dispatcher and the first recognition unit, the second recognition unit for performing a second recognition by comparing the extracted image features to a second set of image data to identify a second set of recognition results, the second set of image data being the first set of image data that are modified by a first level of blur;

a third recognition unit stored on the memory and executable by the processor, the third recognition unit coupled to the dispatcher and the second recognition unit, the third recognition unit for performing a third recognition by comparing the extracted image features to a third set of image data to identify a third set of recognition results, the third set of image data being the first set of image data modified by a noise factor; and

a result combiner coupled to the first recognition unit, the second recognition unit and the third recognition unit, the result combiner for combining the first set of recognition results from performing the first recognition, the second set of recognition results from performing the second recognition and the third set of recognition results from performing the third recognition by merging and sorting the first set of recognition results, the second set of recognition results and the third set of recognition results based on a confidence value of each recognition result.

7. The system of claim 6, wherein performing the second recognition is only performed if performing the first recognition does not return any recognition results.

8. The system of claim 6, wherein the result combiner provides a user that sent the image with information capable of being rendered on a mobile device based on the first set of recognition results, the second set of recognition results and the third set of recognition results.

9. The system of claim 6, wherein the recognizability score is determined using a prediction algorithm on the image query.

10. The system of claim 6, wherein performing the second recognition uses the first recognition algorithm, and the system further comprising:

the third recognition unit for performing the third recognition using a second recognition algorithm different than the first recognition algorithm to identify the third set of recognition results;

wherein the result combiner is further coupled to the third recognition unit and the result combiner is further configured for combining the first set of recognition results, the second set of recognition results and the third set of recognition results.

11. A computer program product comprising a non-transitory computer useable medium including a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

receive an image query;

determine a recognizability score based at least in part on the image query;

determine a first recognition algorithm based at least in part on the recognizability score;

extract image features from the image query;

perform a first recognition by comparing the extracted image features to a first set of image data using the first recognition algorithm to identify a first set of recognition results that are substantially pristine images, each recognition result including at least a portion of a matching document and a location of the image query in the portion of the matching document;

perform a second recognition by comparing the extracted image features to a second set of image data to identify a second set of recognition results, the second set of image data being the first set of image data that are modified by a first level of blur;

perform a third recognition by comparing the extracted image features to a third set of image data to identify a third set of recognition results, the third set of image data being the first set of image data modified by a noise factor; and

combine the first set of recognition results from performing the first recognition, the second set of recognition results from performing the second recognition and the third set of recognition results from performing the third recognition by merging and sorting the first set of recognition results, the second set of recognition results and the third set of recognition results based on a confidence value of each recognition result.

12. The computer program product of claim 11 wherein performing the second recognition is only performed if performing the first recognition does not return any recognition results.

13. The computer program product of claim 11 wherein the computer-readable program when executed on the computer further causes the computer to provide a user that sent the image with information capable of being rendered on a mobile device based on the first set of recognition results, the second set of recognition results and the third set of recognition results.

14. The computer program product of claim 11, wherein the recognizability score is determined using a prediction algorithm on the image query.

15. The computer program product of claim 11 wherein performing the second recognition uses the first recognition algorithm, and wherein the computer-readable program when executed on the computer further causes the computer to:

perform the third recognition using a second recognition algorithm different than the first recognition algorithm to identify the third set of recognition results; and

combine the first set of recognition results, the second set of recognition results and the third set of recognition results.