AUTOMATIC ADAPTION OF AN IMAGE RECOGNITION SYSTEM TO IMAGE CAPTURE DEVICES

• US 20090074300A1
• Filed: 09/15/2008
• Published: 03/19/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Granted Patent: US 8,510,283 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A MMR system for newspaper publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The image registration unit includes an indexing unit for generating images adapted to the environment and capabilities of the image capture device. The indexing unit also automatically adapts the configuration of the plurality of recognition units and index tables based upon image queries applied to the plurality of recognition and index tables. The plurality of recognition units and index tables are configured based on content they reference, recognition algorithm used or other factors. The present invention also includes a number of novel methods including a method for generating blurry images for device training; a method for generating classifiers; a method for selecting a pass classifier; and a method for modifying a hypothesizing and testing classifier on unrecognized images.

Claims

1. An image recognition system adapted for image capture device characteristics, the system comprising:

a first index table for storing a first set of image data and results, the first index table providing results in response to queries based on image data;

a first recognition unit having an input and outputs, for receiving an image query and for recognizing and retrieving information using the image query;

the input of the first recognition unit coupled to receive an image query and an output coupled to the first index table for receiving results, the first recognition unit including a feature extractor for receiving the image query and producing an image feature and a retriever for receiving the image feature and generating a query of the first index table that includes the image feature, the retriever coupled to the feature extractor and the first index table;

a second index table for storing a second set of image data and results, the second index table providing results in response to queries based on the second set of image data, and wherein the second set of image data is the first set of image data modified by a first noise factor;

a second recognition unit having an input and outputs, for receiving the image feature from the feature extractor and coupled to the second index table, the second recognition unit for recognizing and retrieving information using the image feature;

the input of the second recognition unit coupled to the first recognition unit and receive the image query; and

a result combiner having a first input a second input and an output for receiving recognition results and producing a list of matching results the first input of the result combiner coupled to the output of the first recognition unit and the second input of the result combiner coupled to the output of the second recognition unit.

2. The system of claim 1 further comprising

a third recognition unit having an input and outputs, for receiving the image query and for recognizing and retrieving information using the image query;

the input of the third recognition unit coupled to receive the image query; and

a third index table for storing a third set of image data and results, the third index table providing results in response to queries based on image data, the third index table coupled to the third recognition unit, and wherein the third set of image data is the first set of image modified by a second noise factor; and

wherein the result combiner has a third input and the output of the third recognition unit is coupled the third input of the result combiner.

3. The system of claim 1, wherein the noise factor is one from the group of Gaussian blur, x-motion blur, y-motion blur, tilt, distance and rotation.

4. An image recognition system comprising:

a feature extractor for receiving an image query and producing an image feature, the feature extractor coupled to receive the image query; and

a first retriever for receiving the image feature and generating a query of a first index table and for receiving results from the first index table, the retriever coupled to the feature extractor and the first index table;

a second retriever for receiving the image feature, for generating a query of a second index table and for receiving results from the second index table if the first retriever does not recognize and retrieve matching information from the first index table, the second retriever coupled to the feature extractor, the second index table and the first retriever; and

a result combiner for receiving recognition results and producing a list of matching results the result combiner coupled to receive results from the first retriever and the second retriever.

5. The system of claim 4, comprising a third retriever for receiving the image feature, for generating a query of a third index table and for receiving results from the third index table if the second retriever does not recognize and retrieve matching information from the second index table, the third retriever coupled to the third index table and the second retriever.

6. An image registration system adapted to image capture device characteristics, the system comprising:

a generator for receiving image data and page identification numbers, and producing a first set of image data unmodified by a first noise factor, and a second set of image data being the received image data modified by the first noise factor, the generator coupled to receive the image data, the page identification numbers and the noise factor;

a feature extractor for receiving image data and generating at least one image feature, the feature extractor coupled to the generator to receive the first set of image data and the second set of image data;

a first index table for storing image features and page identification numbers for the first set of data;

an second index table for storing image features and page identification numbers for the second set of data; and

an updater for formatting extracted image features and storing them in the first index table and the second index table, the updater coupled to the feature extractor for receiving the image features generated from the first set of image data and storing them in the first index table and receiving image features generated from the second set of image data and storing them in the second index table, the updater adapted for communication with the feature extractor, the first index table and the second index table.

7. The system of claim 6 wherein the generator produces a third set of image data being the received image data modified by a second noise factor, the feature extractor is coupled to the generator to receive the third set of image data, the updater receiving the image features generated from the third set of image data and storing them in the third index table and adapted for communication with a third index table, and the system comprises the third index table for storing image features and page identification numbers for the third set of data.

8. The system of claim 6 further comprising a second feature extractor for receiving image data and generating at least one image feature, the second feature extractor coupled to the generator to the second set of image data, the second feature extractor coupled to the generator and the updater.

9. The system of claim 6 wherein the updater comprises a first updater and a second updater, the first updater for formatting extracted image features and storing them in the first index table and the second updater for formatting extracted image features and storing them in the second index table, the first updater adapted for communication with the feature extractor and the first index table and the second updater adapted for communication with the feature extractor and the second index table.

10. The system of claim 6 wherein the noise factor is one from the group of Gaussian blur, x-motion blur, y-motion blur, tilt, distance and rotation.

12. The method of claim 10 wherein performing the second recognition is only performed if performing the first recognition does not return any recognition results.

13. The method of claim 10 further comprising combining recognition results from performing the first recognition and performing the second recognition.

14. The method of claim 10 further comprising performing a third recognition by comparing the extracted image features to a third set of image data, the third set of image data is the first set of image data modified by a second noise factor.

15. The method of claim 10 wherein performing the first recognition and performing the second recognition utilize a first recognition algorithm, and further comprising

performing a third recognition using a second recognition algorithm different than the first algorithm;

and

combining results of performing the first recognition, performing the second recognition and performing the third recognition.

11. A method for recognizing image, the method comprising:

receiving an image query;

extracting image features from the image query;

performing a first recognition by comparing the extracted image features to a first set of image data; and

performing a second recognition by comparing the extracted image features to a second set of image data, the second set of image data is the first set of image data modified by a first noise factor.

16. A method for registering images, the method comprising:

creating a set of training data;

creating a set of classifiers;

identifying a classifier that recognizes a greatest number of images;

modifying the classifier to increase a number of images the classifier can recognize;

validating the modified classifier against the training data; and

updating an index table with the modified classifier.

17. The method of claim 16 wherein creating the set of training data comprises:

determining an object set, environmental characteristics and device placement characteristics;

capturing images of the object set for one or more environmental characteristics and one or more device placement characteristics;

tagging the captured images with their environmental characteristics and device placement characteristics; and

storing the tagged images in a data set.

18. The method of claim 16 wherein creating the set of classifiers comprises:

defining a range and a sampling interval for a noise factor;

generating a classifier at each point in a multidimensional space;

applying the classifiers to the training data; and

recording performance of the classifiers on the training data.

19. The method of claim 16 wherein identifying the classifiers that recognizes a greatest number of images comprises:

finding classifiers that recognizes the most new images from a training set of images,

adding the classifier to a solution set;

removing the new images from the training set of images; and

repeating the finding, adding and removing until a predefined number of N classifiers are added to the solution set.

20. The method of claim 16 wherein modifying the classifier to increase a number of images the classifier can recognize comprises:

retrieving an unrecognized image;

finding a closest pair of recognized images;

generating a new classifier using the closest pair; and

adding the new classifier to a solution set.