Computation of a recongnizability score (quality predictor) for image retrieval

• US 8,868,555 B2
• Filed: 09/15/2008
• Issued: 10/21/2014
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Erol, Berna, Antunez, Emilio R., Huet, Landry, Hull, Jonathan J., Moraleda, Jorge
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): BROMELL, ALEXANDRIA Y
• Application Number:
  US12/210,532
• Publication Number:
  US 20090067726A1
• Time in Patent Office: 2,227 Days
• Field of Search: 707/915, 707/736, 345/16, 382/225
• US Class Current: 707/736
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: Y10S 707/915 Image

Abstract

A MMR system for newspaper publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR system also includes a quality predictor as a plug-in installed on the mobile device to filter images before they are included as part of a retrieval request or as part of the MMR matching unit. The quality predictor comprises an input for receiving recognition algorithm information, a vector calculator, a score generator and a scoring module. The quality predictor receives as inputs an image query, context information and device parameters, and generates an outputs a recognizability score. The present invention also includes a method for generating robustness features.

Claims

1. An apparatus for generating a recognizability score, the apparatus comprising:

a vector calculator for determining a plurality of quality feature vectors from an input image to measure distortion of the input image including blurriness and coding artifacts and to determine whether the input image is stable to the distortion by applying different levels of the distortion to the input image, measuring distances between the input image and the distorted images and determining whether a combination of the distances is small, the vector calculator adapted for communication to receive the input image;

a score generator for generating a recognition score for each of the quality feature vectors, the score generator adapted to receive the quality feature vectors from the vector calculator; and

a scoring module for determining recognition results, generating the recognizability score from the recognition scores, generating a confidence score for each recognition result that reflects a confidence in the recognition result based on the recognizability score and merging and sorting the recognition results to produce one or more top results using the confidence scores, the scoring module adapted to communicate with the score generator to receive the recognition scores.

2. The apparatus of claim 1 wherein the vector calculator includes one from the group of a blur feature module, a content amount feature module, a luminosity feature module, a bleed through feature module, a coding artifacts feature module, a perspective distortion feature module, a camera noise feature module, a text quality feature module, an object detection feature module, a recognition algorithm feature module and a robustness feature module.

3. The apparatus of claim 1 wherein the vector calculator and the score generator are adapted to receive a parameter of a recognition algorithm.

4. The apparatus of claim 1 wherein the recognizability score is a weighted sum of recognition scores.

5. The apparatus of claim 1 wherein the vector calculator includes a robustness feature module for generating a robustness vector, the robustness feature module receiving the input image, applying the different levels of the distortion to the input image, extracting recognition features from the input image and the distorted images, computing a first set of distances between feature locations, computing a second distance between feature vectors and generating the robustness vector from the second distance between the feature vectors.

6. A computer-implemented method for generating a first recognizability score, the method comprising:

receiving a first image;

computing, with one or more processors, a plurality of quality feature vectors to measure distortion of the first image including blurriness and coding artifacts and to determine whether the input image is stable to the distortion by applying different levels of the distortion to the input image, measuring distances between the input image and the distorted images and determining whether a combination of the distances is small;

generating, with the one or more processors, a recognition score for each of the quality feature vectors;

determining recognition results;

generating the first recognizability score from the recognition scores;

generating a confidence score for each recognition result that reflects a confidence in the recognition result based on the first recognizability score; and

merging and sorting the recognition results to produce one or more top results using the confidence scores.

7. The method of claim 6, wherein the plurality of quality feature vectors are the group of a blur feature vector, a content amount feature vector, a luminosity feature vector, a bleed through feature vector, a coding artifacts feature vector, a perspective distortion feature vector, a camera noise feature vector, a text quality feature vector, an object detection feature vector, a recognition algorithm feature vector and a robustness feature vector.

8. The method of claim 6, wherein generating the first recognizability score uses one from the group of neural networks, naive based classifiers, Bayesian based classifiers and support vector machine (SVM) based classifiers.

9. The method of claim 6, further comprising using the first recognizability score in determining how to evaluate a recognition result.

10. The method of claim 6, further comprising:

comparing the first recognizability score to a threshold;

if the first recognizability score is below the threshold, presenting a message to a user requesting a confirmation signal that the first image should be transmitted for further processing;

receiving the confirmation signal; and

transmitting the first image responsive to the confirmation signal.

11. The method of claim 6, further comprising:

determining whether the first recognizability score for the first image is above a threshold;

if the first recognizability score for the first image is above the threshold, transmitting the first image; and

if the first recognizability score is below the threshold,

presenting a message to a user requesting a second image be received;

receiving the second image;

classifying and computing a second recognizability score for the second image;

determining whether the second recognizability score for the second image is above the threshold; and

if the second recognizability score for the second image is above the threshold, transmitting the second image.

12. The method of claim 6, further comprising:

determining whether the first recognizability score for the first image is above a threshold;

if the first recognizability score for the first image is above the threshold, extracting a first set of recognition features from the first image and transmitting the first set of recognition features; and

if the first recognizability score is below the threshold,

presenting a message to a user requesting a second image be received;

receiving the second image;

classifying and computing a second recognizability score for the second image;

determining whether the second recognizability score for the second image is above the threshold; and

if the second recognizability score for the second image is above the threshold, extracting a second set of recognition features from the second image and transmitting the second set of recognition features.

13. The method of claim 6, further comprising:

determining whether the first recognizability score for the first image is above a threshold; and

if the first recognizability score is below the threshold, presenting a message to a user including the first recognizability score for the first image.

14. The method of claim 6, wherein generating the first recognizability score further comprises performing classifying and computing for a plurality of recognition algorithms to generate a plurality of recognizability scores.

15. The method of claim 6, wherein generating the first recognizability score further comprises adjusting classification parameters for a recognizing unit.

16. The method of claim 6, wherein generating the first recognizability score is performed on the first image prior to indexing and for a plurality of recognition algorithms and further comprising:

determining a first recognition algorithm with a best recognizability score;

selecting the first recognition algorithm for indexing; and

indexing the first image using the first recognition algorithm.

17. The method of claim 16, further comprising:

determining a second recognition algorithm with a second best recognizability score;

selecting the second recognition algorithm for indexing; and

indexing the first image using the second recognition algorithm.

18. The method of claim 6, wherein the first recognizability score is a weighted sum of recognizability scores for a plurality of recognition algorithms.

19. The method of claim 6, further comprising receiving context information, metadata and device parameters for generating the first recognizability score.

20. The method of claim 6, further comprising using the first recognizability score in determining whether a quality of the first image is sufficient to be recognized.