COMBINING RESULTS OF IMAGE RETRIEVAL PROCESSES

• US 20090070110A1
• Filed: 09/15/2008
• Published: 03/12/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Granted Patent: US 8,856,108 B2
• Time in Patent Office: Days
• US Class Current: 704/236
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00312 with a digital transmission apparatus, e.g. a switched network of teleprinters for the distribution of text-based information, SMS or ISDN device transmitting or receiving via a computer, e.g. using e-mail, a computer network or the internet H04N1/00214; transmission of digital information H04L
• CPC Class Codes: H04N 1/2179 Interfaces allowing access to a plurality of users, e.g. connection to electronic image libraries
• CPC Class Codes: H04N 1/2191 for simultaneous, independent access by a plurality of different users
• CPC Class Codes: H04N 2201/0039 Connection via a network

Abstract

A MMR system for newspaper publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit also includes a result combiner coupled to each of the recognition units to receive recognition results. The result combiner produces a list of most likely results and associated confidence scores. This list of results is sent by the result combiner back to the MMR gateway for presentation on the mobile device. The result combiner uses the quality predictor as an input in deciding which results are best. The present invention also includes a number of novel methods including a method for generating the list of best results.

Claims

1. An method for combining recognition results, the method comprising:

receiving a first recognition result and a first confidence score associated with the first recognition result;

receiving a second recognition result and a second confidence score associated with the second recognition result;

computing a first recognizability score corresponding to the first recognition result;

computing a second recognizability score corresponding to the second recognition result;

determining a recognition output using the first confidence score, the second confidence score, the first recognizability score and the second recognizability score.

2. The method of claim 1 wherein the first recognition result was generated using a first recognition algorithm and the second recognition result was generated using a second recognition algorithm.

3. The method of claim 1 wherein the first recognition result and the second recognition result each are a matched document, a page, and a location on the page.

4. The method of claim 1 wherein the first recognizability score is computed using an input image, a recognition algorithm used to generate the first recognition result, and a predication algorithm.

5. The method of claim 1 wherein the second recognizability score is computed using an input image, a recognition algorithm used to generate the second recognition result, and a predication algorithm.

6. The method of claim 1 wherein the recognition output is a recognition result that is a best match.

7. The method of claim 1 wherein the recognition output is an ordered list of recognition results produced by merging and sorting the recognition results by confidence value.

8. The method of claim 1 further comprising:

modifying the first confidence score using the first recognizability score;

modifying the second confidence score using the second recognizability score; and

wherein the modified first confidence score and the modified second confidence score are used to the determining the recognition output.

9. The method of claim 1 further comprising:

determining location information of the first recognition result;

determining location information of the second recognition result;

using the location information of the first recognition result to vote for a recognition result; and

wherein determining the recognition output includes selecting a confidence score for the recognition result with the most votes.

10. The method of claim 1 wherein the location information is one from the group of document identification number;

document identification number and page identification number; and

document identification number. page identification number and image patch location.

11. The method of claim 1 wherein the image patch location is an actual image patch location plus a proximity measure.

12. The method of claim 9 selecting the confidence score comprises:

determining whether there is a plurality of recognition results with the most votes; and

selecting a highest confidence score from the confidence scores of the plurality of recognition results.

13. The method of claim 12 wherein selecting the highest confidence score is one from the group of:

summing confidence values of the plurality of results and selecting the result corresponding to the highest confidence summation;

computing modified confidence scores and summing the modified confidence values and selecting the result corresponding to the highest confidence summation;

summing the recognizability score of the plurality of results selecting the result corresponding to the highest recognizability score summation;

outputting the highest confidence score;

outputting the highest modified confidence score; and

outputting the highest recognizability score.

14. The method of claim 1 further comprising reconciling the recognition output with the first recognition result and the second recognition result by determining an exact location of an image patch by averaging mass centers of the first recognition result and the second recognition result.

19. The method of claim 1 wherein the image patch location is an actual image patch location plus a proximity measure.

15. An method for combining recognition results using predicted values, the method comprising:

computing a first recognizability score corresponding to a first recognition algorithm;

computing a second recognizability score corresponding to a second recognition algorithm;

determining whether the first recognizability score is above a threshold;

determining whether the second recognizability score is above a threshold; and

if the first recognizability score and the second recognizability score are both above a threshold

performing recognition on an input image to generate a first recognition result and a first confidence score,

performing recognition on the input image to generate a second recognition result and a second confidence score; and

determining a recognition output using the first confidence score, the second confidence score.

16. The method of claim 15 further com

modifying the first confidence score using the first recognizability score;

modifying the second confidence score using the second recognizability score; and

wherein the modified first confidence score and the modified second confidence score are used to the determining the recognition output.

17. The method of claim 16 further comprising:

determining location information of the first recognition result;

determining location information of the second recognition result;

using the location information of the first recognition result to vote for a recognition result; and

wherein determining the recognition output includes selecting a confidence score for the recognition result with the most votes.

18. The method of claim 17 wherein the location information is one from the group of document identification number;

document identification number and page identification number; and

document identification number. page identification number and image patch location.

20. The method of claim 16 wherein selecting the highest confidence score is one from the group of:

summing confidence values of the plurality of results and selecting the result corresponding to the highest confidence summation;

computing modified confidence scores and summing the modified confidence values and selecting the result corresponding to the highest confidence summation;

summing the recognizability score of the plurality of results selecting the result corresponding to the highest recognizability score summation;

outputting the highest confidence score;

outputting the highest modified confidence score; and

outputting the highest recognizability score.