Combining results of image retrieval processes

• US 8,856,108 B2
• Filed: 09/15/2008
• Issued: 10/07/2014
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Erol, Berna, Hull, Jonathan J., Moraleda, Jorge
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Pham, Tuan A
• Application Number:
  US12/210,540
• Publication Number:
  US 20090070110A1
• Time in Patent Office: 2,213 Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00244 with a server, e.g. an internet server fax-servers or the like for store and forward H04N1/324
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/00312 with a digital transmission apparatus, e.g. a switched network of teleprinters for the distribution of text-based information, SMS or ISDN device transmitting or receiving via a computer, e.g. using e-mail, a computer network or the internet H04N1/00214; transmission of digital information H04L
• CPC Class Codes: H04N 1/2179 Interfaces allowing access to a plurality of users, e.g. connection to electronic image libraries
• CPC Class Codes: H04N 1/2191 for simultaneous, independent access by a plurality of different users
• CPC Class Codes: H04N 2201/0039 Connection via a network

Abstract

A MMR system for newspaper publishing comprises a plurality of mobile devices, an MMR gateway, an MMR matching unit and an MMR publisher. The MMR matching unit receives an image query from the MMR gateway and sends it to one or more of the recognition units to identify a result including a document, the page and the location on the page. The MMR matching unit also includes a result combiner coupled to each of the recognition units to receive recognition results. The result combiner produces a list of most likely results and associated confidence scores. This list of results is sent by the result combiner back to the MMR gateway for presentation on the mobile device. The result combiner uses the quality predictor as an input in deciding which results are best.

Claims

1. A method for combining recognition results, the method comprising:

receiving, with a processor, a first recognition result that was generated using a first recognition algorithm, a first confidence score associated with the first recognition result, and first document information associated with the first recognition result;

receiving, with the processor, a second recognition result that was generated using a second recognition algorithm, a second confidence score associated with the second recognition result, and second document information associated with the second recognition result;

computing, with the processor, a first recognizability score corresponding to the first recognition algorithm using at least an input image and contextual information including one or more from a group of a date, a time, a location of a device that transmitted the input image, a personal profile and a retrieval history;

computing, with the processor, a second recognizability score corresponding to the second recognition algorithm using at least the input image and contextual information including one or more from a group of the date, the time, the location of the device that transmitted the input image, the personal profile and the retrieval history;

modifying the first confidence score and the second confidence score using the first recognizability score and the second recognizability score, respectively;

determining a number of votes for each of the first recognition result based in part on the first modified confidence score and the first document information, and the second recognition result based in part on the second modified confidence score and the second document information; and

determining a recognition output by selecting one of the first recognition result and the second recognition result with a highest number of votes.

2. The method of claim 1 further comprising using the first recognition result from the first recognition algorithm and the second recognition result from the second recognition algorithm to vote for a recognition result.

3. The method of claim 1 wherein the first recognition result and the second recognition result each are a matched document, a page, and a location on the page.

4. The method of claim 1 wherein the first recognizability score is also computed using the first recognition algorithm used to generate the first recognition result and a prediction algorithm.

5. The method of claim 1 wherein the second recognizability score is also computed using the input image, the second recognition algorithm used to generate the second recognition result, and a prediction algorithm.

6. The method of claim 1 wherein the recognition output is a recognition result that is a best match.

7. The method of claim 1 wherein the recognition output is an ordered list of recognition results produced by merging and sorting the recognition results by a confidence value.

8. The method of claim 1 wherein the document information is one from a group of document identification number;

document identification number and page identification number; and

document identification number, page identification number and an image patch location.

9. The method of claim 8 wherein the image patch location is an actual image patch location plus a proximity measure.

10. The method of claim 1 wherein selecting the first or second recognition result comprises:

determining whether there is a plurality of recognition results with the highest number of votes; and

selecting a highest confidence score from the plurality of recognition results.

11. The method of claim 10 wherein selecting the highest confidence score is one from a group of:

summing confidence values of the plurality of results and selecting the result corresponding to a highest confidence summation;

computing modified confidence scores, summing the modified confidence values and selecting the result corresponding to the highest confidence summation;

summing the recognizability score of the plurality of results and selecting the result corresponding to the highest recognizability score summation;

outputting the highest confidence score;

outputting the highest modified confidence score; and

outputting the highest recognizability score.

12. The method of claim 1 further comprising reconciling the recognition output with the first recognition result and the second recognition result by determining an exact location of an image patch by averaging mass centers of the first recognition result and the second recognition result.

13. A method for combining recognition results using predicted values, the method comprising:

computing, with a processor, a first recognizability score corresponding to a first recognition algorithm using at least an input image and contextual information including one or more from a group of a date, a time, a location of a device that transmitted the input image, a personal profile and a retrieval history;

computing, with the processor, a second recognizability score corresponding to a second recognition algorithm using the input image and contextual information including one or more from a group of the date, the time, the location of the device that transmitted the input image, the personal profile and the retrieval history;

determining whether the first recognizability score is above a threshold;

determining whether the second recognizability score is above a threshold; and

when the first recognizability score and the second recognizability score are both above a threshold;

performing recognition on the input image to generate a first recognition result, a first confidence score, and first document information associated with the first recognition result;

performing recognition on the input image to generate a second recognition result, a second confidence score, and second document information associated with the second recognition result;

modifying the first confidence score and the second confidence score using the first recognizability score and the second recognizability score, respectively;

determining votes for each of the first recognition result based in part on the first modified confidence score and the first document information, and the second recognition result based in part on the second modified confidence score and the second document information; and

determining a recognition output by selecting one of the first recognition result and the second recognition result with a highest number of votes.

14. The method of claim 13 wherein the document information is one from a group of document identification number;

document identification number and page identification number; and

document identification number, page identification number and an image patch location.

15. The method of claim 14 wherein the image patch location is an actual image patch location plus a proximity measure.

16. The method of claim 13, further comprising:

determining whether there is a plurality of recognition results with the highest number of votes; and

selecting a highest confidence score from the plurality of recognition results;

wherein selecting the highest confidence score is one from a group of;

summing confidence values of the plurality of recognition results and selecting the first or second recognition result corresponding to a highest confidence summation;

computing modified confidence scores and summing the modified confidence values and selecting the first or second recognition result corresponding to the highest confidence summation;

summing the first and second recognizability scores of the plurality of recognition results and selecting the first or second recognition result corresponding to a highest recognizability score summation;

outputting the highest confidence score;

outputting the highest modified confidence score; and

outputting the highest recognizability score.