Multi-Classifier Selection and Monitoring for MMR-based Image Recognition

• US 20090076996A1
• Filed: 10/07/2008
• Published: 03/19/2009
• Est. Priority Date: 07/31/2006
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Hull, Jonathan J., Erol, Berna, Moraleda, Jorge
• Granted Patent: US 8,073,263 B2
• Time in Patent Office: Days
• US Class Current: 706/46
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text

Abstract

A MMR system that uses multiple classifiers for predicting, monitoring, and adjusting index tables for image recognition comprises a plurality of mobile devices, a pre-processing server or MMR gateway, and an MMR matching unit, and may include an MMR publisher. The MMR matching unit includes a plurality of recognition unit and index table pairs corresponding to classifiers to be applied to received image queries, as well as an image registration unit for storing and monitoring performance data for the classifiers. The MMR matching unit receives the image query and identifies, using a classifier set, a result including a document, the page, and the location on the page corresponding to the image query. The present invention also includes methods for monitoring online performance of a multiple classifier image recognition system, for classifier selection and comparison, and for offline classifier prediction.

Claims

1. A method of classifier set prediction, comprising:

dividing a future time interval into a plurality of minimum subintervals; and

for a selected minimum subinterval;

retrieving data for one or more historic time intervals corresponding to the selected minimum subinterval; and

determining a best performing classifier set for the one or more historic time intervals corresponding to the selected minimum subinterval.

2. The method of claim 1, wherein determining the best performing classifier set for the one or more historic time intervals comprises:

for each of the one or more historic time intervals, applying a complete set of classifiers to a set of image queries received during the historic time interval; and

determining the best performing classifier set for the historic time interval.

3. The method of claim 1, wherein the one or more historic time intervals are a plurality of historic time intervals, and determining the best performing classifier set for the one or more historic time intervals further comprises comparing the best performing classifier sets for the plurality of historic time intervals.

4. The method of claim 2, wherein a current classifier set is in use corresponding to an immediately previous minimum subinterval, further comprising calculating a potential performance gain for changing from the current classifier set to the best performing classifier set.

5. The method of claim 4, wherein calculating the potential performance gain further comprises:

determining performance data from application of the current classifier set to the set of image queries received during the historic time interval;

determining performance data from application of the best performing classifier set to the set of image queries received during the historic time interval; and

calculating a difference the between best set performance data and the current classifier set performance data.

6. The method of claim 5, further comprising, in response to the potential performance gain exceeding a switching threshold, changing from the current classifier set to the best performing classifier set for the minimum subinterval.

7. The method of claim 6, further comprising, in response to a determination that additional subintervals exist, performing the method for the additional subintervals.

8. There method of claim 7, wherein the method is performed chronologically on the additional subintervals.

9. The method of claim 1, wherein the method is performed offline.

10. A method of monitoring performance of a multiple classifier image recognition system, comprising:

recording a recognition result for each of a plurality of image queries submit to an initial classifier set during a time interval, each recognition result comprising a time of arrival and an applied classifier identification for each of the plurality of image queries;

for each classifier in the initial classifier set, monitoring a decision percentage based on the applied classifier identifications.

11. The method of claim 10, further comprising:

calculating a difference between the monitored decision percentage and an expected decision percentage; and

in response to the difference exceeding a performance threshold, triggering a performance alert.

12. The method of claim 11, further comprising, in response to a number of performance alerts exceeding an alert threshold for a predetermined subinterval of the time interval, calculating a potential performance gain for the predetermined subinterval associated with changing from the initial classifier set to a best performing classifier set.

13. The method of claim 12, wherein calculating the potential performance gain further comprises:

determining performance data from application of the initial classifier set to a set of image queries received during the time interval;

determining the best performing classifier set for the time interval, comprising;

applying a complete set of classifiers to the set of image queries received during the time interval;

selecting the best performing classifier set for the time interval;

determining performance data from application of the best performing classifier set to the set of image queries received during the historic time interval; and

calculating a difference the between best classifier set performance data and the current classifier set performance data.

14. The method of claim 13, further comprising, in response to the potential performance gain exceeding a switching threshold, changing from the initial classifier set to the best performing classifier set for the minimum subinterval.

15. The method of claim 10, wherein the method is performed online.

16. A system for monitoring performance of a multiple classifiers, comprising:

a results recorder for recording a recognition result for each of a plurality of image queries submit to an initial classifier set during a time interval, each recognition result comprising a time of arrival and an applied classifier identification for each of the plurality of image queries;

a decision monitor for, for each classifier in the initial classifier set, monitoring a decision percentage based on the applied classifier identifications.

17. The system of claim 16, wherein the decision monitor is further configured for calculating a difference between the monitored decision percentage and an expected decision percentage, further comprising:

an alert monitor for, in response to the difference exceeding a performance threshold, triggering a performance alert.

18. The system of claim 17, further comprising:

a classifier switching calculator for, in response to a number of performance alerts exceeding an alert threshold for a predetermined subinterval of the time interval, calculating a potential performance gain for the predetermined subinterval associated with changing from the initial classifier set to a best performing classifier set.

19. The system of claim 18, wherein the classifier switching calculator is further configured for:

determining performance data from application of the initial classifier set to a set of image queries received during the time interval;

determining the best performing classifier set for the time interval, comprising;

applying a complete set of classifiers to the set of image queries received during the time interval;

selecting the best performing classifier set for the time interval;

determining performance data from application of the best performing classifier set to the set of image queries received during the historic time interval; and

calculating a difference the between best classifier set performance data and the current classifier set performance data.

20. The system of claim 16, wherein the monitoring is performed online.