ROBUST PATTERN RECOGNITION SYSTEM AND METHOD USING SOCRATIC AGENTS

• US 20110299765A1
• Filed: 08/11/2011
• Published: 12/08/2011
• Est. Priority Date: 09/13/2006
• Status: Active Grant

• Current Assignee: Aurilab LLC
• Original Assignee: Aurilab LLC
• Inventors: BAKER, JAMES K.
• Granted Patent: US 8,180,147 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques

Abstract

A computer-implemented pattern recognition method, system and program product, the method comprising in one embodiment: creating electronically a linkage between a plurality of models within a classifier module within a pattern recognition system such that any one of said plurality of models may be selected as an active model in a recognition process; creating electronically a null hypothesis between at least one model of said plurality of linked models and at least a second model among said plurality of linked models; accumulating electronically evidence to accept or reject said null hypothesis until sufficient evidence is accumulated to reject said null hypothesis in favor of one of said plurality of linked models or until a stopping criterion is met; and transmitting at least a portion of the electronically accumulated evidence or a summary thereof to accept or reject said null hypothesis to a pattern classifier module.

Claims

1. A computer-implemented method of pattern recognition comprising:

obtaining classification results of a set of at least one electronic lower-level classifier modules performing pattern classification on particular input data;

using a higher-level classifier module that performs pattern classification on a pattern recognition problem different from the set of lower-level classifier modules, wherein said higher-level classifier module performs at least one of the following operations;

controlling, using one or more computers, training of the set of lower-level classifier modules based at least in part on the pattern classification task performed by the higher-level classifier module;

combining, using the one or more computers, the results of the set of lower-level classifier modules based at least in part on combining rules that vary based on the particular input data and based at least in part on the classification task performed by the higher-level classifier module, where the set of lower-level classifier modules consists of a plurality of lower-level classifier modules;

selecting, using the one or more computers, an active subset of the set of lower-level classifier modules based at least in part on a pattern classification task performed by the higher-level classifier module.

2. A pattern recognition method as in claim 1, wherein the higher-level classifier module controls the training of the set of lower-level classifier modules based at least in part on data that is not available to any one of the lower-level classifier modules.

3. A pattern recognition method as in claim 1, wherein each of the at least one lower-level classifiers is structurally restricted to a proper subset of the classification capabilities of the higher-level classifier module.

4. A pattern recognition method as in claim 3, wherein the higher-level classifier module performs classification based at least in part on the classification results of a set of sub-classifiers and each lower-level classifier module performs classification based on at most a proper subset of the set of sub-classifiers.

5. A pattern recognition method as in claim 4, wherein each lower-level classifier module performs classification based on only one of the sub-classifiers.

6. A pattern recognition method as in claim 4, wherein the higher-level classifier module controls the training of the set of lower-level classifiers at least in part by selecting for each item in the training data a subset of the set of lower-level classifiers to which that particular training item will be presented for training of the lower-level classifiers in that subset.

7. A pattern recognition method as in claim 3, wherein for each of the at least one lower-level classifier modules at least one of the following restrictions is true:

the lower-level classifier module is restricted to use substantially less computation than the higher-level classifier module;

the lower-level classifier module is restricted to use substantially less memory than the higher-level classifier module;

the lower-level classifier module is restricted to use a proper subset of the classification techniques available to the higher-level classifier module;

there is at least one knowledge source available to the higher-level classifier that is not available to the lower-level classifier;

the lower-level classifier module is based on parametric models using a proper subset of the model parameter values available to the higher-level classifier module.

10. A pattern recognition method as in claim 1, wherein the higher-level classifier module combines the results of the plurality of lower-level classifier modules based at least in part on combining rules that vary based on the particular input data where, for each lower-level classifier the higher-level classifier uses at least some data not available to that lower-level classifier.

11. A pattern recognition method as in claim 1, wherein the higher-level classifier module selects an active subset of the lower-level modules during pattern recognition based at least in part on a pattern classification task that estimates reliability of the classification results obtained by the lower-level classifier modules wherein the higher-level classifier module estimates the reliability of the classification results of the plurality of lower-level classifier modules at least in part based on data not available to any one lower-level classifier module.

12. A pattern recognition method as in claim 1, wherein for a given training sample the higher-level classifier module selects an active subset of the plurality of lower-level classifier modules to be trained on the given training sample based at least in part on the higher-level module performing a pattern classification to estimate which of the plurality of lower-level classifiers will most improve a specified performance measure by being trained on the given training sample.

13. A pattern recognition method as in claim 2, wherein the higher-level classifier module controls the training of the set of lower-level classifier modules based at least in part on the diversity of the recognition results of at least two of the lower-level classifier modules.

8. A pattern recognition method as in 7, wherein the higher-level classifier module controls the training of the set of lower level modules at least in part by presenting as training data to at least one of the lower-classifiers data that has been automatically labeled by the higher-level classifier.

9. A pattern recognition training method as in 7, wherein the higher-level classifier module controls the training of the set of lower level modules at least in part by presenting as practice data to at least one of the lower-classifiers data that has been automatically labeled by the higher-level classifier.

14. A computer-implemented method of sharing knowledge among a plurality of pattern classifiers, comprising:

obtaining a plurality of classifier modules including a first classifier module;

obtaining a communicable model that is either a new model or a model that has been modified by a knowledge acquisition process in the first classifier module;

transmitting, using one or more computers, said communicable model to at least one second classifier module in the plurality of classifier modules;

creating, using the one or more computers, a pair of model sets for said second classifier module in which one member of the pair of model sets is an unmodified model set for the second classifier module and one member of the pair of model sets is a modified model set that includes the communicable model;

testing, using the one or more computers, comparative performance of the pair of model sets in said at least one second classifier module; and

making, using the one or more computers, the modified model set active in the at least one second classifier module if the modified model set performs better in said second classifier module.

15. A method of sharing knowledge among a plurality of pattern classifiers as in claim 14, further comprising:

transmitting to the first classifier module information obtained from said testing of the pair of model sets in said at least one second classifier module.

16. A method of sharing knowledge among a plurality of pattern classifiers as in claim 14, further comprising:

deciding whether to transmit said communicable model to additional classifiers based at least in part on the information obtained from said testing of the of the pair of model sets in said at least one second classifier module;

if the decision is to transmit said communicable model to additional classifiers, then transmitting said communicable model to at least one third classifier module in the plurality of classifier modules, where the communicable model has not yet been transmitted to said at least one third classifier module;

creating a pair of model sets for said at least one third classifier module in which one member of the pair of model sets is an unmodified model set for the third classifier module and one member of the pair of model sets is a modified model set that includes the communicable model;

testing comparative performance of the pair of model sets in said at least one third classifier module; and

making the modified model set electronically active in said at least one third classifier module if the modified model set performs better in said third classifier module.

17. A method of sharing knowledge among a plurality of pattern classifiers as in claim 14, further comprising:

a. creating software associated with a model or set of models to be transmitted from a first classifier module to a second classifier module allowing said model or set of models to be utilized in the context of said second classifier; and

b. transmitting to said second classifier module a module comprising the model or set of models to be transmitted and the associated software.

18. A method of sharing knowledge among a plurality of pattern classifiers as in claim 14, wherein the plurality of classifiers are distributed among a plurality of recognition systems, further comprising:

a. creating at least one new recognition system that has different subsets of classifier modules from among the plurality of classifier modules to thereby obtain an expanded set of recognition systems;

b. measuring comparative performance of the expanded set of recognition systems;

c. deleting at least one recognition system from the set of recognition systems based on the measurement of comparative performance.

19. A method of sharing knowledge among a plurality of pattern classifiers as in claim 18, wherein the plurality of classifiers are distributed among a plurality of recognition systems, further comprising simulating a process of natural selection on the population of recognition systems by repeating the processes of creating and deleting recognition systems based on comparative performance.

20. A computer-implemented multi-stage pattern recognition method, comprising:

obtaining a sample of data to be recognized;

obtaining a plurality of labels for the given sample from a set of one or more recognition systems;

creating, using one or more computers, a set of linked model sets for at least one of the one or more recognition systems based on training said at least one recognition system on the sample of data wherein each model in the set of linked models is created by training on the given sample with a training label comprising a particular one of the plurality of labels obtained for the given sample;

obtaining a set of practice data;

testing, using the one or more computers, comparative performance of the linked model sets on the practice data;

correcting, using the one or more computers, the label on the given data sample to agree with the label associated with model from the linked set of models that performs best in the comparative performance testing on the practice data; and

returning, using the one or more computers, a corrected the label as corrected as the final recognition result of the multi-stage recognition process.

21. A computer-implemented pattern recognition method as in claim 20, further comprising

obtaining a plurality of streams of data to be recognized;

obtaining a plurality of labels for at least one given sample of data from the plurality of streams from the recognition results of at least one recognition system;

creating, using the one or more computers, at least one set of linked models for the at least one recognition system based on training said at least one recognition system on the at least one given sample of data wherein each model in the set of linked models is created by training on the given sample with a training label that comprises a particular one of the plurality of labels obtained for the given sample;

obtaining practice data from the plurality of streams of data;

performing, using the one or more computers, comparative performance testing of the linked model sets on the plurality of streams of data to be recognized;

accumulating, using the one or more computers, the comparative performance measurements across the plurality of data streams to be recognized; and

reporting as the final recognition result for the at least one given sample of data the label value corresponding to the model in the linked set of model that performed best in the accumulated performance measurements.

22. A method of pattern recognition system development comprising:

obtaining a first and second recognition system;

obtaining a collection of development test data to be recognized by the first and second recognition systems;

recognizing, using one or more computers, the collection of development test data using each of the first and second recognition system;

obtaining a third recognition system;

recognizing, using the one or more computers, the collection of development text data using the third recognition system;

evaluating, using the one or more computers, the comparative performance of the first and second systems based on the output of the third recognition system.

23. A method of pattern recognition system development as in claim 22, in which the first and second recognition system are variants of the same recognition system with a relatively small difference between the variants.

24. A method of pattern recognition system development as in claim 22, wherein the first and second recognition systems are each restricted in capability relative to the third recognition system.

25. A method of pattern recognition system development as in claim 24, wherein the first and second recognition systems can only use an amount of computation that is less than that used by the third recognition system.

26. A method of pattern recognition system development as in claim 24, wherein the first and second recognition system can only use an amount of computer memory that is less than that used by the third recognition system.

27. A method of pattern recognition system development as in claim 22, wherein the third recognition system has at least one additional knowledge source not available to the first and second recognition systems.

28. A method of pattern recognition system development as in claim 22, wherein the third recognition system is designed to be unbiased in comparing the first recognition system to the second recognition system.

29. A method of pattern recognition system development as in claim 22, further comprising

performing a test of statistical significance of a null hypothesis that the performance of the first and the second recognition systems are the same, where the test statistic for the null hypothesis is based at least in part on the output of the third recognition system.

30. A method of pattern recognition system development as in claim 22, further comprising

selecting for further development the one of the first recognition system and the second recognition system that performs better in the comparative evaluation.

31. A method of pattern recognition system development as in claim 22, wherein the first recognition system uses fewer computer resources than the second recognition system and further comprising

performing, using the one or more computers, a test of statistical significance between a null hypothesis that the performance of the first recognition performs at least as well as the second recognition system and the alternative hypothesis that the second recognition system performs better than the first recognition system.

32. A method of pattern recognition system development as in claim 31, wherein the third recognition system is biased in favor of the first recognition system.

33. A method of pattern recognition system development as in claim 31, further comprising

selecting the second recognition system for further development if the second recognition system performs better at a statistically significant level otherwise selecting the first recognition system for further development.

34. A method of pattern recognition system development as in claim 22, further comprising

obtaining a set of recognition systems containing at least the first recognition system and a plurality of second recognition systems;

performing the comparative evaluation of the first recognition system and each of the plurality of second recognition systems;

deciding whether the first recognition system should be deleted from the set of recognition systems based at least in part on the comparative evaluations of the first recognition system and the plurality of second recognition systems.

35. A method of pattern recognition system development as in claim 22, further comprising

obtaining a first recognition system;

obtaining a set of recognition systems containing a plurality of second recognition systems;

performing the comparative evaluation of the first recognition system and each of the plurality of second recognition systems;

deciding whether the first recognition system should be added to the set of recognition systems based at least in part on the comparative evaluations of the first recognition system and the plurality of second recognition systems.