ROBUST PATTERN RECOGNITION SYSTEM AND METHOD USING SOCRATIC AGENTS

• US 20120203720A1
• Filed: 04/13/2012
• Published: 08/09/2012
• Est. Priority Date: 09/13/2006
• Status: Active Grant

• Current Assignee: Aurilab LLC
• Original Assignee: Aurilab LLC
• Inventors: Baker, James K.
• Granted Patent: US 8,331,656 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques

Abstract

A computer-implemented pattern recognition method, system and program product, the method comprising in one embodiment: creating electronically a linkage between a plurality of models within a classifier module within a pattern recognition system such that any one of said plurality of models may be selected as an active model in a recognition process; creating electronically a null hypothesis between at least one model of said plurality of linked models and at least a second model among said plurality of linked models; accumulating electronically evidence to accept or reject said null hypothesis until sufficient evidence is accumulated to reject said null hypothesis in favor of one of said plurality of linked models or until a stopping criterion is met; and transmitting at least a portion of the electronically accumulated evidence or a summary thereof to accept or reject said null hypothesis to a pattern classifier module.

Claims

1. A computer-implemented pattern recognition method comprising:

obtaining a trainable recognition system using one or more computers;

obtaining a first item to be recognized comprising one or more units to be recognized;

obtaining, using the one or more computers, a plurality of hypotheses for class labels of said one or more units;

creating, using the one or more computers, a plurality of trained model variants for said trainable recognition system by training said recognition system based at least in part on said first item to be recognized, and labeling each of the trained model variants respectively with the class labels based at least in part on a different one of the hypotheses in said plurality of hypotheses for the class of labels of said one or more units;

obtaining a set of practice data with labels;

performing recognition, using the one or more computers, of said practice data respectively using each of said plurality of trained model variants to obtain recognition results;

measuring performance, using the one or more computers, of each of said trained model variants based at least in part on the recognition results obtained for the set of practice data and the labels for the practice data;

determining, using the one or more computers, one of said trained model variants with a best measured performance; and

selecting, using the one or more computers, as the class labels for said first item to be recognized the class labels of said one or more units associated with the one trained model variant determined to have the best measured performance, thereby using the hypothesis that among the plurality of hypotheses has a best measured performance.

2. The pattern recognition method as in claim 1, wherein said trainable pattern recognition system obtained is capable of generating a plurality of hypotheses as to the class labels of said one or more units of said first item to be recognized, and the pattern recognition method further comprises:

obtaining, using the one or more computers, said one or more of the hypotheses as to the class labels of said one or more units of said first item to be recognized based at least in part on the recognition results obtained from said trainable recognition system.

3. The pattern recognition method as in claim 1, further comprising:

obtaining, using the one or more computers, said plurality of hypotheses based at least in part from a results lattice produced by said trainable pattern recognition system.

4. The pattern recognition method as in claim 1, further comprising:

obtaining, using the one or more computers, one or more additional recognition systems;

performing, using the one or more computers, the method of claim 1 for the first item using the one or more additional recognition systems; and

obtaining, using the one or more computers, one or more of said hypotheses based at least in part on recognition results obtained for said one or more additional recognition systems on said first item to be recognized.

5. The pattern recognition method as in claim 1, wherein said obtained practice data has been manually labeled.

6. The pattern recognition method as in claim 1, wherein said obtained practice data has been read from a known script.

7. The pattern recognition method as in claim 1, wherein said obtained practice data has been automatically labeled using one or more computers.

8. The pattern recognition method as in claim 1, further comprising:

performing a recognition task using the trained model variants on the practice data for various items therein , using the one or more computers, to obtain recognition results; and

accumulating, using the one or more computers, a test statistic measurement of performance of each of said trained model variants on the recognition task performed on each of the various items in said practice data based at least in part on the labels for the practice data and the recognition results.

9. The pattern recognition method as in claim 8, further comprising:

forming, using the one or more computers, a statistical null hypothesis that two or more of said plurality of trained model variants have equal performance as measured by said test statistic measurement;

testing, using the one or more computers, whether said null hypothesis can be rejected at a specified level of statistical significance; and

selecting, using the one or more computers, the class labels of said one or more units associated with the one trained model variant determined to have the best test statistic measurement of performance as the class labels for said first item to be recognized, thereby using the hypothesis that among the plurality of hypotheses has a best test statistic measurement of performance.

10. The pattern recognition method as in claim 9, further comprising:

performing, using the one or more computers, the method of claim 1 for the first item using one or more additional recognition systems;

obtaining, using the one or more computers, one or more of said hypotheses based at least in part on recognition results obtained from using said one or more additional recognition systems on said first item to be recognized; and

obtaining one or more continuing streams of the practice data by collecting data as one or more additional recognition systems are used on the one or more computers.

11. The pattern recognition method as in claim 10, further comprising:

selecting, using the one or more computers, one or more of the class labels for said first item to be recognized based at least in part on using the continuing one or more streams of practice data to continue accumulating said test statistic measurement of performance until either statistical significance is achieved or a stopping criterion is met.

15. The pattern recognition system as in claim 12, wherein the one or more computers are further configured to:

obtain, using the one or more computers, one or more additional recognition systems;

perform, using the one or more computers, the operations of claim 1 for the first item using the one or more additional recognition systems; and

obtain, using the one or more computers, one or more of said hypotheses based at least in part on recognition results obtained for said one or more additional recognition systems on said first item to be recognized.

21. The pattern recognition system as in claim 20, wherein the one or more computers are further configured to:

perform, using the one or more computers, the operations of claim 1 for the first item using one or more additional recognition systems;

obtain, using the one or more computers, one or more of said hypotheses based at least in part on recognition results obtained from using said one or more additional recognition systems on said first item to be recognized; and

obtain one or more continuing streams of the practice data by collecting data as one or more additional recognition systems are used on the one or more computers.

22. The pattern recognition system as in claim 21, wherein the one or more computers are further configured to:

select, using the one or more computers, one or more of the class labels for said first item to be recognized based at least in part on using the continuing one or more streams of practice data to continue accumulating said test statistic measurement of performance until either statistical significance is achieved or a stopping criterion is met.

12. A system, comprising:

one or more computer configured to;

obtain a trainable recognition system using the one or more computers;

obtain, by or to the one or more computers, a first item to be recognized comprising one or more units to be recognized;

obtain, using the one or more computers, a plurality of hypotheses for class labels of said one or more units;

create, using the one or more computers, a plurality of trained model variants for said trainable recognition system by training said recognition system based at least in part on said first item to be recognized, and labeling each of the trained model variants respectively with the class labels based at least in part on a different one of the hypotheses in said plurality of hypotheses for the class of labels of said one or more units;

obtain, by or to the one or more computers, a set of practice data with labels;

perform recognition, using the one or more computers, of said practice data respectively using each of said plurality of trained model variants to obtain recognition results;

measure performance, using the one or more computers, of each of said trained model variants based at least in part on the recognition results obtained for the set of practice data and the labels for the practice data;

determine, using the one or more computers, one of said trained model variants with a best measured performance; and

select, using the one or more computers, as the class labels for said first item to be recognized the class labels of said one or more units associated with the one trained model variant determined to have the best measured performance, thereby using the hypothesis that among the plurality of hypotheses has a best measured performance.

13. The pattern recognition system as in claim 12, wherein said trainable pattern recognition system obtained is capable of generating a plurality of hypotheses as to the class labels of said one or more units of said first item to be recognized, and wherein the one or more computers are further configured to:

obtain, using the one or more computers, said one or more of the hypotheses as to the class labels of said one or more units of said first item to be recognized based at least in part on the recognition results obtained from said trainable recognition system.

14. The pattern recognition system as in claim 12, wherein the one or more computers are further configured to:

obtain, using the one or more computers, said plurality of hypotheses based at least in part from a results lattice produced by said trainable pattern recognition system.

16. The pattern recognition system as in claim 12, wherein said obtained practice data has been manually labeled.

17. The pattern recognition system as in claim 12, wherein said obtained practice data has been read from a known script.

18. The pattern recognition system as in claim 12, wherein said obtained practice data has been automatically labeled using one or more computers.

19. The pattern recognition system as in claim 12, wherein the one or more computers are further configured to:

perform a recognition task using the trained model variants on the practice data for various items therein , using the one or more computers, to obtain recognition results; and

accumulating, using the one or more computers, a test statistic measurement of performance of each of said trained model variants on the recognition task performed on each of the various items in said practice data based at least in part on the labels for the practice data and the recognition results.

20. The pattern recognition system as in claim 19, wherein the one or more computers are further configured to:

form, using the one or more computers, a statistical null hypothesis that two or more of said plurality of trained model variants have equal performance as measured by said test statistic measurement;

test, using the one or more computers, whether said null hypothesis can be rejected at a specified level of statistical significance; and

select, using the one or more computers, the class labels of said one or more units associated with the one trained model variant determined to have the best test statistic measurement of performance as the class labels for said first item to be recognized, thereby using the hypothesis that among the plurality of hypotheses has a best test statistic measurement of performance.