Accuracy model for recognition signal processing engines

• US 7,580,570 B2
• Filed: 12/09/2003
• Issued: 08/25/2009
• Est. Priority Date: 12/09/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sarac, Radmila, Manu, Mitica
• Primary Examiner(s): Ahmed; Samir A.
• Assistant Examiner(s): Fitzpatrick; Atiba O
• Application Number:
  US10/730,113
• Publication Number:
  US 20050125225A1
• Time in Patent Office: 2,086 Days
• Field of Search: 382/155, 382/156, 382/157, 382/158, 382/159, 382/160, 382/161, 382/181, 382/186, 382/187, 382/190, 382/224, 382/225, 382/226, 382/227, 382/228, 382/229, 704/200, 704/231, 704/232, 704/236, 704/242, 704/251, 704/255, 704/256, 704/256.1, 704/256.2, 706/15, 706/16, 706/20, 706/22, 706/25, 706/26, 706/27, 706/28, 706/31
• US Class Current: 382/181
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

Signals to be processed are categorized based on signal characteristics such as physical aspects, context, conditions under which the signals were generated and source, and/or based on other variables. Categorized sets of signals are processed, and an accuracy for each set calculated. Weights are then applied to accuracy values for the sets, and the weighted values summed. In some cases, certain sums are then weighted and further summed.

Claims

1. A method for evaluating a computer learning signal processing engine, comprising:

employing at least one processor to execute computer-executable instructions stored on at least one computer-readable medium to perform the following acts;

identifying a first group of signal sets representing inputs by humans, each signal set of the first group having an associated range of values for a variable corresponding to the first group, the variable being one of a plurality of variables having values characterizing multiple signals to be processed, the plurality of variables including at least a variable characterizing a user scenario in which a signal was generated, the user scenario including at least one of a software application or an operation performed within a software application;

calculating an accuracy score for each signal set of the first group using the signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the first group signal sets, each weight factor representing a relative importance of one of the associated ranges of values for the first variable;

summing weighted accuracy scores for the first group of signal sets to yield a first summed accuracy score;

identifying additional groups of signal sets, each group having a corresponding variable of the plurality of variables, each signal set of a group having an associated range of values for the corresponding variable;

calculating accuracy scores for each signal set of each additional group using the signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the signal sets of the additional groups;

summing the weighted accuracy scores within each of the additional groups to yield additional summed accuracy scores;

using summed accuracy scores from at least two separate training sets comprising samples, wherein each separate training set is distinguished by a feature characteristic identified based upon a demographic characteristic associated with a source of the samples, to create one or more signal processing engines to handle multiple applications to one or more new groups of signal sets for which a frequency of the feature characteristic of the separate training sets are known or assumed, by weighting the summed accuracy score associated with each training set according to the frequency and then combining the weighted summed accuracy scores to yield scone-specific accuracy scores;

applying weights to each of the scope-specific accuracy scores;

summing the weighted scope-specific accuracy scores to yield a combined accuracy score;

summing the combined accuracy score with at least one other summed accuracy score relating to a physical attribute of one or more input signals to yield an input accuracy score; and

summing the input accuracy score with a summed accuracy score for a group of signal sets corresponding to the user scenario variable to yield an overall accuracy score.

2. The method of claim 1, wherein at least some of the associated ranges of values are single values.

3. The method of claim 1, wherein combining the summed accuracy scores comprises:

applying weight factors to a plurality of the summed accuracy scores, each of the weight factors representing a relative importance of the variable corresponding to the group from which a different one of the plurality of summed accuracy scores was derived, and

summing the plurality of weighted summed accuracy scores to yield a subsequent sum.

4. The method of claim 3, wherein combining the summed accuracy scores further comprises:

applying additional weight factors to the subsequent sum and to at least one of the summed accuracy scores, the additional weight factors representing the relative importance of at least one variable corresponding to groups from which the subsequent sum was derived and of the variable corresponding to the group from which the at least one summed accuracy score was derived, and

summing the additionally weighted subsequent sum and the additionally weighted at least one summed accuracy score.

5. The method of claim 3, further comprising:

identifying an additional variable having values characterizing multiple signals to be processed;

identifying another group of signal sets, each signal set of the group having an associated range of values for the additional variable;

calculating an accuracy score for each signal set of the additional variable group using the signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the additional group signal sets, each weight factor representing a relative importance of one of the associated ranges of values for the additional variable;

summing weighted accuracy scores for additional group signal sets to yield an additional summed accuracy score;

applying weight factors to the plurality of the summed accuracy scores and to the additional summed accuracy score, each of the weight factors representing a relative importance of the variable corresponding to the group from which a different one of the plurality of summed accuracy scores was derived or of the additional variable, and

summing the plurality of weighted summed accuracy scores and the weighted additional accuracy score.

6. The method of claim 1, further comprising

selecting a variable of the plurality of variables;

selecting sub-variables, each sub-variable having a range of values for a value of the selected variable;

calculating an accuracy score for signal sets in groups of signal sets corresponding to the sub-variables, each of the signal sets within a group corresponding to a range of values for the corresponding sub-variable;

applying weights to the accuracy scores for the signal sets in each sub-variable group and summing the weighted scores within each of said groups to yield sub-variable accuracy scores; and

applying weights to the sub-variable accuracy scores and summing the weighted sub-variable accuracy scores.

7. The method of claim 1, wherein:

at least one variable is a source variable having values characterizing a source of at least one of the signals to be processed,

at least one variable is a context variable having values characterizing the context of at least one of the signals to be processed, and

at least one variable is a physical variable having values characterizing physical attributes of at least one of the signals to be processed.

8. The method of claim 1, wherein:

the signal processing engine to be evaluated comprises handwriting recognition software,

the signals to be processed comprise handwriting samples,

values of the source variable comprise demographic data regarding users creating handwriting samples,

values of the context variable comprise data regarding the context of handwriting samples, and

values of the physical variable comprise at least one of data regarding the scaling of a handwriting sample, data regarding the relative angle of components of a handwriting sample, or data regarding the spacing between components of a handwriting sample.

9. The method of claim 8, wherein:

the groups of signal sets comprise;

groups of demographic signal sets, each demographic signal set in a group having an associated range of values for the source variable and the same value for the context variable,

groups of scaling signal sets, each scaling signal set in a group having an associated range of values for the scaling of a handwriting sample and the same value for the context variable, and

groups of angle signal sets, each angle signal set in a group having an associated range of values for at least one angle of a component of a handwriting sample and the same value for the context variable,

at least the summed accuracy scores for demographic signal set, scaling signal set and angle signal set groups of signals having a common context variable value are weighted and summed to yield the scope-specific accuracy scores,

the scope-specific scores are weighted and summed to yield the combined accuracy score,

the combined accuracy score and at least one other summed accuracy score are weighted and summed to yield a word accuracy score, and

at least the word accuracy score and a summed accuracy score for a group corresponding to the user scenario variable are weighted and summed to yield the overall accuracy score.

10. The method of claim 1, wherein:

the signal processing engine to be evaluated comprises speech recognition software,

the signals to be processed comprise speech samples,

values of the source variable comprise demographic data regarding users creating speech samples,

values of the context variable comprise data regarding the context of speech

samples, and

values of the physical variable comprise at least one of data regarding the scaling of a speech sample or data regarding the spacing between components of a speech sample.

11. The method of claim 10, wherein:

the groups of signal sets comprise;

groups of demographic signal sets, each demographic signal set in a group having an associated range of values for the source variable and the same value for the context variable, and

groups of scaling signal sets, each scaling signal set in a group having an associated range of values for the scaling of a speech sample and the same value for the context variable,

at least the summed accuracy scores for demographic signal set and scaling signal set groups of signals having a common context variable value are weighted and summed to yield the scope-specific accuracy scores,

the scope-specific accuracy scores are weighted and summed to yield the combined accuracy score,

the combined accuracy score and at least one other summed accuracy score are weighted and summed to yield a word accuracy score, and

at least the word accuracy score and a summed accuracy score for a group corresponding to the user scenario variable are weighted and summed to yield the overall accuracy score.

12. The method of claim 1, further comprising applying a transforming function to a sum.

13. The method of claim 12, wherein the transform function comprises:

outputting the sum when the sum is above a threshold, and

outputting number having a large absolute value when the sum is not above the threshold.

14. The method of claim 1, further comprising:

setting accuracy scores for signal sets to equal 1;

applying confidence scores to weight factors; and

calculating an overall confidence score.

15. A computer-readable storage medium having stored thereon data representing sequences of instructions which, when executed by a processor, cause the processor to perform steps comprising:

identifying a first group of signal sets representing inputs by humans, each signal set of the first group having an associated range of values for a variable corresponding to the first group, the variable being one of a plurality of variables having values characterizing multiple signals to be processed, the plurality of variables including at least a variable characterizing a user scenario in which a signal was generated, the user scenario including at least one of a software application or an operation performed within a software application;

calculating an accuracy score for each signal set of the first group using a signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the first group signal sets, each weight factor representing a relative importance of one of the associated ranges of values for the first variable;

summing weighted accuracy scores for first group signal sets to yield a first summed accuracy score;

identifying additional groups of signal sets, each group having a corresponding variable of the plurality, each signal set of a group having an associated range of values for the corresponding variable;

calculating accuracy scores for each signal set of each additional group using the signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the signal sets of the additional groups, the weight factors within each of the additional groups representing a relative importance of associated ranges of values for the variable corresponding to the group;

summing the weighted accuracy scores within each of the additional groups to yield additional summed accuracy scores;

using summed accuracy scores from at least two separate training sets comprising samples, each separate training set is identified by a feature characteristic determined based upon a demographic characteristic associated with a source of the samples, to create one or more signal processing engines to handle multiple applications to one or more new groups of signal sets for which a frequency of the feature characteristic of the separate training sets are known or assumed, by weighting the summed accuracy score associated with each training set according to the frequency and then combining the weighted summed accuracy scores to yield scope-specific accuracy scores;

applying weights to each of the scope-specific accuracy scores;

summing the weighted scope-specific accuracy scores to yield a combined accuracy score;

summing the combined accuracy score with at least one other summed accuracy score relating to a physical attribute of one or more input signals to yield an input accuracy score; and

summing the input accuracy score with a summed accuracy score for a group of signal sets corresponding to the user scenario variable to yield an overall accuracy score.

16. The computer-readable storage medium of claim 15, wherein at least some of the associated ranges of values are single values.

17. The computer-readable storage medium of claim 15, wherein combining the summed accuracy scores comprises:

applying weight factors to a plurality of the summed accuracy scores, each of the weight factors representing a relative importance of the variable corresponding to the group from which a different one of the plurality of summed accuracy scores was derived, and

summing the plurality of weighted summed accuracy scores to yield a subsequent sum.

18. The computer-readable storage medium of claim 17, wherein combining the summed accuracy scores further comprises:

applying additional weight factors to the subsequent sum and to at least one of the summed accuracy scores, the additional weight factors representing the relative importance of at least one variable corresponding to groups from which the subsequent sum was derived and of the variable corresponding to the group from which the at least one summed accuracy score was derived, and

summing the additionally weighted subsequent sum and the additionally weighted at least one summed accuracy score.

19. The computer-readable storage medium of claim 17, comprising additional data representing sequences of instructions which, when executed by a processor, cause the processor to perform additional steps comprising:

identifying an additional variable having values characterizing multiple signals to be processed;

identifying another group of signal sets, each signal set of the group having an associated range of values for the additional variable;

calculating an accuracy score for each signal set of the additional variable group using the signal processing engine to be evaluated;

applying weight factors to the accuracy scores for the additional group signal sets, each weight factor representing a relative importance of one of the associated ranges of values for the additional variable;

summing weighted accuracy scores for additional group signal sets to yield an additional summed accuracy score;

applying weight factors to the plurality of the summed accuracy scores and to the additional summed accuracy score, each of the weight factors representing a relative importance of the variable corresponding to the group from which a different one of the plurality of summed accuracy scores was derived or of the additional variable, and

summing the plurality of weighted summed accuracy scores and the weighted additional accuracy score.

20. The computer-readable storage medium of claim 15, comprising additional data representing sequences of instructions which, when executed by a processor, cause the processor to perform additional steps comprising:

selecting a variable of the plurality;

selecting sub-variables, each sub-variable having a range of values for a value of the selected variable;

calculating an accuracy score for signal sets in groups of signal sets corresponding to the sub-variables, each of the signal sets within a group corresponding to a range of values for the corresponding sub-variable;

applying weights to the accuracy scores for the signal sets in each sub-variable group and summing the weighted scores within each of said groups to yield sub-variable accuracy scores; and

applying weights to the sub-variable accuracy scores and summing the weighted sub-variable accuracy scores.

21. The computer-readable storage medium of claim 15, wherein:

at least one variable is a source variable having values characterizing a source of at least one of the signals to be processed,

at least one variable is a context variable having values characterizing the context of at least one of the signals to be processed, and

at least one variable is a physical variable having values characterizing physical attributes of at least one of the signals to be processed.

22. The computer-readable storage medium of claim 15, wherein:

the signal processing engine to be evaluated comprises handwriting recognition software,

the signals to be processed comprise handwriting samples,

values of the source variable comprise demographic data regarding users creating handwriting samples,

values of the context variable comprise data regarding the context of handwriting samples, and

values of the physical variable comprise at least one of data regarding the scaling of a handwriting sample, data regarding the relative angle of components of a handwriting sample or data regarding the spacing between components of a handwriting sample.

23. The computer-readable storage medium of claim 22, wherein:

the groups of signal sets comprise;

groups of demographic signal sets, each demographic signal set in a group having an associated range of values for the source variable and the same value for the context variable,

groups of scaling signal sets, each scaling signal set in a group having an associated range of values for the scaling of a handwriting sample and the same value for the context variable, and

groups of angle signal sets, each angle signal set in a group having an associated range of values for at least one angle of a component of a handwriting sample and the same value for the context variable,

at least the summed accuracy scores for demographic signal set, scaling signal set and angle signal set groups of signals having a common context variable value are weighted and summed to yield the scope-specific accuracy scores,

the scope-specific accuracy scores are weighted and summed to yield the combined accuracy score,

the combined accuracy score and at least one other summed accuracy score are weighted and summed to yield a word accuracy score, and

at least the word accuracy score and a summed accuracy score for a group corresponding to the user scenario variable are weighted and summed to yield the overall accuracy score.

24. The computer-readable storage medium of claim 15, wherein:

the signal processing engine to be evaluated comprises speech recognition software,

the signals to be processed comprise speech samples,

values of the source variable comprise demographic data regarding users creating speech samples,

values of the context variable comprise data regarding the context of speech samples, and

values of the physical variable comprise at least one of data regarding the scaling of a speech sample or data regarding the spacing between components of a speech sample.

25. The computer-readable storage medium of claim 24, wherein:

the groups of signal sets comprise;

groups of demographic signal sets, each demographic signal set in a group having an associated range of values for the source variable and the same value for the context variable, and

groups of scaling signal sets, each scaling signal set in a group having an associated range of values for the scaling of a speech sample and the same value for the context variable,

at least the summed accuracy scores for demographic signal set and scaling signal set groups of signals having a common context variable value are weighted and summed to yield the scope-specific accuracy scores,

the scope-specific accuracy scores are weighted and summed to yield the combined accuracy score,

the combined accuracy score and at least one other summed accuracy score are weighted and summed to yield a word accuracy score, and

at least the word accuracy score and a summed accuracy score for a group corresponding to the user scenario variable are weighted and summed to yield the overall accuracy score.

26. The computer-readable storage medium of claim 15, comprising additional data representing sequences of instructions which, when executed by a processor, cause the processor to perform additional steps comprising applying a transforming function to a sum.

27. The computer-readable storage medium of claim 26, wherein the transform function comprises:

outputting the sum when the sum is above a threshold, and

outputting number having a large absolute value when the sum is not above the threshold.

28. The computer-readable storage medium of claim 15, comprising additional data representing

sequences of instructions which, when executed by a processor, cause the processor to perform additional steps comprising:

setting accuracy scores for signal sets to equal 1;

applying confidence scores to weight factors; and

calculating an overall confidence score.