System and method for automatic quality assurance of user enrollment in a recognition system

• US 6,826,306 B1
• Filed: 01/29/1999
• Issued: 11/30/2004
• Est. Priority Date: 01/29/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lewis, James R., Ortega, Kerry A., Subrahmonia, Jayashree, Wang, Huifang, Ratzlaff, Eugene H., Perrone, Michael P., Buskirk, Ron Van, Maners, Julia E.
• Primary Examiner(s): Dastouri, Mehrdad
• Application Number:
  US09/240,146
• Time in Patent Office: 2,132 Days
• Field of Search: 382/187, 382/189, 382/229, 382/230, 382/228, 382/159, 382/119, 382/160, 382/161, 382/186, 382/188, 704/231, 704/243, 704/244, 340/825.34, 713/186
• US Class Current: 382/187
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G10L 15/07 to the speaker

Abstract

A system and method for automatically providing quality assurance for user enrollment in a recognition system. Advantageously, the quality a new enrollment (i.e., a newly trained user-dependent prototype) is assessed before the new enrollment is accepted in place of a current enrollment. This quality check is performed by decoding stored user test data using the new enrollment, comparing the decoding results of the new enrollment to the known script used to generate the test data to obtain an accuracy score for the new enrollment, and then comparing the accuracy score for the new enrollment with an accuracy score of a previous qualified enrollment (or, in the case where there is no previous, qualified enrollment, to the accuracy of the speaker independent model). If the decoding results of the new enrollment are acceptable, the new enrollment will be used for recognition; otherwise it will be rejected and discarded.

Claims

1. A method for assuring the quality of user enrollment in a recognition system, comprising the steps of:

training a new user-dependent prototype;

computing an accuracy score for the new user-dependent prototype;

determining if the new user-dependent prototype is acceptable by comparing the computed accuracy score for the new user-dependent prototype with a previously computed accuracy score for a previously generated user-dependent prototype in a previous training process; and

applying the new user-dependent prototype for recognition if the new user-dependent prototype is deemed acceptable.

2. The method of claim 1, wherein the computed accuracy score for the new user-dependent prototype is determined from decoding results obtained by decoding the new user-dependent prototype with user test data, and wherein the previously computed accuracy score for the previously generated user-dependent prototype is obtained by decoding the previously generated user-dependent prototype with the user test data.

3. The method of claim 2, wherein the recognition system is a handwriting recognition system and user test data comprises handwriting data of words of a scripted text.

4. The method of claim 2, wherein the recognition system is a speech recognition system and the test data comprises recorded acoustic utterances of spoken words of a scripted text.

5. The method of claim 1, wherein the step of determining if the new user-dependent prototype is acceptable includes the step of automatically accepting the new user-dependent prototype if the computed accuracy score of the user-dependent prototype exceeds the previously computed accuracy score of the previously computed user-dependent prototype by a prespecified threshold.

6. The method of claim 5, wherein the prespecified threshold is selected by the user.

7. The method of claim 1, wherein the step of determining if the new user-dependent prototype is acceptable includes the step of displaying the computed accuracy score and the previously computed accuracy score and prompting the user to select the new user-dependent prototype if the computed accuracy score is acceptable to the user.

8. The method of claim 1, further comprising the step of storing the computed accuracy score of the new user-enrollment, wherein the stored accuracy score is applied for assessing the quality of a subsequent enrollment by the user.

9. The method of claim 1, further comprising the step of one of storing and discarding the previously computed user-dependent prototype if the new user-dependent prototype is accepted.

10. A method for assuring the quality of user enrollment in a recognition system, comprising the steps of:

collecting test data from a user seeking enrollment;

computing a first accuracy score from decoding results obtained by decoding the test data with a user-independent prototype;

training a new user-dependent prototype using the test data;

computing a second accuracy score for the new user-dependent prototype from decoding results obtained by decoding the user-dependent prototype with the test data;

comparing the first accuracy score with the second accuracy score to determine if the new user-dependent prototype is acceptable; and

applying the new user-dependent prototype for recognition if the new user-dependent prototype is deemed acceptable.

11. The method of claim 10, wherein the test data is collected from scripted text.

12. The method of claim 10, further including the step of verifying the quality of the test data.

13. The method of claim 12, wherein the step of verifying the quality of the test data includes the steps of:

comparing the first accuracy score with a prespecified threshold;

accepting the test data and storing the first accuracy score if the first accuracy score exceeds the prespecified threshold; and

discarding the test data if the first accuracy score does not exceed the prespecified threshold.

14. The method of claim 13, further comprising the steps of:

collecting and decoding additional test data if the first accuracy score is determined to be less than the prespecified threshold limit; and

repeating the steps of collecting and decoding the additional test data until the first accuracy score exceeds the prespecified threshold value.

15. The method of claim 10, wherein the step of training the user-dependent model includes the steps of:

collecting training data provided from the user; and

training the user-dependent model with the training data.

16. The method of claim 15, further comprising the step of cleaning and verifying the collected training data.

17. The method of claim 10, wherein the user-dependent model is automatically accepted if the second accuracy score exceeds the first accuracy score by a prespecified threshold.

18. The method of claim 17, wherein the prespecified threshold is selected by the user.

19. The method of claim 10, wherein the user-dependent prototype is manually accepted by a user.

20. A program storage device readable by a recognition machine, tangibly embodying a program of instructions executable by the machine to perform method steps for assuring the quality of user enrollment, the method steps comprising:

training a new user-dependent prototype;

computing an accuracy score for the new user-dependent prototype;

determining if the new user-dependent prototype is acceptable by comparing the computed accuracy score for the new user-dependent prototype with a previously computed accuracy score for a previously generated user-dependent prototype in a previous training process; and

applying the new user-dependent prototype for recognition if the new user-dependent prototype is deemed acceptable.

21. The program storage device of claim 20, wherein the computed accuracy score for the new user-dependent prototype is determined from decoding results obtained by decoding the new user-dependent prototype with user test data, and wherein the previously computed accuracy score for the previously generated user-dependent prototype is obtained by decoding the previously generated user-dependent prototype with the user test data.

22. The program storage device of claim 20, wherein the instructions for performing the step of determining if the new user-dependent prototype is acceptable includes instructions for performing the step of automatically accepting the new user-dependent prototype if the computed accuracy score of the new user-dependent prototype exceeds the previously computed accuracy score of the previously generated user-dependent prototype by a prespecified threshold.

23. The program storage device of claim 22, further comprising instructions for allowing the user to select the prespecified threshold.

24. The program storage device of claim 20, wherein the step of determining if the new user-dependent prototype is acceptable includes instructions for performing the steps of displaying the computed accuracy score and the previously computed accuracy score and prompting the user to select the new user-dependent prototype if the computed accuracy score is acceptable to the user.

25. The program storage device of claim 20, further comprising instructions for performing the step of storing the computed accuracy score of the new user-enrollment, wherein the stored accuracy score is applied for assessing the quality of a subsequent enrollment by the user.

26. A recognition system for assuring the quality of user enrollment, comprising:

means for training a new user-dependent prototype;

means for computing an accuracy score for the new user-dependent prototype;

means for comparing the computed accuracy score for the new user-dependent prototype with a previously computed accuracy score for a previously generated user-dependent prototype in a previous training process to determine if the new user-dependent prototype is acceptable; and

means for applying the new user-dependent prototype for recognition if the new user-dependent prototype is deemed acceptable.

27. The system of claim 26, wherein the computed accuracy score for the new user-dependent prototype is determined from decoding results obtained by decoding the new user-dependent prototype with user test data, and wherein the previously computed accuracy score for the previously generated user-dependent prototype is obtained by decoding the previously generated user-dependent prototype with the user test data.

28. The system of claim of claim 27, wherein the recognition system is a handwriting recognition system and the user test data comprises handwriting data of words of a scripted text.

29. The system of claim 27, wherein the recognition system is a speech recognition system and the test data comprises recorded acoustic utterances of spoken words of a scripted text.

30. The system of claim 26, wherein the means for comparing includes means for automatically accepting the new user-dependent prototype if the computed accuracy score of the new user-dependent prototype exceeds the previously computed accuracy score of the previously generated user-dependent prototype by a prespecified threshold.

31. The system of claim 26, wherein the means for comparing includes means for displaying the computed accuracy score and the previously computed accuracy score and means for prompting the user to select the new user-dependent prototype if the computed accuracy score is acceptable to the user.

32. The system of claim 26, further comprising means for storing the computed accuracy score of the new user-enrollment and means for accessing and applying the stored accuracy score for assessing the quality of a subsequent enrollment by the user.