Method and apparatus for training an automated speech recognition-based system

• US 7,346,507 B1
• Filed: 06/04/2003
• Issued: 03/18/2008
• Est. Priority Date: 06/05/2002
• Status: Expired due to Fees

• Current Assignee: Ramp Holdings Incorporated (Clean Harbors Incorporated)
• Original Assignee: BBN Technologies (Rtx Corporation)
• Inventors: Prasad, Rohit, Natarajan, Premkumar
• Primary Examiner(s): {hacek over (S)}mits; Talivaldis Ivars
• Assistant Examiner(s): SIEDLER, DOROTHY S
• Application Number:
  US10/454,213
• Time in Patent Office: 1,749 Days
• Field of Search: 704/207.1, 704/244, 704/256.2, 704/266, 704/270.1, 379/88.04, 379/88.07, 379/88.08, 379/88.09, 379/88.18, 379/88.05
• US Class Current: 704/244
• CPC Class Codes: G10L 15/063 Training

Abstract

A method and apparatus for building a training set for an automated speech recognition-based system, which determines the statistically optimal number of frequently requested responses to automate in order to achieve a desired automation rate. The invention may be used to select the appropriate tokens and responses to train the system and to achieve a desired “phrase coverage” for all of the many different ways human beings may phrase a request that calls for one of a plurality of frequently-requested responses. The invention also determines the statistically optimal number of tokens (spoken requests) required to train a speech recognition-based system to achieve the desired phrase coverage and optimal allocation of tokens over the set of responses that are to be automated.

Claims

1. A method for building a training set of token-response pairings for an automated speech-recognition-based system, comprising the steps of:

(a) for each response in a plurality of possible responses, calculating, based on an expected phrase coverage for said each response and a probability of occurrence for said each response, a benefit that would be achieved by adding to the training set a token-response pairing for said each response;

(b) identifying a maximum benefit response, said maximum benefit response being equal to the response from the plurality of possible responses having the maximum benefit;

(c) adding to the training set, a token-response pairing containing the maximum benefit response;

(d) incrementing a current phrase coverage for the training set by an amount equal to the product of the expected phrase coverage for the number of token-response pairings in the training set that contain the maximum benefit response, and the probability of occurrence of the maximum benefit response; and

(e) repeating steps (a) to (d) until the current phrase coverage is greater than or equal to a target phrase coverage.

2. The method of claim 1, wherein the token-response pairing containing the maximum benefit response is selected from a supply set of randomly-selected token-response pairings.

3. The method of claim 1, wherein the probability of occurrence for each response in the plurality of possible responses is determined by:

(f) providing a collection of responses supplied in response to a predetermined number of user requests; and

(g) for each response in the plurality of possible responses,

i) counting the number of times said each response occurs in the collection to generate a frequency of occurrence for said each response, and

ii) dividing the frequency of occurrence by the predetermined number of user requests.

4. The method of claim 1, wherein the expected phrase coverage for said each response is determined by:

(f) providing a collection of token-response pairings supplied in response to a predetermined number of user requests; and

(g) for said each response in the plurality of possible responses,

i) randomly selecting from the collection a predetermined number of token-response pairings containing said each response to form a random set of token-response pairings,

ii) dividing the random set into a training subset and a test subset,

iii) determining a number of tokens in the test subset that are adequately predicted by the training subset, and

iv) dividing the number of adequately predicted tokens by a number of tokens in the test subset.

5. The method of claim 4, wherein the number of tokens adequately predicted equals the number of phrases in the training subset that exactly match a phrase in the test subset.

6. The method of claim 4, wherein the number of tokens adequately predicted equals the number of phrases in the training subset that match a phrase in the test subset, based on a perplexity threshold of a statistical n-gram language model.

7. The method of claim 6, wherein the statistical n-gram language model is trained against another training set of token-response pairings.

8. The method of claim 6, wherein the statistical n-gram language model is measured against a test set of token-response pairings.

9. The method of claim 1, wherein the current phrase coverage is equal to zero prior to carrying out step (d) for the first time.

10. The method of claim 1, wherein the step of calculating the benefit comprises:

(f) computing an incremental phrase coverage, said incremental phrase coverage comprising a difference between the expected phrase coverage for said each response if the training set held one additional token corresponding to said each response and the current phrase coverage for said each response; and

(g) multiplying the incremental phrase coverage by the probability of occurrence of said each response in the plurality of possible responses.

11. The method of claim 10, wherein the step of calculating the benefit further comprises multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a retrieval efficiency for said each response in the plurality of possible responses.

12. The method of claim 10, wherein the step of calculating the benefit is further carried out by multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a probabilistic cost for said each response in the plurality of possible responses.

13. The method of claim 1, wherein each token in the training set comprises a transcribed directory assistance request.

14. The method of claim 1, wherein each response in the training set comprises a transcribed response to a directory assistance request.

15. The method of claim 1, further comprising providing the training set for training an automated speech-recognition-based system.

16. The method of claim 15, wherein providing the training set comprises storing the training set.

17. The method of claim 15, wherein providing the training set comprises loading the training set onto a computer system for training the automated speech-recognition-based system.

18. The method of claim 1, further comprising providing an automated response to a request received by an automated speech-recognition-based system, wherein the response is selected based on a token-response pair in the training set.

19. A system for building a training set of token-response pairings for an automated speech-recognition-based system, comprising:

means for calculating, for each response in a plurality of possible responses, a benefit that would be achieved by adding to the training set a token-response pairing for said each response;

means for identifying a maximum benefit response, said maximum benefit response being equal to the response from the plurality of possible responses having the maximum benefit;

means for adding to the training set, a token-response pairing containing the maximum benefit response; and

means for incrementing a current phrase coverage for the training set by an amount equal to the product of the expected phrase coverage for the number of token-response pairings in the training set that contain the maximum benefit response, and the probability of occurrence of the maximum benefit response.

20. The system of claim 19, further comprising means for selecting the maximum benefit response from a supply set of randomly-selected token-response pairings.

21. The system of claim 19, wherein the calculating means is responsive to a means for computing the probability of occurrence for said each response and a means for computing the expected phrase coverage for said each response.

22. The system of claim 19, wherein the means for computing the probability of occurrence for said each response comprises:

a collection of responses supplied in response to a predetermined number of user requests;

means for counting the number of times said each response occurs in the collection to generate a frequency of occurrence for said each response; and

means for dividing the frequency of occurrence for said each response by the predetermined number of user requests.

23. The system of claim 19, wherein the means for computing the expected phrase coverage for said each response comprises:

a collection of token-response pairings supplied in response to a predetermined number of user requests;

means for randomly selecting from the collection a predetermined number of token-response pairings containing said each response to form a random set of token-response pairings;

means for dividing the random set into a training subset and a test subset;

means for determining a number of tokens in the test subset that are adequately predicted by the training subset; and

means for dividing the number of adequately predicted tokens by a number of tokens in the test subset.

24. The system of claim 23, wherein the number of tokens adequately predicted equals the number of phrases in the training subset that exactly match a phrase in the test subset.

25. The system of claim 23, wherein the number of tokens adequately predicted equals the number of phrases in the training subset that match a phrase in the test subset, based on a perplexity threshold of a statistical n-gram language model.

26. The system of claim 25, wherein the statistical n-gram language model is trained against another training set of token-response pairings.

27. The system of claim 25, wherein the statistical n-gram language model is measured against a test set of token-response pairings.

28. The system of claim 19, wherein the means for calculating the benefit comprises:

means for computing an incremental phrase coverage, said incremental phrase coverage comprising the difference between the expected phrase coverage for said each response if the training set held one additional token

corresponding to said each response and the current phrase coverage for said each response; and

means for multiplying the incremental phrase coverage by the probability of occurrence of said each response in the plurality of possible responses.

29. The system of claim 28, wherein the means for calculating the benefit further comprises means for multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a retrieval efficiency for said each response in the plurality of possible responses.

30. The system of claim 28, wherein the means for calculating the benefit further comprises means for multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a probabilistic cost for said each response in the plurality of possible responses.

31. The system of claim 19, further comprising means for providing an training set for training an automated speech-recognition-based system.

32. The system of claim 31, wherein the means for providing the training set comprises means for storing the training set.

33. The system of claim 31, wherein the means for providing the training set comprises means for loading the training set onto a computer system for training the automated speech-recognition-based system.

34. The system of claim 19, further comprising means for providing an automated response to a request received by an automated speech-recognition-based system, wherein the response is selected based on a token-response pair in the training set.

35. A system for generating a training set of token-response pairings for an automated speech-recognition-based system, comprising:

a phrase coverage processor module configured to calculate a phrase coverage associated with a response out of a plurality of possible responses;

a probability of occurrence module configured to compute, responsive to a prior collection of token-response pairings, a statistical probability that said response will occur in a predetermined number of responses;

a benefit processor configured to determine, responsive to the phrase coverage processor module and the probability of occurrence module,

a benefit that would be achieved by adding a token-response pairing to the training set, and

a maximum benefit response, said maximum benefit response being equal to the response from the plurality of responses having maximum benefit; and

a training set generation module configured to add to the training set a token response pairing from the supply set containing the maximum benefit response.

36. The system of claim 35, wherein the training set generation module selects the token-response pairing containing the maximum benefit from a supply set of randomly-selected token-response pairings.

37. The system of claim 35, wherein the benefit processor comprises:

means for computing an incremental phrase coverage, said incremental phrase coverage comprising the difference between the expected phrase coverage for said each response if the training set held one additional token corresponding to said each response and the current phrase coverage for said each response; and

means for multiplying the incremental phrase coverage by the probability of occurrence of said each response in the plurality of possible responses.

38. The system of claim 37, wherein the means for calculating the benefit further comprises means for multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a retrieval efficiency for said each response in the plurality of possible responses.

39. The system of claim 37, wherein the means for calculating the benefit further comprises means for multiplying the product of the incremental phrase coverage and the probability of occurrence for said each response by a probabilistic cost for said each response in the plurality of possible responses.

40. The system of claim 35, further comprising a training set memory module that stores the training set for training an automated speech-recognition-based system.