System for predicting speech recognition accuracy and development for a dialog system

• US 20060190252A1
• Filed: 02/11/2004
• Published: 08/24/2006
• Est. Priority Date: 02/11/2003
• Status: Active Grant

• Current Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Original Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Inventors: Starkie, Bradford
• Granted Patent: US 7,917,363 B2
• Time in Patent Office: Days
• US Class Current: 704/240
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: H04M 2203/355 Interactive dialogue design tools, features or methods
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

A system for developing dialog systems estimates the speech recognition accuracy achievable when using a dialog system, and the number of example input phrases required to achieve a desired speech recognition accuracy. The maximum achievable speech recognition accuracy is also predicted. The development system thereby allows a developer of a dialog system to determine how and whether to continue development of the dialog system. The development system includes a grammatical inference engine for generating a grammar for the dialog system on the basis of example input phrases, and an in-grammar speech recognition accuracy estimator for generating the estimate of speech recognition accuracy on the basis of probabilities of confusing phonemes of input phrases with phonemes of the grammar.

Claims

1. A process for estimating the speech recognition accuracy of a dialog system, including:

generating a grammar from a plurality of example phrases;

determining respective probabilities for correctly identifying words of an input phrase with corresponding words of said grammar; and

generating a probability for correctly recognising said input phrase by multiplying said respective probabilities.

2. A process as claimed in claim 1, wherein said probabilities are probabilities of confusing words of said input phrase and words of said grammar.

3. A process as claimed in claim 2, wherein said probabilities of confusing are determined on the basis of phonetic similarities of words of said input phrase and words of said grammar.

4. A process as claimed in claim 3, wherein the probability of confusing one word with another is determined on the basis of a confusion matrix generated from one or more probabilities of confusing phonemes with other phonemes, deleting phonemes, and/or inserting phonemes.

5. A process as claimed in claim 4, wherein said probabilities are determined on the basis of phonetic Levenstein distances between phonemes.

6. A process as claimed in claim 4, wherein a probability of confusing one word with another is determined from a maximum probability alignment of phonemes of said Words, said alignment generated from said confusion matrix.

7. A process as claimed in claim 3, wherein said probability of confusing one phrase with another is derived using a list of branching decisions within a context-free grammar.

8. A process as claimed in claim 7, wherein said branching decisions are determined by generating from said grammar inactive edges corresponding to a parse tree, and active edges including only terminal symbols.

9. A process as claimed in claim 8, wherein said edges are generated on the basis of a form of said grammar wherein each rule of said grammar is either empty, or the one or more symbols on the right hand side of the rule are all terminal symbols or all non-terminal symbols.

10. A process as claimed in claim 1, including generating an estimate for the speech recognition accuracy of said dialog system from the probabilities for correctly recognising each of a plurality of input phrases.

31. A development system having components for executing the steps of any one of claims 1 to 30.

32. A computer readable storage medium having stored thereon program code for executing the steps of any one of claims 1 to 30.

11. A process for predicting in-grammar speech recognition accuracy of a dialog system, including comparing phonetic similarities of phrases allowed by a grammar for said dialog system and example phrases.

12. A process for use in developing a dialog system, including:

generating a grammar for said dialog system on the basis of example input phrases for said dialog system; and

generating an estimate of speech recognition accuracy achievable when using said dialog system on the basis of probabilities of confusing phonemes of input phrases with phonemes of said grammar to allow a developer of said dialog system to determine development of said dialog system.

13. A process as claimed in claim 12, including generating a measure of the uncertainty of said estimate.

14. A process as claimed in claim 12, including generating, on the basis of said relationship, an estimate of the maximum speech recognition accuracy achievable when using said dialog system to provide said predetermined speech recognition accuracy.

15. A process for use in developing a dialog system, including:

generating grammars for said dialog system on the basis of respective sets of example input phrases for said dialog system, said sets including different numbers of example input phrases;

determining, on the basis of said grammars, a relationship between a number of example input phrases and an estimated probability that an input phrase is correctly recognised; and

generating an estimate of the number of example input phrases required to achieve a predetermined speech recognition accuracy when using said dialog system to allow a developer of said dialog system to determine development of said dialog system.

16. A process for use in developing a dialog system, including generating estimate data representative of the number of example phrases required to achieve a predetermined speech recognition accuracy when using said dialog system.

17. A process as claimed in claim 16, wherein said generating includes:

generating a test set of example input phrases for said dialog system;

generating training sets of example input phrases for said dialog system, said training sets comprising different respective numbers of said example input phrases;

generating respective grammars on the basis of said training sets;

determining the respective portions of said test set covered by said training sets; and

determining, on the basis of said portions and the respective numbers of input phrases, a relationship between a number of example input phrases and an estimated probability that an input phrase is correctly recognised.

18. A process as claimed in claim 17, including generating a value for the number of example phrases required to achieve said predetermined speech recognition accuracy on the basis of said relationship.

19. A process as claimed in claim 17, wherein said step of determining a relationship includes:

determining a first relationship between a number of example input phrases and a probability that an input phrase is in-grammar;

determining a second relationship between a number of example input phrases and a probability that an in-grammar input phrase is correctly recognised; and

determining, on the basis of said first relationship and said second relationship, said relationship between a number of example input phrases and an estimated probability that an input phrase is correctly recognised.

20. A process as claimed in claim 17, including determining a relationship between a probability that an input phrase is correctly recognised and a probability that a meaning of said input phrase is correctly recognised.

21. A process as claimed in claim 17, wherein said test set and said training sets are generated by randomly selecting phrases from a plurality of example input phrases for said dialog system.

22. A process as claimed in claim 16, wherein said predetermined speech recognition accuracy includes an estimate of the maximum speech recognition accuracy achievable when using said dialog system.

23. A process for predicting development of a dialog system, including:

providing example phrases for said spoken dialog system;

generating a test set and training sets of various sizes from said example phrases;

generating respective grammars from said training sets;

determining respective portions of said test set not predicted by said grammars;

determining a relationship between said portions and said sizes; and

determining a probability that a spoken phrase provided to said dialog system is consistent with at least one of said grammars.

24. A process as claimed in claim 23, wherein said relationship is determined by regression.

25. A process as claimed in claim 24, wherein said process includes generating respective values for said regression from said portions of said test set not predicted by said grammars.

26. A process for use in developing a dialog system, including:

generating first function data, representing the respective probabilities that a phrase provided to said dialog system is predicted by grammars of said dialog system as a function of the number of example phrases used to generate said grammars;

generating second function data, representing the probability that an in-grammar phrase provided to said dialog system is correctly recognised as a function of the number of example phrases used to generate said grammars;

generating a third function on the basis of said first function and said second function, said third function representing the probability that a spoken phrase provided to said system is correctly recognised as a function of the number of examples used to generate said grammars.

27. A process as claimed in claim 26, wherein the process includes generating an estimate of the maximum speech recognition accuracy achievable for said spoken dialog system.

28. A process as claimed in claim 26, wherein the process includes generating an estimate of said number of said example phrases corresponding to said maximum speech recognition accuracy.

29. A process as claimed in claim 26, wherein said estimate represents the effort required to develop said spoken dialog system.

30. A process as claimed in claim 26, wherein measurements of speech recognition accuracy and/or grammar coverage at various stages of development of said spoken dialog system may be used to update the estimate of maximum speech recognition accuracy and/or the estimate of the number of samples required to be collected to achieve said maximum speech recognition accuracy.

33. A system for developing a dialog system, including:

a grammatical inference engine for generating a grammar for said dialog system on the basis of example input phrases for said dialog system; and

an in-grammar speech recognition accuracy estimator for generating an estimate of speech recognition accuracy achievable when using said dialog system on the basis of probabilities of confusing phonemes of input phrases with phonemes of said grammar to allow a developer of said dialog system to determine development of said dialog system.

34. A system for developing a dialog system, including:

a grammatical inference engine for generating grammars for said dialog system on the basis of respective sets of example input phrases for said dialog system, said sets including different numbers of example input phrases;

a learning curve estimator for determining, on the basis of said grammars, a relationship between a number of example input phrases and an estimated probability that an input phrase is correctly recognised; and

an estimator for generating an estimate of the number of example input phrases required to achieve a predetermined speech recognition accuracy when using said dialog system to allow a developer of said dialog system to determine development of said dialog system.

35. A system as claimed in claim 34, including a simulator for generating prompts for said dialog system and for receiving example input phrases in response to said prompts.

36. A system as claimed in claim 35, including a sampler for randomly selecting from said example input phrases to provide said sets.

37. A system as claimed in claim 36, wherein said sampler is adapted to randomly selecting from example input phrases to provide a test set of input phrases, and wherein said relationship is determined on the basis of respective portions of said test set predicted by said grammars.