System for automated testing of perceptual distortion of prompts from voice response systems

• US 6,477,492 B1
• Filed: 06/15/1999
• Issued: 11/05/2002
• Est. Priority Date: 06/15/1999
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Connor, Kevin J.
• Primary Examiner(s): Banks-Harold, Marsha D.
• Assistant Examiner(s): Storm, Donald L.
• Application Number:
  US09/333,778
• Time in Patent Office: 1,239 Days
• Field of Search: 704/236, 704/231, 704/200.1, 379/1.02
• US Class Current: 704/236
• CPC Class Codes: G10L 25/69 for evaluating synthetic or decoded voice signals

Abstract

A Perceptual Speech Distortion Metric (PSDM) generates perceptual distortion values for voice prompts received from a voice response system by comparing the received voice prompts with reference signals associated with the same states in the voice response system. The perceptual distortion values identify the voice prompts as either correct or incorrect responses to signal generator inputs and also quantify an amount of perceptual distortion in the voice prompts.

Claims

1. A system for testing a voice response system, comprising:

a signal generator generating inputs for the voice response system;

a signal recorder receiving voice prompts output by the voice response system in response to the inputs; and

a perceptual sound quality analyzer outputting perceptual distortion values by comparing the received voice prompts with reference voice prompts, the perceptual distortion values identifying the received voice prompts as either correct or incorrect responses to the signal generator inputs while also identifying different amounts of distortion in the received voice prompts.

2. A system according to claim 1 including a script file that generates sequences of inputs that traverses through different states in the voice response system.

3. A system according to claim 2 including a reference speech library that stores and accesses the reference voice prompts associated with the different states traversed in the voice response system.

4. A system according to claim 1 wherein the inputs generated by the signal generator are DTMF tones.

5. A system according to claim 1 including a telephone network coupling the signal generator and signal recorder to the voice response system.

6. A system according to claim 1 wherein the perceptual sound quality analyzer comprises a perceptual speech quality metric using a psychoacoustic model and a cognitive model to generate the perceptual distortion values.

7. A system according to claim 1 including a processor that identifies the received voice prompts according to the perceived distortion values as either incorrect, correct-unintelligible, or correct-intelligible.

8. A system according to claim 7 wherein the processor identifies different distortion levels for the voice prompts identified as correct.

9. A method for testing an audio response system, comprising:

generating inputs for the audio response system;

receiving audio prompts output from the audio response system in response to the generated inputs;

generating perceptual distortion values by comparing the received audio prompts with associated reference audio prompts;

using the perceptual distortion values to identify received audio prompts that correctly respond to the generated inputs; and

using the perceptual distortion values to quantify different amounts of perceptual distortion in the audio prompts.

10. A method according to claim 9 including generating a series of inputs that automatically progress through each state in the voice response system.

11. A method according to claim 10 including storing reference audio prompts associated with each state in the audio response system and comparing the stored reference audio prompts with the received audio prompts associated with the same audio response system state.

12. A method according to claim 9 wherein the input signals comprise DTMF tones.

13. A method according to claim 12 including transmitting the DTMF tones over a telephone network to the audio response system and receiving the audio prompts back over the same telephone network.

14. A method according to claim 12 including generating the same DTMF tones multiple times for different time durations.

15. A method according to claim 9 including generating the perceptual distortion values using a perceptual speech quality metric.

16. A method according to claim 9 including using the perceptual distortion values to automatically generate a report quantifying the received voice prompts as incorrect, correct-unintelligible, or correct-intelligible.

17. A method according to claim 9 including using the perceptual distortion values to identify the received voice prompts as correct, incorrect, or unintelligible and further quantify the correct voice prompts as having high distortion, medium distortion or low distortion.

18. A method according to claim 9 including for recording the audio prompts for an amount of time according to a current state of the audio response system.

19. A system for testing a voice response system;

comprising;

a voice quality test platform automatically initiating an off-hook condition and dialing a phone number over a telephone line;

an auto-attendant connected to the telephone line automatically answering the dialed phone number and establishing a connection with the test platform, the auto-attendant generating voice prompts in response to DTMF tones sent over the telephone line;

a signal generator on the test platform automatically generating sequences of DTMF tones associated with different states in the auto-attendant;

a signal recorder on the test platform recording voice prompts generated by the auto-attendant in response to the DTMF tones generated by the signal generator;

a reference speech library containing reference voice prompts associated with different states in the voice response system; and

a perceptual sound quality metric generating perceptual distortion values for the received voice prompts by comparing the received voice prompts with the reference voice prompts associated with the same voice response system states.

20. A system according to claim 19 wherein the perceptual distortion values indicate different levels of understandability of the voice prompts received at the test platform.

21. An electronic storage medium storing computer-readable program code executable for testing an audio response system, the computer-readable program code comprising:

code for generating inputs for the audio response system;

code for receiving audio prompts output from the audio response system in response to the generated inputs;

code for generating perceptual distortion values by comparing the received audio prompts with associated reference audio prompts;

code for using the perceptual distortion values to identify received audio prompts that correctly respond to the generated inputs; and

code for using the perceptual distortion values to quantify different amounts of perceptual distortion in the audio prompts.

22. An electronic storage medium according to claim 21 including code for generating a series of inputs that automatically progress through each state in the voice response system.

23. An electronic storage medium according to claim 22 including code for storing reference audio prompts associated with each state in the audio response system and code for comparing the stored reference audio prompts with the received audio prompts associated with the same audio response system state.

24. An electronic storage medium according to claim 21 wherein the input signals comprise DTMF tones.

25. An electronic storage medium according to claim 24 including code for transmitting the DTMF tones over a telephone network to the audio response system and code for receiving the audio prompts back over the same telephone network.

26. An electronic storage medium according to claim 24 including code for generating the same DTMF tones multiple times for different time durations.

27. An electronic storage medium according to claim 21 including code for generating the perceptual distortion values using a perceptual speech quality metric.

28. An electronic storage medium according to claim 21 including code for using the perceptual distortion values to automatically generate a report quantifying the received voice prompts as incorrect, correct-unintelligible, or correct-intelligible.

29. An electronic storage medium according to claim 21 including code for using the perceptual distortion values to identify the received voice prompts as correct, incorrect, or unintelligible and further quantify the correct voice prompts as having high distortion, medium distortion or low distortion.

30. An electronic storage medium according to claim 21 including code for recording the audio prompts for an amount of time according to a current state of the audio response system.

31. A system for testing an audio response system, comprising:

means for generating inputs for the audio response system;

means for receiving audio prompts output from the audio response system in response to the generated inputs;

means for generating perceptual distortion values by comparing the received audio prompts with associated reference audio prompts;

means for using the perceptual distortion values to identify received audio prompts that correctly respond to the generated inputs; and

means for using the perceptual distortion values to quantify different amounts of perceptual distortion in the audio prompts.

32. A system according to claim 31 including means for generating a series of inputs that automatically progress through each state in the voice response system.

33. A system according to claim 32 including means for storing reference audio prompts associated with each state in the audio response system and means for comparing the stored reference audio prompts with the received audio prompts associated with the same audio response system state.

34. A system according to claim 31 wherein the input signals comprise DTMF tones.

35. A system according to claim 34 including means for transmitting the DTMF tones over a telephone network to the audio response system and means for receiving the audio prompts back over the same telephone network.

36. A system according to claim 34 including means for generating the same DTMF tones multiple times for different time durations.

37. A system according to claim 31 including means for generating the perceptual distortion values using a perceptual speech quality metric.

38. A system according to claim 31 including means for using the perceptual distortion values to automatically generate a report quantifying the received voice prompts as incorrect, correct-unintelligible, or correct-intelligible.

39. A system according to claim 31 including means for using the perceptual distortion values to identify the received voice prompts as correct, incorrect, or unintelligible and further quantify the correct voice prompts as having high distortion, medium distortion or low distortion.

40. A system according to claim 31 including means for recording the audio prompts for an amount of time according to a current state of the audio response system.