Detection and Use of Acoustic Signal Quality Indicators

• US 20090299741A1
• Filed: 04/03/2007
• Published: 12/03/2009
• Est. Priority Date: 04/03/2006
• Status: Active Grant

• Current Assignee: Promptu Systems Corporation
• Original Assignee: Promptu Systems Corporation
• Inventors: Printz, Harry, Pratt, Matthew, Gulati, Vikas, Chittar, Naren
• Granted Patent: US 8,521,537 B2
• Time in Patent Office: Days
• US Class Current: 704/233
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

A computer-driven device assists a user in self-regulating speech control of the device. The device processes an input signal representing human speech to compute acoustic signal quality indicators indicating conditions likely to be problematic to speech recognition, and advises the user of those conditions.

Claims

1. A computer-driven method to regulate a speaker'"'"'s issuance of speech-based commands, the method comprising operations of:

receiving an input signal representing audio content including speech of the speaker, the audio content occurring between a prescribed begin event and a prescribed end event;

processing at least a portion of the input signal to determine whether the processed portion of the input signal contains speech exhibiting one or more predetermined conditions capable of posing difficulty in successfully performing speech recognition upon the input signal;

where the processed portion of the input signal comprises one of;

all of the input signal, a part of the input signal, speech present in the input signal;

if the processing operation determines that the processed portion of the input signal contains speech exhibiting one or more of the predetermined conditions, then performing operations comprising;

issuing at least one human comprehensible speech quality alert corresponding to each exhibited predetermined condition.

2. The method of claim 1,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker began speaking before activation of the push-to-talk mechanism.

3. The method of claim 2, the operation of determining whether the speaker began speaking before activation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period after the begin event, and if so, concluding that the speaker began speaking before activation of the push-to-talk mechanism.

4. The method of claim 1,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism.

5. The method of claim 4, the operation of determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period before the end event, and if so, concluding that the speaker was still speaking upon deactivation of the push-to-talk mechanism.

6. The method of claim 3 or 5, where the operation of applying the predetermined speech detection process further comprises:

excluding breath noise from consideration as speech.

7. The method of claim 3 or 5, where the operation of applying the predetermined speech detection process further comprises:

excluding energy of a prescribed frequency range from consideration as speech.

8. The method of claim 1, the processing operation comprising:

determining whether the speaker was speaking too loudly relative to prescribed criteria.

9. The method of claim 8, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity exceeds a prescribed maximum, and if so, concluding that the speaker was speaking too loudly relative to prescribed criteria.

10. The method of claim 8, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal with acoustic energy above a first threshold;

computing a loud-frame-count by computing a total number of frames of the input signal with acoustic energy above a second threshold higher than the first threshold;

computing a ratio between the loud-frame-count and the non-silent-frame-count;

determining whether the ratio exceeds a third threshold, and if so, concluding that the speaker was speaking too loudly relative to the prescribed criteria.

11. The method of claim 1, the processing operation comprising:

determining whether the speaker was speaking too softly relative to prescribed criteria.

12. The method of claim 11, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity is less than a predetermined minimum, and if so, concluding that the speaker was speaking too softly relative to prescribed criteria.

13. The method of claim 11, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal that have acoustic energy above a first threshold;

computing a soft-frame-count by computing a total number of frames of the input signal that have acoustic energy less than a second threshold;

computing a ratio between the soft-frame-count and the non-silent-frame-count;

determining whether the ratio is greater than a third threshold, and if so, concluding that the speaker was speaking too softly relative to the prescribed criteria.

14. The method of claim 9 or 12, where the computing operation comprises one of:

analyzing raw audio stream;

analyzing mel frequency cepstral coefficients.

15. The method of claim 9 or 12,

the operations further comprising utilizing the input signal to compute a C0 element of a mel-frequency cepstral coefficient vector;

the computing operation utilizing the computed C0 element.

16. The method of claim 1,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too near to the speaker.

17. The method of claim 16, the operation of determining whether the microphone was held too near to the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy of all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy above a threshold, and if so, concluding that the microphone was held too near to the speaker.

18. The method of claim 1,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too far from the speaker.

19. The method of claim 18, the operation of determining whether the microphone was held too far from the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy in all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy below a given threshold, and if so, concluding that the microphone was held too far from the speaker.

20. The method of claim 1, the operation of processing at least a portion of the input signal comprising one or more of:

computing one or more acoustic signal quality indicators each indicating an affirmative output when the processed portion of the input signal contains speech exhibiting a corresponding one of the predetermined conditions;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions.

21. The method of claim 1, the operations further comprising:

receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

comparing the utterance confidence score with a prescribed threshold;

if the utterance confidence score exceeds the prescribed threshold, omitting performance of one or both of the following;

the operation of processing at least a portion of the input signal as to one or more of the predetermined conditions;

the operation of issuing the at least one human comprehensible speech quality alert.

22. The method of claim 21, where:

the comparing operation uses different prescribed thresholds corresponding to different predetermined conditions;

the omitting operation is only performed for each of the predetermined conditions whose corresponding prescribed threshold was not exceeded by the utterance confidence score.

23. The method of claim 1, where:

the operations further comprise receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

the operation of processing at least the portion of the input signal comprises;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions;

applying a predetermined function to each combination of;

computed acoustic signal quality indicator and utterance confidence score;

the operations further comprise;

if the input signal contains speech exhibiting a given one of the predetermined conditions, suspending issuance of the human comprehensible speech quality alert corresponding to the given predetermined condition if the application of the predetermined function to the utterance confidence score and the acoustic signal quality indicator corresponding to the given predetermined condition yields a predetermined result.

24. A computer readable medium storing one of (1) a program to perform operations for regulating a speaker'"'"'s issuance of speech-based commands, or (2) a program for installing a target program of machine-readable instructions on a computer, where the target program is executable to perform the operations for regulating a speaker'"'"'s issuance of speech-based commands, the operations comprising:

receiving an input signal representing audio content including speech of the speaker, the audio content occurring between a prescribed begin event and a prescribed end event;

processing at least a portion of the input signal to determine whether the processed portion of the input signal contains speech exhibiting one or more predetermined conditions capable of posing difficulty in successfully performing speech recognition upon the input signal;

where the processed portion of the input signal comprises one of;

all of the input signal, a part of the input signal, speech present in the input signal;

if the processing operation determines that the processed portion of the input signal contains speech exhibiting one or more of the predetermined conditions, then performing operations comprising;

issuing at least one human comprehensible speech quality alert corresponding to each exhibited predetermined condition.

25. The computer readable medium of claim 24,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker began speaking before activation of the push-to-talk mechanism.

26. The computer readable medium of claim 25, the operation of determining whether the speaker began speaking before activation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period after the begin event, and if so, concluding that the speaker began speaking before activation of the push-to-talk mechanism.

27. The computer readable medium of claim 24,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism.

28. The computer readable medium of claim 27, the operation of determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period before the end event, and if so, concluding that the speaker was still speaking upon deactivation of the push-to-talk mechanism.

29. The computer readable medium of claim 26 or 28, where the operation of applying the predetermined speech detection process further comprises:

excluding breath noise from consideration as speech.

30. The computer readable medium of claim 26 or 28, where the operation of applying the predetermined speech detection process further comprises:

excluding energy of a prescribed frequency range from consideration as speech.

31. The computer readable medium of claim 24, the processing operation comprising:

determining whether the speaker was speaking too loudly relative to prescribed criteria.

32. The computer readable medium of claim 31, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity exceeds a prescribed maximum, and if so, concluding that the speaker was speaking too loudly relative to prescribed criteria.

33. The computer readable medium of claim 31, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal with acoustic energy above a first threshold;

computing a loud-frame-count by computing a total number of frames of the input signal with acoustic energy above a second threshold higher than the first threshold;

computing a ratio between the loud-frame-count and the non-silent-frame-count;

determining whether the ratio exceeds a third threshold, and if so, concluding that the speaker was speaking too loudly relative to the prescribed criteria.

34. The computer readable medium of claim 24, the processing operation comprising:

determining whether the speaker was speaking too softly relative to prescribed criteria.

35. The computer readable medium of claim 34, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity is less than a predetermined minimum, and if so, concluding that the speaker was speaking too softly relative to prescribed criteria.

36. The computer readable medium of claim 34, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal that have acoustic energy above a first threshold;

computing a soft-frame-count by computing a total number of frames of the input signal that have acoustic energy less than a second threshold;

computing a ratio between the soft-frame-count and the non-silent-frame-count;

determining whether the ratio is greater than a third threshold, and if so, concluding that the speaker was speaking too softly relative to the prescribed criteria.

37. The computer readable medium of claim 32 or 35, where the computing operation comprises one of:

analyzing raw audio stream;

analyzing mel frequency cepstral coefficients.

38. The computer readable medium of claim 32 or 35,

the operations further comprising utilizing the input signal to compute a C0 element of a mel-frequency cepstral coefficient vector;

the computing operation utilizing the computed C0 element.

39. The computer readable medium of claim 24,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too near to the speaker.

40. The computer readable medium of claim 39, the operation of determining whether the microphone was held too near to the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy of all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy above a threshold, and if so, concluding that the microphone was held too near to the speaker.

41. The computer readable medium of claim 24,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too far from the speaker.

42. The computer readable medium of claim 41, the operation of determining whether the microphone was held too far from the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy in all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy below a given threshold, and if so, concluding that the microphone was held too far from the speaker.

43. The computer readable medium of claim 24, the operation of processing at least a portion of the input signal comprising one or more of:

computing one or more acoustic signal quality indicators each indicating an affirmative output when the processed portion of the input signal contains speech exhibiting a corresponding one of the predetermined conditions;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions.

44. The computer readable medium of claim 24, the operations further comprising:

receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

comparing the utterance confidence score with a prescribed threshold;

if the utterance confidence score exceeds the prescribed threshold, omitting performance of one or both of the following;

the operation of processing at least a portion of the input signal as to one or more of the predetermined conditions;

the operation of issuing the at least one human comprehensible speech quality alert.

45. The computer readable medium of claim 44, where:

the comparing operation uses different prescribed thresholds corresponding to different predetermined conditions;

the omitting operation is only performed for each of the predetermined conditions whose corresponding prescribed threshold was not exceeded by the utterance confidence score.

46. The computer readable medium of claim 24, where:

the operations further comprise receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

the operation of processing at least the portion of the input signal comprises;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions;

applying a predetermined function to each combination of;

computed acoustic signal quality indicator and utterance confidence score;

the operations further comprise;

if the input signal contains speech exhibiting a given one of the predetermined conditions, suspending issuance of the human comprehensible speech quality alert corresponding to the given predetermined condition if the application of the predetermined function to the utterance confidence score and the acoustic signal quality indicator corresponding to the given predetermined condition yields a predetermined result.

47. Circuitry of multiple interconnected electrically conductive elements configured to perform operations to regulate a speaker'"'"'s issuance of speech-based commands, the operations comprising:

receiving an input signal representing audio content including speech of the speaker, the audio content occurring between a prescribed begin event and a prescribed end event;

processing at least a portion of the input signal to determine whether the processed portion of the input signal contains speech exhibiting one or more predetermined conditions capable of posing difficulty in successfully performing speech recognition upon the input signal;

where the processed portion of the input signal comprises one of;

all of the input signal, a part of the input signal, speech present in the input signal;

if the processing operation determines that the processed portion of the input signal contains speech exhibiting one or more of the predetermined conditions, then performing operations comprising;

issuing at least one human comprehensible speech quality alert corresponding to each exhibited predetermined condition.

48. The circuitry of claim 47,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker began speaking before activation of the push-to-talk mechanism.

49. The circuitry of claim 48, the operation of determining whether the speaker began speaking before activation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period after the begin event, and if so, concluding that the speaker began speaking before activation of the push-to-talk mechanism.

50. The circuitry of claim 47,

the operations further comprising receiving activation and deactivation of a push-to-talk mechanism;

the activation and the deactivation defining the begin event and end event, respectively;

the processing operation comprising;

determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism.

51. The circuitry of claim 50, the operation of determining whether the speaker was still speaking upon deactivation of the push-to-talk mechanism comprising:

applying a predetermined speech detection process to the input signal;

determining whether the input signal contains detected speech occurring within a predetermined time period before the end event, and if so, concluding that the speaker was still speaking upon deactivation of the push-to-talk mechanism.

52. The circuitry of claim 49 or 51, where the operation of applying the predetermined speech detection process further comprises:

excluding breath noise from consideration as speech.

53. The circuitry of claim 49 or 51, where the operation of applying the predetermined speech detection process further comprises:

excluding energy of a prescribed frequency range from consideration as speech.

54. The circuitry of claim 47, the processing operation comprising:

determining whether the speaker was speaking too loudly relative to prescribed criteria.

55. The circuitry of claim 54, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity exceeds a prescribed maximum, and if so, concluding that the speaker was speaking too loudly relative to prescribed criteria.

56. The circuitry of claim 54, the operation of determining whether the speaker was speaking too loudly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal with acoustic energy above a first threshold;

computing a loud-frame-count by computing a total number of frames of the input signal with acoustic energy above a second threshold higher than the first threshold;

computing a ratio between the loud-frame-count and the non-silent-frame-count;

determining whether the ratio exceeds a third threshold, and if so, concluding that the speaker was speaking too loudly relative to the prescribed criteria.

57. The circuitry of claim 47, the processing operation comprising:

determining whether the speaker was speaking too softly relative to prescribed criteria.

58. The circuitry of claim 57, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing one of the following quantities;

acoustic energy of the input signal, acoustic energy of speech of the input signal;

determining whether the computed quantity is less than a predetermined minimum, and if so, concluding that the speaker was speaking too softly relative to prescribed criteria.

59. The circuitry of claim 57, the operation of determining whether the speaker was speaking too softly relative to prescribed criteria comprising:

computing a non-silent-frame-count by computing a total number of frames of the input signal that have acoustic energy above a first threshold;

computing a soft-frame-count by computing a total number of frames of the input signal that have acoustic energy less than a second threshold;

computing a ratio between the soft-frame-count and the non-silent-frame-count;

determining whether the ratio is greater than a third threshold, and if so, concluding that the speaker was speaking too softly relative to the prescribed criteria.

60. The circuitry of claim 55 or 58, where the computing operation comprises one of:

analyzing raw audio stream;

analyzing mel frequency cepstral coefficients.

61. The circuitry of claim 55 or 58,

the operations further comprising utilizing the input signal to compute a C0 element of a mel-frequency cepstral coefficient vector;

the computing operation utilizing the computed C0 element.

62. The circuitry of claim 47,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too near to the speaker.

63. The circuitry of claim 62, the operation of determining whether the microphone was held too near to the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy of all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy above a threshold, and if so, concluding that the microphone was held too near to the speaker.

64. The circuitry of claim 47,

where the input signal originated from a microphone;

the processing operation comprising;

determining whether the microphone was held too far from the speaker.

65. The circuitry of claim 64, the operation of determining whether the microphone was held too far from the speaker comprising:

comparing amounts of low-frequency acoustic energy in the input signal with amounts of acoustic energy in all frequencies in the input signal;

determining if the comparing operation reveals low-frequency acoustic energy below a given threshold, and if so, concluding that the microphone was held too far from the speaker.

66. The circuitry of claim 47, the operation of processing at least a portion of the input signal comprising one or more of:

computing one or more acoustic signal quality indicators each indicating an affirmative output when the processed portion of the input signal contains speech exhibiting a corresponding one of the predetermined conditions;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions.

67. The circuitry of claim 47, the operations further comprising:

receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

comparing the utterance confidence score with a prescribed threshold;

if the utterance confidence score exceeds the prescribed threshold, omitting performance of one or both of the following;

the operation of processing at least a portion of the input signal as to one or more of the predetermined conditions;

the operation of issuing the at least one human comprehensible speech quality alert.

68. The circuitry of claim 67, where:

the comparing operation uses different prescribed thresholds corresponding to different predetermined conditions;

the omitting operation is only performed for each of the predetermined conditions whose corresponding prescribed threshold was not exceeded by the utterance confidence score.

69. The circuitry of claim 47, where:

the operations further comprise receiving or calculating an utterance confidence score representing a likelihood that results of speech recognition performed upon the input signal are correct;

the operation of processing at least the portion of the input signal comprises;

computing one or more acoustic signal quality indicators each functionally related to a degree of presence of a corresponding one of the predetermined conditions;

applying a predetermined function to each combination of;

computed acoustic signal quality indicator and utterance confidence score;

the operations further comprise;

if the input signal contains speech exhibiting a given one of the predetermined conditions, suspending issuance of the human comprehensible speech quality alert corresponding to the given predetermined condition if the application of the predetermined function to the utterance confidence score and the acoustic signal quality indicator corresponding to the given predetermined condition yields a predetermined result.

70. An apparatus, comprising:

an input device including a microphone;

at least one output device;

a speech recognition unit;

a speech regulation unit programmed to perform operations to regulate a speaker'"'"'s issuance of speech-based commands, the operations comprising;

receiving an input signal representing audio content including speech of the speaker, the audio content occurring between a prescribed begin event and a prescribed end event;

processing at least a portion of the input signal to determine whether the processed portion of the input signal contains speech exhibiting one or more predetermined conditions capable of posing difficulty in successfully performing speech recognition upon the input signal;

where the processed portion of the input signal comprises one of;

all of the input signal, a part of the input signal, speech present in the input signal;

if the processing operation determines that the processed portion of the input signal contains speech exhibiting one or more of the predetermined conditions, then performing operations comprising;

causing the output device to issue at least one human comprehensible speech quality alert corresponding to each exhibited predetermined condition.

71. The apparatus of claim 70, the apparatus integrated into a radiotelephone.

72. The apparatus of claim 70, the apparatus integrated into a remote control device.

73. An apparatus, comprising:

first means for receiving speech by a speaker;

second means for providing human-comprehensible output;

third means for performing speech recognition;

speech regulation means for regulating a speaker'"'"'s issuance of speech-based commands by;

receiving an input signal representing audio content including speech of the speaker, the audio content occurring between a prescribed begin event and a prescribed end event;

processing at least a portion of the input signal to determine whether the processed portion of the input signal contains speech exhibiting one or more predetermined conditions capable of posing difficulty in successfully performing speech recognition upon the input signal;

where the processed portion of the input signal comprises one of;

all of the input signal, a part of the input signal, speech present in the input signal;

if the processing operation determines that the processed portion of the input signal contains speech exhibiting one or more of the predetermined conditions, then performing operations comprising;

causing the second means to issue at least one human comprehensible speech quality alert corresponding to each exhibited predetermined condition.