Psycho-physical state sensitive voice dialogue system

• US 7,222,074 B2
• Filed: 06/20/2001
• Issued: 05/22/2007
• Est. Priority Date: 06/20/2001
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Zhou, Guojun
• Primary Examiner(s): Azad; Abul K.
• Application Number:
  US09/884,423
• Publication Number:
  US 20020198707A1
• Time in Patent Office: 2,162 Days
• Field of Search: 704/251, 704/254, 704/257, 704/260, 704/270, 704/275
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

An arrangement is provided to achieve automated voice responding dialogue that is psycho-physical state sensitive. The input speech data from a user is used to detect the psycho-physical state of the user. Spoken words are then recognized from the input speech data based on the detected psycho-physical state of the user. The recognized words form a transcription which is then used to generate a voice response that is appropriate according to the transcription and the psycho-physical state of the user.

Claims

1. A system, comprising:

a psycho-physical state detection mechanism for detecting a psycho-physical state of a user based on input speech data from the user;

a spoken dialogue mechanism for carrying on a dialogue with the user based on the psycho-physical stale of the user, detected by the psycho-physical detection mechanism from the input speech data from the user;

a speech understanding mechanism for understanding the input speech data from the user based on the psycho-physical state of the user to generate a literal meaning of the speech data; and

a voice response generation mechanism for generating a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

2. The system according to claim 1, wherein the speech understanding mechanism comprises:

at least one acoustic model for characterizing acoustic properties of the input speech data, each of the at least one acoustic model corresponding to some distinct characteristic related to a psycho-physical state of a speaker;

an acoustic model selection mechanism for selecting an acoustic model that is appropriate to according to the psycho-physical state detected by the psycho-physical state detection mechanism;

a speech recognizer for generating a transcription of spoken words recognized from the input speech data using the acoustic model selected by the acoustic model selection mechanism; and

a language understanding mechanism for interpreting the literal meaning of the input speech data based on the transcription.

3. The system according to claim 1, wherein the voice response generation mechanism comprises:

a natural language response generator for generating a response based on an understanding of the transcription, the response being generated appropriately according to the psycho-physical state of the user;

a prosodic pattern determining mechanism for determining a prosodic pattern to be applied to the response that is considered as appropriate according to the psycho-physical state; and

a text-to-speech engine for synthesizing the voice response based on the response and the prosodic pattern.

4. The system according to claim 1, wherein the psycho-physical state detection mechanism comprises:

an acoustic feature extractor for extracting acoustic features from the input speech data to generate at least one acoustic feature; and

a psycho-physical state classifier for classifying the input speech data into one or more psycho-physical states based on the at least one acoustic feature.

5. The system according to claim 4, further comprising:

at least one psycho-physical state model, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and characterizing the acoustic properties of the single psycho-physical state; and

an off-line training mechanism for establishing the at least one psycho-physical model based on labeled training speech data.

6. The system according to claim 1, further comprising a dialogue manager to control the dialogue flow.

7. A method, comprising:

receiving, by a psycho-physical state detection mechanism, input speech data from a user;

detecting a psycho-physical state of the user from the input speech data;

understanding, by a speech understanding mechanism, a literal meaning of spoken words recognized from the input speech data based on the psycho-physical state of the user, detected by the detecting; and

generating, by a voice response generation mechanism, a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

8. The method according to claim 7, wherein the detecting comprises:

extracting, by an acoustic feature extractor, at least one acoustic feature from the input speech data; and

classifying, by a psycho-physical state classifier and based on said at least one acoustic feature, the input speech data into the psycho-physical state according to at least one psycho-physical state model.

9. The method according o claim 8, further comprising:

receiving, by an off-line training mechanism, labeled training data, wherein each of the data items in the labeled training data is labeled by a psycho-physical state; and

building the at least one psycho-physical state model using the labeled training data, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and being established based on the data items in the labeled training data that have a label corresponding to the single psycho-physical state.

10. The method according to claim 7, wherein the understanding comprises:

selecting, by an acoustic model selection mechanism, an acoustic model, from at least one acoustic model, that is appropriate to according to the psycho-physical state, detected by the detecting, each of the at least one acoustic model corresponding to some distinct speech characteristic related to the psycho-physical state;

recognizing, by a speech recognizer, the spoken words from the input speech data using the acoustic model, selected by the selecting, to generate a transcription; and

interpreting, by a language understanding mechanism, the literal meaning of the spoken words based on the transcription.

11. The method according to claim 7, wherein the generating comprises:

constructing, by a natural language response generator, a natural language response based on an understanding of the transcription, the natural language response being constructed appropriately according to the psycho-physical state of the user;

determining, by a prosodic pattern determining mechanism, a prosodic pattern to be applied to said natural language response, wherein the prosodic pattern is considered to be appropriate according to the psycho-physical state; and

synthesizing, by a text-to-speech engine, the voice response based on the natural language response and the prosodic pattern.

12. A computer-readable medium encoded with a program, said program comprising instructions that when executed by a computer cause the computer to:

receive, by a psycho-physical state detection mechanism, input speech data from a user;

detect a psycho-physical state of the user from the input speech data;

understand, by a speech understanding mechanism, a literal meaning of spoken words recognized from the input speech data based on the psycho-physical state of the user; and

generate, by a voice response generation mechanism, a voice response to the user based on the literal meaning of the input speech data and the psycho-physical state of the user, wherein the voice response to the user is linguistically and acoustically adjusted according to the detected psycho-physical state of the user.

13. The medium according to claim 12, wherein the to detect comprises instructions that when executed by the computer cause the computer to:

extract, by a acoustic feature extractor, at least one acoustic feature from the input speech data; and

classify, by a psycho-physical state classifier and based on the at least one feature, the input speech data into the psycho-physical state according to at least one psycho-physical state model.

14. The medium according to claim 13, further comprising instructions that when executed by the computer cause the computer to:

receive, by an off-line training mechanism, labeled training data, wherein each of the data items in the labeled training data is labeled by a psycho-physical state; and

building the at least one psycho-physical state model using the labeled training data, each of the at least one psycho-physical state model corresponding to a single psycho-physical state and being established based on the data items in the labeled training data that have a label corresponding to the single psycho-physical state.

15. The medium according to claim 12, wherein the understanding comprises instructions that when executed by the computer cause the computer to:

select, by an acoustic model selection mechanism, an acoustic model, from at least one acoustic model, that is appropriate according to the psycho-physical state, each of the at least one acoustic model corresponding to some distinct speech characteristic related to a psycho-physical state;

reconize, by a speech recognizer, the spoken words from the input speech data using the acoustic model, to generate a transcription; and

interpret, by a language understanding mechanism, the literal meaning of the spoken words based on the transcription.

16. The medium according to claim 12, wherein the to generate comprises instructions that when executed by the computer cause the computer to:

construct, by a natural language response generator, a natural language response based on an understanding of the transcription, the natural language response being constructed appropriately according to the psycho-physical state of the user;

determine, by a prosodic pattern determining mechanism, a prosodic pattern to be applied to the natural language response, wherein the prosodic pattern is considered to be appropriate according to the psycho-physical state; and

synthesize, by a text-to-speech engine, the voice response based on the natural language response and the prosodic pattern.