Dialogue system incorporating unique speech to text conversion method for meaningful dialogue response

• US 10,347,244 B2
• Filed: 04/21/2017
• Issued: 07/09/2019
• Est. Priority Date: 04/21/2017
• Status: Active Grant

• Current Assignee: Go-Vivace Inc.
• Original Assignee: Go-Vivace Inc.
• Inventors: Goel, Nagendra Kumar, Sarma, Mousmita
• Primary Examiner(s): Blankenagel, Bryan S
• Application Number:
  US15/494,193
• Publication Number:
  US 20180308487A1
• Time in Patent Office: 809 Days
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/18 Artificial neural networks; Connectionist approaches
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

A real-time dialogue system that provides real-time transcription of the spoken text, with a sub-second delay by keeping track of word timings and word accuracy is provided. The system uses a grammar or a list of keywords to produce the transcripts by using a statistical language model. In addition, the system uses a deep neural network based I-vector system to constantly analyze the audio quality to assess and to identify additional metadata such as gender, language, accent, age, emotion and identity of an end user to enhance the response. The present invention provides a conversational dialogue system, to robustly identify certain specific user commands or intents, while otherwise allowing for a natural conversation, without switching between grammar based and natural language modes.

Claims

1. A system for providing real-time transcripts of spoken text, the system comprising:

a speech to text engine for converting an input speech of an end user into a text input, the text input comprises one or more sequences of recognized word strings or a word lattice in text form;

a semantic engine to receive the text input for producing one or more transcripts using a language model and extracting semantic meanings for said one or more transcripts;

wherein the semantic engine utilizes a grammar model and the language model to extract a meaning for said one or more transcripts;

an age and an emotion identification subsystem that detects the age and emotional state of the end user, said age and emotion identification subsystem comprises an end to end LSTM-RNN based DNN classifier;

said end to end classifier has two convolutional layers followed by two Network-in-Network (NIN) layers which performs the role of feature extraction from raw waveforms;

the end to end DNN classifier has 2 LSTM layers after the feature extraction layers followed by a softmax layer;

wherein the end to end DNN classifier has no separate acoustic feature extraction module at signal processing level and raw speech frames obtained from end user'"'"'s input speech waveform are directly presented to the input layer of the DNN.

2. The system of claim 1 further configured to identify and store additional metadata about the end user selected from the group consisting of age, gender, language, accent, and emotional state and search and verify the end user'"'"'s identity.

3. The system of claim 1, wherein the semantic engine, on receiving the one or more sequences of recognized word strings or the word lattice, extracts semantic meaning from the one or more sequences of recognized word strings, or the word lattice, and associates that with one or more action tags and entities known to the system.

4. The system of claim 3, wherein the semantic engine further comprises a semantic mapper that is configured to label the one or more action tags by including multiple possible meanings of a word from the one or more sequences of recognized word strings in a particular context.

5. The system of claim 1, further comprising:

a query generator configured to map one or more action tags to one or more database queries present in an interactive workflow logic module, wherein the interactive workflow logic module handles situations arising subsequent to a dialogue response; and

a natural language generator trained to receive the mapped one or more action tags and additional metadata about the end user selected from the group consisting of age, gender, language, accent, and emotional state, wherein the mapped one or more action tags and the additional metadata are being mapped into one or more logical sentences to form a spoken response of the dialogue system in real-time.

6. The system of claim 1, wherein the system utilizes GRXML or JSGF or ABNF format grammars to learn one or more action tags and one or more transcripts of the semantic engine, and also to enhance a vocabulary based on the grammar model and a vocabulary based on the language model.

7. The system of claim 1 further comprising a language/accent recognition subsystem that extracts acoustic features from the input speech of the end user to identify the language and/or accent of the end user, wherein said language/accent recognition subsystem comprises:

a speech activity detection module to detect speech activity;

a shifted delta cepstral (SDC) module to compute cepstral mean and variance normalization of the input speech and to produce SDC feature vectors;

an I-vector extractor module to receive SDC feature vectors and to produce I-vectors using a deep neural network-universal background model (DNN-UBM); and

a logistic regression classifier module to receive and classify the I-vectors in order to identify the end user'"'"'s language or accent.

8. The system of claim 1 wherein the system further comprises a speaker recognition (SR) subsystem that extracts acoustic features from the input speech of the end user to identify and verify the end user, said speaker recognition subsystem comprising:

a speech activity detection module to detect speech activity of the end user;

an MFCC computation module to calculate Mel Frequency Cepstral Coefficient along with cepstral mean and variance normalization of the speech activity and to generate feature vectors;

a keyword spotter module to provide keyword spotting based enrollment and verification of the end user;

a DNN-UBM based I-vector extractor module to produce I-vectors using a deep neural network-universal background model and a probabilistic linear discriminant analysis (PLDA) based classifier module to classify the identity of the end user.

9. The system of claim 1, wherein the age and emotion identification subsystem comprises a speech activity detection module to detect speech information and to generate an output for an MFCC computation module;

wherein said MFCC computation module performs analysis of the acoustic features followed by cepstral mean and variance normalization of the input speech to identify the age and emotion of the end user;

a DNN-UBM based I-vector extractor to generate I-vectors for the identified acoustic features; and

a logistic regression classifier to classify the I-vectors to identify the end user'"'"'s age and emotion.

10. The system of claim 1, wherein the emotion identification system provides provisions of both discrete and continuous classification of end user'"'"'s emotional level;

said discrete classification of end user'"'"'s emotion comprises anger, happiness, anxiety, neutral, boredom and sadness; and

the continuous classification of end user'"'"'s emotion provides rating of emotional levels in two continuous scales called valence and arousal.

11. A method for providing real-time transcripts of spoken text, the method comprising:

converting, by a speech to text engine, an input speech of an end user into a text input, the text input comprising one or more sequences of recognized word strings and confusions in text form; and

receiving, by a semantic engine, the text input for producing one or more transcripts using a language model and extracting semantic meanings for said one or more transcripts;

wherein the semantic engine utilizes a grammar model and the language model to extract meaning for said one or more transcripts

detecting the age and emotional state of the end user by an age and emotion identification subsystem, comprising an end to end LSTM-RNN based DNN classifier;

said end to end classifier has two convolutional layers followed by two Network-in-Network (NIN) layers which performs the role of feature extraction from raw waveforms;

the end to end DNN classifier has 2 LSTM layers after the feature extraction layers followed by a softmax layer; and

the end to end DNN classifier has no separate acoustic feature extraction module at signal processing level and raw speech frames obtained from end user'"'"'s input speech waveform are directly presented to the input layer of the DNN.

12. The method of claim 11, further comprising identifying and storing additional metadata about the speaker, selected from the group consisting of age, gender, accent and emotional state of the end user.

13. The method of claim 11, wherein the one or more sequences of recognized word strings are assigned with one or more action tags and entities.

14. The method of claim 11 further comprising the step of extracting acoustic features from the input speech of the end user to identify language and/or accent of the end user.

15. The method of claim 11 further comprising the step of extracting acoustic features from the input speech of the end user to identify and verify the end user.

16. The method of claim 11 further comprising the step of extracting acoustic and pitch features from the input speech to identify the age and emotion of the end user.