DIALOG DEVICE WITH DIALOG SUPPORT GENERATED USING A MIXTURE OF LANGUAGE MODELS COMBINED USING A RECURRENT NEURAL NETWORK

• US 20170316775A1
• Filed: 04/27/2016
• Published: 11/02/2017
• Est. Priority Date: 04/27/2016
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Le, Phong, Dymetman, Marc, Renders, Jean-Michel
• Granted Patent: US 10,431,205 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A dialog device comprises a natural language interfacing device (chat interface or a telephonic device), and a natural language output device (the chat interface, a display device, or a speech synthesizer outputting to the telephonic device). A computer stores natural language dialog conducted via the interfacing device and constructs a current utterance word-by-word. Each word is chosen by applying a plurality of language models to a context comprising concatenation of the stored dialog and the current utterance thus far. Each language model outputs a distribution over the words of a vocabulary. A recurrent neural network (RNN) is applied to the distributions to generate a mixture distribution. The next word is chosen using the mixture distribution. The output device outputs the current natural language utterance after it has been constructed by the computer.

Claims

1. A dialog device comprising:

a natural language interfacing device comprising a chat interface or a telephonic device;

a natural language output device comprising the chat interface, a display device, or a speech synthesizer outputting speech to the telephonic device; and

a computer programmed to store natural language dialog conducted via the natural language interfacing device and to construct a current natural language utterance word-by-word with each word of the current natural language utterance being chosen by operations including;

applying a plurality of language models to a context comprising a concatenation of the stored natural language dialog and the current natural language utterance up to but not including the word being chosen to output, for each applied language model, a distribution over the words of a vocabulary;

normalizing the distributions output by the plurality of language models to generate corresponding normalized distributions;

applying a recurrent neural network (RNN) to the normalized distributions to generate a mixture distribution; and

choosing the next word using the mixture distribution;

wherein the natural language output device is configured to output the current natural language utterance after it has been constructed by the computer.

2. The dialog device of claim 1 wherein:

the computer is programmed to choose the next word by sampling the mixture distribution and is further programmed to repeat construction of a current natural language utterance a plurality of times to construct a plurality of current utterance candidates, and

the natural language output device comprises the display device configured to output a list of the current utterance candidates on the display device.

3. The dialog device of claim 1 wherein:

the natural language interfacing device comprises the chat interface; and

the natural language output device comprises the chat interface.

4. The dialog device of claim 1 wherein:

the natural language interfacing device comprises the telephonic device; and

the natural language output device comprises the speech synthesizer.

5. The dialog device of claim 1 wherein the RNN is a Long Short-Term Memory (LSTM) model or a Gated Recurrent Unit (GRU).

6. The dialog device of claim 1 wherein the normalizing comprises applying a softmax function to generate the normalized distributions.

7. The dialog device of claim 1 wherein the plurality of language models includes two language models and the normalizing comprises applying a sigmoid function to generate the normalized distributions.

8. The dialog device of claim 1 wherein the plurality of language models includes at least one QA language model configured to provide a distribution of answers obtained from a knowledge base (KB) over questions directed to the KB.

9. The dialog device of claim 8 wherein the plurality of language models further includes at least one language model that is not configured to provide answers obtained from a KB.

10. The dialog device of claim 8 wherein the computer is programmed to apply the QA language model to the context by operations including:

computing a first probability distribution over a set of first KB question parameters for the context;

computing a second probability distribution over a set of second KB question parameters for the context;

computing a question probability distribution as a product of the first probability distribution and the second probability distribution;

identifying a question posed by the context using the question probability distribution; and

outputting, for the QA language model, a distribution of answers obtained from the KB over the question probability distribution.

11. A dialog method comprising:

conducting a natural language dialog using a chat interface or a telephonic device;

while conducting the natural language dialog, constructing a current natural language utterance word-by-word using a computer programmed to choose each word of the current natural language utterance by operations including;

applying a plurality of language models to a context comprising a concatenation of the natural language dialog and the current natural language utterance up to but not including the word being chosen,

applying a recurrent neural network (RNN) to word distributions output by the applied plurality of language models to generate a mixture distribution, and

choosing the next word using the mixture distribution; and

outputting the constructed current natural language utterance via one of the chat interface, a display device, and a speech synthesizer outputting speech to the telephonic device.

12. The dialog method of claim 11 wherein:

the choosing of the next word is by sampling the mixture distribution,

the constructing is repeated a plurality of times to construct a plurality of current natural language utterance candidates, and

the outputting comprises outputting a list of the current natural language utterance candidates via a display device.

13. The dialog method of claim 11 wherein the RNN is a Long Short-Term Memory (LSTM) model.

14. The dialog method of claim 11 wherein the computer is programmed to choose each word of the current natural language utterance by operations further including:

normalizing the word distributions output by the applied plurality of language models using a softmax or sigmoid function.

15. The dialog method of claim 11 wherein the plurality of language models includes at least one QA language model configured to provide a distribution of answers obtained from a knowledge base (KB) over questions directed to the KB.

16. The dialog method of claim 15 wherein the plurality of language models further includes at least one language model not configured to provide answers obtained from a KB.

17. The dialog method of claim 15 wherein the applying of the QA language model to the context includes:

computing a first probability distribution over a set of first KB question parameters for the context;

computing a second probability distribution over a set of second KB question parameters for the context;

computing a question probability distribution as a product of the first probability distribution and the second probability distribution;

identifying a question posed by the context using the question probability distribution; and

outputting, for the QA language model, a distribution of answers obtained from the KB over the question probability distribution.

18. A non-transitory storage medium storing instructions readable and executable by a computer to construct a current natural language utterance for continuing a natural language dialog by a method in which each word of the current natural language utterance is chosen by operations including:

applying a plurality of language models to a context comprising a concatenation of the natural language dialog and the current natural language utterance up to but not including the word being chosen;

normalizing word distributions output by the plurality of language models to generate corresponding normalized word distributions;

applying a recurrent neural network (RNN) to the normalized distributions to generate a mixture distribution; and

choosing the next word using the mixture distribution.

19. The non-transitory storage medium of claim 18 wherein the RNN is a Long Short-Term Memory (LSTM) model or a Gated Recurrent Unit (GRU).

20. The non-transitory storage medium of claim 18 wherein the plurality of language models includes:

at least one QA language model configured to provide a distribution of answers contained in a knowledge base (KB); and

at least one language model that is not configured to provide a distribution of answers contained in a KB.