Task-independent conversational systems

• US 10,339,919 B1
• Filed: 04/20/2018
• Issued: 07/02/2019
• Est. Priority Date: 04/20/2018
• Status: Active Grant

• Current Assignee: Botbotbotbot, Inc.
• Original Assignee: Botbotbotbot, Inc.
• Inventors: Raux, Antoine
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US15/959,109
• Time in Patent Office: 438 Days
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating responses using task-independent conversational systems.

Claims

1. A method comprising:

obtaining multi-task training data, the multi-task training data comprising a plurality of sequences of conversational inputs, wherein each sequence corresponds to a respective task, and the multi-task training data comprises sequences corresponding to multiple different tasks, wherein the multi-task training data comprises a respective reward and a respective conversational output for each conversational input, and wherein the respective rewards are generated based on one or more observable metrics that relate to a quality of conversational outputs generated by the conversational machine learning model; and

training a conversational machine learning model on the multi-task training data to determine trained values of the parameters of the conversational machine learning model, wherein the conversational machine learning model is configured to receive as input a conversational input and to generate as output a conversational output that defines a response to a user that is independent of a task being performed when the conversational input was generated, wherein training the conversational machine learning model comprises training the conversational machine learning model using the respective rewards using reinforcement learning.

2. The method of claim 1, wherein each reward is based on whether a task was successfully completed after the corresponding conversational output was generated.

3. The method of claim 1, wherein each reward is based on a length of a conversation after the corresponding conversational output was generated.

4. The method of claim 1, further comprising:

obtaining task-specific training data, wherein the task-specific training data includes multiple sequences of task inputs each generated during performance of a particular task; and

training a task machine learning model on the task-specific training data, wherein the task-specific machine learning model is configured to receive as input a task input and to generate as output a task output that is specific to the particular task.

5. The method of claim 4, wherein the task-specific training data further comprises a respective ground truth task output for each task input, and wherein training the task machine learning model comprises training the task machine learning model on the task-specific training data using supervised learning.

6. The method of claim 4, wherein the task output defines whether the conversational machine learning model should be invoked.

7. The method of claim 1, wherein the reinforcement learning technique is a policy gradient based technique.

8. The method of claim 1, wherein the conversational input is an input that removes task-dependent aspects of a state of a conversation between a current user and a computer-implemented dialogue system while maintaining a conversational context of the conversation.

9. The method of claim 1, wherein the conversational output comprises respective scores for each intent-slot value combination in a set of conversational intent-slot value combinations.

10. The method of claim 1, wherein the conversational output is a feature vector.

11. A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining multi-task training data, the multi-task training data comprising a plurality of sequences of conversational inputs, wherein each sequence corresponds to a respective task, and the multi-task training data comprises sequences corresponding to multiple different tasks, wherein the multi-task training data comprises a respective reward and a respective conversational output for each conversational input, and wherein the respective rewards are generated based on one or more observable metrics that relate to a quality of conversational outputs generated by the conversational machine learning model; and

training a conversational machine learning model on the multi-task training data to determine trained values of the parameters of the conversational machine learning model, wherein the conversational machine learning model is configured to receive as input a conversational input and to generate as output a conversational output that defines a response to a user that is independent of a task being performed when the conversational input was generated, wherein training the conversational machine learning model comprises training the conversational machine learning model using the respective rewards using reinforcement learning.

12. The system of claim 11, wherein each reward is based on whether a task was successfully completed after the corresponding conversational output was generated.

13. The system of claim 11, wherein each reward is based on a length of a conversation after the corresponding conversational output was generated.

14. The system of claim 11, the operations further comprising:

obtaining task-specific training data, wherein the task-specific training data includes multiple sequences of task inputs each generated during performance of a particular task; and

training a task machine learning model on the task-specific training data, wherein the task-specific machine learning model is configured to receive as input a task input and to generate as output a task output that is specific to the particular task.

15. The system of claim 14, wherein the task-specific training data further comprises a respective ground truth task output for each task input, and wherein training the task machine learning model comprises training the task machine learning model on the task-specific training data using supervised learning.

16. The system of claim 14, wherein the task output defines whether the conversational machine learning model should be invoked.

17. The system of claim 11, wherein the reinforcement learning technique is a policy gradient based technique.

18. The system of claim 11, wherein the conversational input is an input that removes task-dependent aspects of a state of a conversation between a current user and a computer-implemented dialogue system while maintaining a conversational context of the conversation.

19. The system of claim 11, wherein the conversational output comprises respective scores for each intent-slot value combination in a set of conversational intent-slot value combinations.

20. The system of claim 11, wherein the conversational output is a feature vector.

21. One or more non-transitory computer readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining multi-task training data, the multi-task training data comprising a plurality of sequences of conversational inputs, wherein each sequence corresponds to a respective task, and the multi-task training data comprises sequences corresponding to multiple different tasks, wherein the multi-task training data comprises a respective reward and a respective conversational output for each conversational input, and wherein the respective rewards are generated based on one or more observable metrics that relate to a quality of conversational outputs generated by the conversational machine learning model; and

training a conversational machine learning model on the multi-task training data to determine trained values of the parameters of the conversational machine learning model, wherein the conversational machine learning model is configured to receive as input a conversational input and to generate as output a conversational output that defines a response to a user that is independent of a task being performed when the conversational input was generated, wherein training the conversational machine learning model comprises training the conversational machine learning model using the respective rewards using reinforcement learning.

22. The non-transitory computer readable storage media of claim 21, wherein each reward is based on whether a task was successfully completed after the corresponding conversational output was generated.

23. The non-transitory computer readable storage media of claim 21, wherein each reward is based on a length of a conversation after the corresponding conversational output was generated.

24. The non-transitory computer readable storage media of claim 21, the operations further comprising:

obtaining task-specific training data, wherein the task-specific training data includes multiple sequences of task inputs each generated during performance of a particular task; and

training a task machine learning model on the task-specific training data, wherein the task-specific machine learning model is configured to receive as input a task input and to generate as output a task output that is specific to the particular task.

25. The non-transitory computer readable storage media of claim 24, wherein the task-specific training data further comprises a respective ground truth task output for each task input, and wherein training the task machine learning model comprises training the task machine learning model on the task-specific training data using supervised learning.

26. The non-transitory computer readable storage media of claim 24, wherein the task output defines whether the conversational machine learning model should be invoked.

27. The non-transitory computer readable storage media of claim 21, wherein the reinforcement learning technique is a policy gradient based technique.

28. The non-transitory computer readable storage media of claim 21, wherein the conversational input is an input that removes task-dependent aspects of a state of a conversation between a current user and a computer-implemented dialogue system while maintaining a conversational context of the conversation.

29. The non-transitory computer readable storage media of claim 21, wherein the conversational output comprises respective scores for each intent-slot value combination in a set of conversational intent-slot value combinations.

30. The non-transitory computer readable storage media of claim 21, wherein the conversational output is a feature vector.