Natural language generation based on user speech style

• US 10,380,992 B2
• Filed: 11/13/2017
• Issued: 08/13/2019
• Est. Priority Date: 11/13/2017
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Talwar, Gaurav, Zhao, Xu Fang
• Primary Examiner(s): Albertalli, Brian L
• Application Number:
  US15/811,204
• Publication Number:
  US 20190147849A1
• Time in Patent Office: 638 Days
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

A system and method of generating a natural language generation (NLG) output, wherein the method includes: receiving speech signals from a user at a microphone of a client device; determining a requested communication goal and at least one inputted communication value based on the received speech signals; determining to use a static natural language generation (NLG) template or a dynamic NLG template to generate an NLG output, wherein the determination of whether to use a static NLG template or a dynamic NLG template is made using a neural network NLG template selection process; selecting an NLG template after the determination of whether to use a static NLG template or a dynamic NLG template; and generating an NLG output based on the selected NLG template.

Claims

1. A method of generating a natural language generation (NLG) output, wherein the method comprises:

receiving speech signals from a user at a microphone of a client device;

determining a requested communication goal and at least one inputted communication value based on the received speech signals;

determining to use a static natural language generation (NLG) template or a dynamic NLG template to generate an NLG output, wherein the determination of whether to use a static NLG template or a dynamic NLG template is made using a neural network NLG template selection process;

selecting an NLG template after the determination of whether to use a static NLG template or a dynamic NLG template; and

generating an NLG output based on the selected NLG template.

2. The method of claim 1, further comprising the step of identifying a user or a likely user that transmitted the speech signals.

3. The method of claim 2, wherein the neural network NLG template selection process is carried out based on a history of the identified user or likely user and/or a profile of the identified user or likely user.

4. The method of claim 3, wherein the selected NLG template is selected based on the history of the identified user or likely user and/or the profile of the identified user or likely user.

5. The method of claim 4, further comprising the step of identifying a style based on the received speech signals, the history of the identified user or likely user, and/or the profile of the identified user or likely user, and wherein the identified style is used in the neural network NLG template selection process.

6. The method of claim 1, further comprising the step of, when it is determined that a dynamic NLG template is to be used to generate the NLG output, then generating the dynamic NLG template using an NLG template generation process.

7. The method of claim 6, wherein the dynamic NLG template is generated based on the communication goal and one or more obtained communication values.

8. The method of claim 7, further comprising the step of identifying a user or a likely user that transmitted the speech signals, wherein the dynamic NLG template is generated based on user information that is associated with the user or likely user, and wherein the user information includes either or both of user history information or user profile information.

9. The method of claim 8, wherein the NLG template generation process is a rule-based NLG template generation process.

10. The method of claim 8, wherein the NLG template generation process is a neural network NLG template generation process.

11. The method of claim 1, further comprising the steps of:

sending a communication value request to a remote server, wherein the communication value request is a request to obtain responsive communication values that are to be used in the NLG output and that are based on the communication goal and inputted communication values; and

receiving a communication value response from the remote server, wherein the communication value response includes responsive communication values that are to be used in generating the NLG output.

12. The method of claim 1, further comprising the step of adapting the neural network NLG template selection process based on previous iterations of the neural network NLG template selection process.

13. The method of claim 12, wherein the adapting step further comprises adapting the neural network NLG template selection process based on a set of inputs, the NLG output, and a measured success of the neural network NLG template selection process, wherein the set of inputs includes the communication goal, the inputted communication values, and information pertaining to a user or a likely user that transmitted the speech signals.

14. A method of generating a natural language generation (NLG) output, wherein the method comprises:

receiving speech signals from a user at a microphone of a client device;

identifying a user or a likely user that transmitted the speech signals;

determining a communication goal and at least one inputted communication value based on the received speech signals;

determining whether to use a static NLG template or dynamic NLG template for use in generating an NLG output, wherein the determination of whether to use a static NLG template or dynamic NLG template is made using a neural network NLG template selection process, wherein the neural network NLG template selection process uses an artificial neural network to resolve a set of inputs to select whether to use a static NLG template or dynamic NLG template for use in generating an NLG output, and wherein the set of inputs include the communication goal, the inputted communication values, and either a user history that is associated with the user or a user profile that is associated with the user;

where it is determined to use a static NLG template, then selecting a static NLG template;

when it is determined to use a dynamic NLG template, then generating the dynamic NLG template; and

generating the NLG output using the selected NLG template.

15. The method of claim 14, wherein the set of inputs includes both the user history that is associated with the user and the user profile that is associated with the user.

16. The method of claim 14, wherein the neural network NLG template selection process includes a set of potential outputs that include a first NLG template output value and a second NLG template output value, wherein the first NLG template output value corresponds to a probability that a static NLG template will be used in generating the NLG output, and wherein the second NLG template output value corresponds to a probability that a dynamic NLG template will be used in generating the NLG output.

17. The method of claim 14, wherein the identifying step includes using the received speech signals to identify the user and wherein it is determined to use a static NLG template or a dynamic NLG template based on an identified style, emotion, and/or dialect of the received speech signals.

18. The method of claim 17, wherein the dynamic NLG template is generated and/or the static NLG template is selected based on the identified style, emotion, and/or dialect of the received speech signals.

19. The method of claim 14, wherein the dynamic NLG template is generated using a machine learning NLG template generation process, and wherein the machine learning NLG template generation process uses a different machine learning technique than that which is used in the neural network NLG template selection process.

20. The method of claim 14, wherein the dynamic NLG template is generated using a rule-based approach that includes a predefined set of rules that are followed in generating the dynamic NLG template.