Using textual input and user state information to generate reply content to present in response to the textual input

• US 10,192,551 B2
• Filed: 08/30/2016
• Issued: 01/29/2019
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Carbune, Victor, Keysers, Daniel, Deselaers, Thomas
• Primary Examiner(s): Singh, Satwant
• Application Number:
  US15/252,049
• Publication Number:
  US 20180061400A1
• Time in Patent Office: 882 Days
• Field of Search: None
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Methods, apparatus, and computer readable media related to receiving textual input of a user during a dialog between the user and an automated assistant (and optionally one or more additional users), and generating responsive reply content based on the textual input and based on user state information. The reply content is provided for inclusion in the dialog. In some implementations, the reply content is provided as a reply, by the automated assistant, to the user'"'"'s textual input and may optionally be automatically incorporated in the dialog between the user and the automated assistant. In some implementations, the reply content is suggested by the automated assistant for inclusion in the dialog and is only included in the dialog in response to further user interface input.

Claims

1. A method implemented by one or more processors, comprising:

receiving textual input, the textual input being based on user interface input generated by a user via one or more user interface input devices of a computing device of the user,

wherein the user interface input is generated by the user as part of a dialog that includes the user and an automated assistant implemented by one or more of the processors;

determining user state information for the user, wherein the user state information identifies a state of the user, is in addition to the textual input, and is based on sensor data generated by the computing device or an additional computing device of the user;

generating reply content based on both the textual input and the user state information, wherein the reply content includes at least one selectable graphical element that, when selected via further user interface input, causes the computing device of the user to present additional content to the user, and wherein generating the reply content based on both the textual input and the user state information comprises;

issuing a search of one or more databases based on both the textual input and the user state information,

receiving one or more search results in response to issuing the search, and

incorporating one or more of the search results into the reply content, wherein the selectable graphical element is associated with one of the search results; and

providing the reply content in response to the user interface input, wherein the reply content is provided for inclusion in the dialog in response to the textual input, and wherein the reply content is provided for presentation via one or more user interface output devices.

2. The method of claim 1, wherein a transcript of the dialog between the user and the automated assistant is displayed in a graphical user interface rendered by one of the user interface output devices of the computing device of the user and wherein providing the reply content comprises:

incorporating the reply content into the transcript for display along with previous content of the dialog.

3. The method of claim 2, wherein incorporating the reply content into the transcript comprises:

transmitting, via one or more network interfaces, a command to the computing device, wherein the command causes the computing device to incorporate the reply content into the transcript.

4. The method of claim 1, wherein issuing the search based on both the textual input and the user state information comprises:

determining a search parameter based on the textual input;

determining an additional search parameter based on the user state information, wherein the additional search parameter is either;

a search term used to identify responsive content on which the search results are based, or

a ranking parameter used to rank the search results.

5. The method of claim 1, wherein selection of the selectable graphical element causes the computing device of the user to establish a network connection with a computing device of an additional user and wherein the additional content indicates initiation of a new dialog with the additional user.

6. The method of claim 1, wherein determining the user state information comprises:

identifying a plurality of user state indicators based on the sensor data;

applying the user state indicators as input to at least one user state model stored in one or more computer readable media; and

generating, over the at least one user state model and based on the input, output that indicates the user state information.

7. The method of claim 6, wherein the at least one user state model includes a trained sentiment classifier.

8. The method of claim 1, wherein the sensor data includes user interface sensor data that is based on the user interface input and wherein determining the user state information includes:

determining the user state information based on the user interface input sensor data.

9. The method of claim 1, wherein the dialog includes an additional user and further comprising:

determining additional user state information for the additional user;

wherein generating the reply content is further based on the additional user state information.

10. The method of claim 1, wherein generating the reply content based on the user state information comprises:

determining a style feature of the reply content based on the user state information.

11. The method of claim 1, wherein the sensor data on which the user state information is based is from a plurality of sensors of the computing device or the additional computing device.

12. A method implemented by one or more processors, comprising:

receiving textual input, the textual input being based on user interface input generated via one or more user interface input devices of a computing device of a user or an additional user,

wherein the user interface input is generated as part of a dialog that includes the user, the additional user, and an automated assistant implemented by one or more of the processors;

determining user state information for at least one of;

the user and the additional user;

generating reply content based on both the textual input and the user state information, wherein the reply content includes at least one selectable graphical element that, when selected via further user interface input, causes the computing device of the user to present additional content to the user, and wherein generating the reply content based on both the textual input and the user state information comprises;

issuing a search of one or more databases based on both the textual input and the user state information,

receiving one or more search results in response to issuing the search, and

incorporating one or more of the search results into the reply content, wherein the selectable graphical element is associated with one of the search results; and

providing the reply content in response to the textual input, wherein the reply content is provided for inclusion in the dialog, and wherein the reply content is provided for presentation via one or more user interface output devices.

13. The method of claim 12, wherein providing the reply content comprises incorporating the reply content in the dialog for presentation to the user and to the additional user.

14. The method of claim 12, wherein determining the user state information comprises determining the user state information for both the user and the additional user.

15. A method implemented by one or more processors, comprising:

generating a trained machine learning model based on a plurality of training examples, each of the training examples including;

training example input based on a corresponding neutral text segment and corresponding user state information, and

training example output based on a corresponding modified text segment that is a modification of the corresponding neutral text segment in view of the corresponding user state information,

wherein generating the trained machine learning model comprises training the trained machine learning model based on application of the training example input of the training examples and backpropagation based on the training example output of the training examples;

receiving textual input, the textual input being based on user interface input generated by a user via one or more user interface input devices of a computing device of the user,

wherein the user interface input is generated by the user as part of a dialog that includes the user and an automated assistant implemented by one or more of the processors;

determining user state information for the user, wherein the user state information identifies a state of the user, is in addition to the textual input, and is based on sensor data generated by the computing device or an additional computing device of the user;

generating reply content based on both the textual input and the user state information, wherein generating the reply content based on both the textual input and the user state information comprises;

providing the textual input to a text generation engine;

receiving an initial textual output from the text generation engine; and

generating the reply content by modifying the initial textual output based on the user state information, wherein modifying the initial textual output based on the user state information comprises;

applying input and additional input to the trained machine learning model, the input being based on one or more segments of the initial textual output, and the additional input being based on the user state information, and

generating, over the trained machine learning model and based on the input and the additional input, output that indicates one or more terms for modifying the initial textual output; and

providing the reply content in response to the user interface input, wherein the reply content is provided for inclusion in the dialog in response to the textual input, and wherein the reply content is provided for presentation via one or more user interface output devices.