Signal analysis in a conversational scheduling assistant computing system

• US 10,394,957 B2
• Filed: 09/25/2017
• Issued: 08/27/2019
• Est. Priority Date: 09/25/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Cheung, Benjamin Gene, Monroy-Hernandez, Andres, Newman, Todd Daniel, Carvalho Neto, Mayerber Loureiro De, Palmer, Michael Brian, Bhattacharya, Pamela, Cranshaw, Justin Brooks, Lee, Charles Yin-Che
• Primary Examiner(s): Abebe, Daniel
• Application Number:
  US15/714,077
• Publication Number:
  US 20190095424A1
• Time in Patent Office: 701 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services

Abstract

A software agent, that is used to assist in providing a service, receives communications from a set of users that are attempting to use the software agent. The communications include communications that are interacting with the software agent, and communications that are not interacting with the software agent. The software agent performs natural language processing on all communications to identify such things as user sentiment, user concerns or other items in the content of the messages, and also to identify actions taken by the users in order to obtain a measure of user satisfaction with the software agent. One or more action signals are then generated based upon the identified user satisfaction with the software agent.

Claims

1. A computing system, comprising:

a messaging system that sends and receives group messages among a plurality of different users in a group of users;

an assistance computing system comprising a scheduling assistant that receives a natural language request message from the messaging system, indicative of a user request, from a requesting user in the group of users, for interaction with the assistance computing system and that conducts a dialog with a user, using the messaging system, to render a scheduling assistance service in response to the natural language request message;

an analyzer system comprising calendar mining logic configured to access calendar information corresponding to the requesting user to identify a usage level indicative of a level of usage of the scheduling assistant by the requesting user, wherein the analyzer system detects other natural language messages, in addition to the natural language request message and messages in the dialog, from the plurality of different users in the group of users, generates satisfaction indicia indicative of user satisfaction corresponding to the assistance computing system based on the usage level, a content of the other natural language messages, the natural language request message and messages in the dialog; and

control signal generator logic that generates a control signal to control the computing system based on the satisfaction indicia.

2. The computing system of claim 1 wherein the control signal generator logic is configured to generate the control signal to control the messaging system to prompt the requesting user to recommend the assistance computing system to another user.

3. The computing system of claim 1 wherein the analyzer system comprises:

feature discovery engagement logic configured to identify a feature discovery natural language message inquiring about additional functionality offered by the assistance computing system and generating a feature discovery engagement indicator indicative of the feature discovery natural language message; and

satisfaction level detection logic configured to generate the satisfaction indicia based on the feature discovery engagement indicator.

4. The computing system of claim 1 wherein the calendar mining logic comprises:

service coverage determination logic configured to access the calendar information and identify how often the requesting user used the scheduling assistant in performing the scheduling operations relative to how often the user performed the scheduling operations without using the scheduling assistant to obtain a coverage indicator, the satisfaction level detection logic generating the satisfaction indicia based on the coverage indicator.

5. The computing system of claim 4 wherein the calendar mining logic comprises:

meeting categorization logic configured to access the calendar information and identify different categories of scheduling items for which the requesting user has performed the scheduling operation.

6. The computing system of claim 5 wherein the service coverage determination logic is configured to obtain the coverage indicator corresponding to each category of scheduling items.

7. The computing system of claim 4 wherein the calendar mining logic comprises:

trend identifying logic configured to identify a trend in how often the requesting user used the scheduling assistant in performing the scheduling operations relative to how often the user performed the scheduling operations without using the scheduling assistant.

8. The computing system of claim 3 wherein the analyzer system comprises:

user recommendation detection logic configured to detect whether the requesting user recommended the assistance computing system to other users and to generate a recommendation detected indicator, the satisfaction level detection logic being configured to generate the satisfaction indicia based on the recommendation detected indicator.

9. The computing system of claim 3 wherein the analyzer system comprises:

sentiment analysis logic configured to identify sentiment of the requesting user in the content of the messages and identify a correlation between the identified sentiment and the assistance computing system, the satisfaction level detection logic generating the satisfaction indicia based on the identified sentiment and the correlation to the assistance computing system.

10. The computing system of claim 9 wherein the sentiment analysis logic is configured to identify the sentiment of the requesting user in the content of the messages and identify a correlation between the identified sentiment and an individual feature of the assistance computing system, the satisfaction level detection logic generating the satisfaction indicia based on the identified sentiment and the correlation to the individual feature of the assistance computing system.

11. A computer implemented method, comprising:

receiving, at an assistance computing system, a natural language request message from a messaging system that sends and receives group messages among a plurality of different users in a group of users, the natural language request message being indicative of a user request, from a requesting user in the group of users, for interaction with the assistance computing system;

controlling the messaging system to conduct a dialog with a user to render a service in response to the natural language request message;

detecting other natural language messages, in addition to the natural language request message and messages in the dialog, from the plurality of different users in the group of users;

generating satisfaction indicia indicative of user satisfaction corresponding to the assistance computing system based on a content of the other natural language messages, the natural language request message and messages in the dialog, wherein generating satisfaction indicia comprises;

identifying a feature discovery natural language message inquiring about additional functionality offered by the assistance computing system;

generating a feature discovery engagement indicator indicative of the feature discovery natural language message; and

generating the satisfaction indicia based on the feature discovery engagement indicator; and

generating a control signal to control the messaging system based on the satisfaction indicia.

12. The computer implemented method of claim 11 wherein generating the control signal comprises:

generating the control signal to control the messaging system to prompt the requesting user to recommend the assistance computing system to another user.

13. The computer implemented method of claim 11 wherein the assistance computing system comprises a scheduling assistant configured to render a scheduling assistance service, and wherein generating satisfaction indicia comprises:

accessing calendar information corresponding to the requesting user;

identifying a usage level indicative of a level of usage of the scheduling assistant by the requesting user; and

generating the satisfaction indicia based on the usage level.

14. The computer implemented method of claim 13 wherein identifying a usage level comprises:

identifying the level of usage, from the calendar information, by identifying how often the requesting user used the scheduling assistant in performing the scheduling operations relative to how often the user performed the scheduling operations without using the scheduling assistant; and

identifying a coverage indicator based on the level of usage.

15. The computer implemented method of claim 14 wherein identifying the usage level comprises:

identifying, from the calendar information, different categories of scheduling items for which the requesting user has performed the scheduling operation; and

identifying the coverage indicator corresponding to each category of scheduling items.

16. The computer implemented method of claim 15 further comprising:

identifying a trend in how often the requesting user used the scheduling assistant in performing the scheduling operations relative to how often the user performed the scheduling operations without using the scheduling assistant, in each of the categories of scheduling items, based on a set of coverage indicators corresponding to each of the categories of scheduling items.

17. The computer implemented method of claim 11 wherein generating satisfaction indicia comprises:

identifying sentiment of the requesting user in the content of the messages;

identifying a correlation between the identified sentiment and the assistance computing system; and

generating the satisfaction indicia based on the identified sentiment and the correlation to the assistance computing system.

18. A computing system, comprising:

an assistance system that receives a natural language request message indicative of a request for interaction and that provides a service based on the natural language request for interaction;

a messaging system that conducts a conversation that has a set of messages sent from a user to the assistance system, wherein at least one message in the set of messages comprises the natural language request message;

a signal analyzer that analyzes content of the natural language request message and other messages in the set of messages and generates satisfaction indicia indicative of user satisfaction corresponding to the assistance system based on the content of the natural language request message and other messages in the set of messages; and

control signal generator logic that controls the messaging system, based on the satisfaction indicia, to prompt the user to recommend the assistance system to another user.