Updating natural language interfaces by processing usage data

• US 10,210,244 B1
• Filed: 02/12/2018
• Issued: 02/19/2019
• Est. Priority Date: 02/12/2018
• Status: Active Grant

• Current Assignee: Asapp, Inc.
• Original Assignee: Asapp, Inc.
• Inventors: Branavan, Satchuthananthavale Rasiah Kuhan, Hackman, Joseph Ellsworth, Heckel, Frederick William Poe, Isaksen, Aaron
• Primary Examiner(s): Colucci, Michael C
• Application Number:
  US15/894,504
• Time in Patent Office: 372 Days
• Field of Search: 704 9, 726 5, 715769
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06Q 30/016 After-sales

Abstract

A third-party company may assist companies in providing natural language interfaces for their customers. To implement a natural language interface for a company, a configuration may be received that includes information, such as a list intents, seed messages for the intents, and hierarchical information of the intents. An intent classifier may be trained using the configuration, and the natural language interface may be deployed for use with customers. Usage data of the natural language classifier may be collected and used to improve the natural language interface. Messages corresponding to an intent may be clustered into clusters of similar messages, and a prototype message may be obtained for each cluster to provide a human understandable description of the cluster. The information about the clusters may be used to improve the natural language interface, such as by creating a new intent with a cluster or moving a cluster to a different intent.

Claims

1. A system for implementing natural language interfaces to allow customers to interact with companies, the system comprising:

at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to;

receive a configuration for a natural language interface for a company, wherein the configuration indicates a list of intents and a plurality of seed messages for each intent of the list of intents;

train an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents of the list of intents;

deploy the natural language interface using the intent classifier, wherein the natural language interface is used by customers of the company;

collect usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers, an intent selected by the intent classifier, and an indication of success of the interaction;

analyze the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing the associated cluster;

provide a user interface to view information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receive an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

update the configuration using the received input;

update the intent classifier by retraining the intent classifier using the updated configuration and the usage data; and

deploy the natural language interface using the updated intent classifier.

2. The system of claim 1, wherein the at least one server computer is configured to:

provide a second user interface to the company to allow the company to configure the natural language interface; and

receive the configuration from the company.

3. The system of claim 1, wherein the at least one server computer is configured to:

provide a third user interface to a second company to allow the second company to configure a second natural language interface; and

deploy the second natural language interface, wherein the second natural language interface is used by customers of the second company.

4. The system of claim 1, wherein the at least one server computer is configured to:

receive a message from a customer seeking support from the company;

select two or more intents by processing the message with the intent classifier;

receive clarification information relating to the two or more intents;

select a final intent from the two or more intents using the clarification information; and

implementing an action with the customer corresponding to the final intent.

5. The system of claim 4, wherein the at least one server computer is configured to present hierarchical information of the two or more intents to the customer or a customer service representative.

6. The system of claim 4, wherein the clarification information is received from the customer or from a customer service representative who is assisting the customer.

7. The system of claim 1, wherein a first indication of success of a first interaction is determined by one or more of:

asking the customer if the first interaction was a success;

asking a customer service representative assisting the customer if the first interaction was a success;

determining that the customer completed an action;

comparing an output of the intent classifier with an intent selected by the customer;

or

comparing the output of the intent classifier with the action implemented with the customer.

8. The system of claim 1, wherein the at least one server computer is configured to:

compute a success rate for a cluster;

determine to present a suggestion for improving the natural language interface using the success rate of the cluster;

present a suggestion (i) to create a new intent with the cluster or (ii) to move the cluster to another intent.

9. The system of claim 1, wherein the natural language interface processes a message received from the customer, determines an intent of the message, selects an action using the determined intent, and causes the action to be implemented with the customer.

10. A computer-implemented method for modifying a customer service message, the method comprising:

receiving a configuration for a natural language interface for a company, wherein the configuration indicates a list of intents and a plurality of seed messages for each intent of the list of intents;

training an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents of the list of intents;

deploying the natural language interface using the intent classifier, wherein the natural language interface is used by customers of the company;

collecting usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers, an intent selected by the intent classifier, and an indication of success of the interaction;

analyzing the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing the associated cluster;

providing a user interface to view information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receiving an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

updating the configuration using the received input;

updating the intent classifier by retraining the intent classifier using the updated configuration and the usage data;

deploying the natural language interface using the updated intent classifier.

11. The method of claim 10, wherein the intent classifier comprises a recurrent neural network.

12. The method of claim 10, wherein the configuration comprises a specification of an action to be performed with the customer for each intent of the list of intents.

13. The method of claim 10, wherein clustering the messages of the first set of interactions comprises:

computing message embeddings for the messages of the first set of interactions; and

clustering the messages of the first set of interactions using the message embeddings.

14. The method of claim 10, wherein obtaining the prototype message for a cluster of the plurality of clusters comprises selecting a message with a largest count or selecting a message closest to a center of the cluster.

15. The method of claim 10, wherein obtaining the prototype message for a cluster of the plurality of clusters comprises generating the prototype message with a neural network.

16. The method of claim 10, comprising computing a success rate for each intent of the list of intents.

17. One or more non-transitory computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:

receiving a configuration for a natural language interface for a company, wherein the configuration indicates a list of intents and a plurality of seed messages for each intent of the list of intents;

training an intent classifier using the configuration, wherein the intent classifier processes a message to classify the message as corresponding to one or more intents of the list of intents;

deploying the natural language interface using the intent classifier, wherein the natural language interface is used by customers of the company;

collecting usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers, an intent selected by the intent classifier, and an indication of success of the interaction;

analyzing the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing the associated cluster;

providing a user interface to view information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receiving an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

updating the configuration using the received input;

updating the intent classifier by retraining the intent classifier using the updated configuration and the usage data;

deploying the natural language interface using the updated intent classifier.

18. The one or more non-transitory computer-readable media of claim 17, wherein the configuration comprises hierarchical information for the list of intents.

19. The one or more non-transitory computer-readable media of claim 18, wherein the actions comprise:

receiving an input to modify the hierarchical information relating to the list of intents; and

updating the configuration using the input to modify the hierarchical information relating to the list of intents.

20. The one or more non-transitory computer-readable media of claim 18, wherein the actions comprise:

presenting a suggestion for modifying the hierarchical information for the list of intents.

21. The one or more non-transitory computer-readable media of claim 17, wherein the plurality of seed messages for intent are extracted from existing logs of customer support sessions or created by a person.

22. The one or more non-transitory computer-readable media of claim 17, wherein the actions comprise:

testing the updated intent classifier by obtaining classification results with the updated intent classifier on a corpus of messages.