Updating natural language interfaces by processing usage data

• US 10,515,104 B2
• Filed: 01/07/2019
• Issued: 12/24/2019
• Est. Priority Date: 02/12/2018
• Status: Active Grant

• Current Assignee: Asapp, Inc.
• Original Assignee: Asapp, Inc.
• Inventors: Branavan, Satchuthananthavale Rasiah Kuhan, Hackman, Joseph Ellsworth, Heckel, Frederick William Poe, Isaksen, Aaron
• Primary Examiner(s): Colucci, Michael C
• Application Number:
  US16/240,958
• Publication Number:
  US 20190251104A1
• Time in Patent Office: 351 Days
• Field of Search: 704 9, 726 5, 715769
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06Q 30/016 After-sales

Abstract

A third-party company may assist companies in providing natural language interfaces for their customers. To implement a natural language interface for a company, a configuration may be received that includes information, such as a list intents, seed messages for the intents, and hierarchical information of the intents. An intent classifier may be trained using the configuration, and the natural language interface may be deployed for use with customers. Usage data of the natural language classifier may be collected and used to improve the natural language interface. Messages corresponding to an intent may be clustered into clusters of similar messages, and a prototype message may be obtained for each cluster to provide a human understandable description of the cluster. The information about the clusters may be used to improve the natural language interface, such as by creating a new intent with a cluster or moving a cluster to a different intent.

Claims

1. A system for implementing natural language interfaces to allow customers to interact with companies, the system comprising:

at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to;

receive a configuration for a natural language interface for a company, wherein the configuration includes a list of intents;

deploy the natural language interface using the configuration, wherein the natural language interface is used by customers of the company;

collect usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers and an intent selected by the natural language interface;

analyze the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing a corresponding cluster;

present information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receive an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

update the configuration using the received input;

deploy an updated natural language interface using the updated configuration.

2. The system of claim 1, wherein the at least one server computer is configured to:

provide a user interface to the company to allow the company to configure the natural language interface; and

receive the configuration from the company.

3. The system of claim 1, wherein the at least one server computer is configured to:

receive a second configuration for a second natural language interface for a second company; and

deploy the second natural language interface, wherein the second natural language interface is used by customers of the second company.

4. The system of claim 1, wherein the usage data includes an indication of success for each interaction of the plurality of interactions.

5. The system of claim 4, wherein the at least one server computer is configured to compute a success rate for each cluster of the plurality of clusters.

6. The system of claim 4, wherein the at least one server computer is configured to compute a success rate for the first intent.

7. The system of claim 4, wherein a first indication of success of a first interaction is determined by one or more of:

asking a customer if the first interaction was a success;

asking a customer service representative assisting the customer if the first interaction was a success;

or

determining that the customer completed an action.

8. The system of claim 1, wherein the at least one server computer is configured to:

determine to present a suggestion for improving the natural language interface;

present a suggestion (i) to create a new intent with the first cluster or (ii) to move the first cluster to another intent.

9. The system of claim 1, wherein the natural language interface processes a message received from a customer, determines an intent of the message, selects an action using the intent, and causes the action to be implemented with the customer.

10. A computer-implemented method for implementing natural language interfaces to allow customers to interact with companies, the method comprising:

receiving a configuration for a natural language interface for a company, wherein the configuration includes a list of intents;

deploying the natural language interface using the configuration, wherein the natural language interface is used by customers of the company;

collecting usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers, and an intent selected by the natural language interface;

analyzing the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing a corresponding cluster;

presenting information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receiving an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

updating the configuration using the received input;

deploying an updated natural language interface using the updated configuration.

11. The method of claim 10, wherein the natural language interface comprises a recurrent neural network.

12. The method of claim 10, wherein the configuration comprises a specification of an action to be performed with a customer for each intent of the list of intents.

13. The method of claim 10, wherein clustering the messages of the first set of interactions comprises:

computing message embeddings for the messages of the first set of interactions; and

clustering the messages of the first set of interactions using the message embeddings.

14. The method of claim 10, wherein obtaining the prototype message for a cluster of the plurality of clusters comprises selecting a message with a largest count or selecting a message closest to a center of the cluster.

15. The method of claim 10, wherein obtaining the prototype message for a cluster of the plurality of clusters comprises generating the prototype message with a neural network.

16. The method of claim 10, comprising computing a success rate for each intent of the list of intents.

17. One or more non-transitory computer-readable media comprising computer executable instructions that, when executed, cause at least one processor to perform actions comprising:

receiving a configuration for a natural language interface for a company, wherein the configuration includes a list of intents;

deploying the natural language interface using the configuration, wherein the natural language interface is used by customers of the company;

collecting usage data for the natural language interface, wherein the usage data comprises a plurality of interactions, and wherein each interaction comprises a message received from one of a plurality of customers, and an intent selected by the natural language interface;

analyzing the usage data by;

obtaining a first set of interactions from the usage data, wherein the first set of interactions corresponds to a first intent of the list of intents,

clustering messages of the first set of interactions into a plurality of clusters, and

obtaining a prototype message for each cluster of the plurality of clusters, wherein each prototype message comprises an example message representing a corresponding cluster;

presenting information about the plurality of clusters, the information comprising the prototype message for each cluster of the plurality of clusters;

receiving an input to (i) create a new intent using a first cluster of the plurality of clusters or (ii) move the first cluster of the plurality of clusters to a second intent of the list of intents;

updating the configuration using the received input;

deploying an updated natural language interface using the updated configuration.

18. The one or more non-transitory computer-readable media of claim 17, wherein the configuration comprises a plurality of seed messages for each intent of the list of intents.

19. The one or more non-transitory computer-readable media of claim 17, wherein the actions comprise:

train an intent classifier using the configuration; and

wherein the natural language interface comprises the intent classifier.

20. The one or more non-transitory computer-readable media of claim 17, wherein the actions comprise presenting a suggestion (i) to create a new intent with the first cluster or (ii) to move the first cluster to the second intent.