Processing communications using a prototype classifier

• US 10,747,957 B2
• Filed: 11/13/2018
• Issued: 08/18/2020
• Est. Priority Date: 11/13/2018
• Status: Active Grant

• Current Assignee: Asapp, Inc.
• Original Assignee: Asapp, Inc.
• Inventors: Wohlwend, Jeremy Elliot Azriel
• Primary Examiner(s): Yen, Eric
• Application Number:
  US16/189,613
• Publication Number:
  US 20200151254A1
• Time in Patent Office: 644 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages

Abstract

In some applications, it may be desired to process a message to determine an intent of the message, where the intent indicates the meaning of the message. An intent classifier may be used to determine the meaning of a message by processing the message to compute a message embedding vector that represents the message in a vector space. Each possible intent may be represented by a prototype vector, and the intent of the message may be determined by comparing the message embedding to one or more prototype vectors, such as by selecting an intent whose prototype vector is closest to the message embedding. An intent classifier may be used, for example, (i) to implement an automated communications system with states where each state is associated with a subset of the possible intents or (ii) for processing usage data of a communications system to update the intents of the communications system.

Claims

1. A computer-implemented method for processing messages using a neural network, the method comprising:

obtaining the neural network, wherein the neural network is configured to process a message to compute a message embedding that represents a meaning of the message;

obtaining prototype vectors for a plurality of intents, wherein a respective prototype vector for a respective intent was computed using a respective message embedding of a message corresponding to the respective intent, and wherein the respective message embedding was computed using the neural network;

selecting a first state, wherein the first state corresponds to a first outbound message and a first subset of intents for possible responses to the first outbound message, and wherein the first subset of intents is a subset of the plurality of intents;

transmitting the first outbound message to a user;

receiving a first inbound message from the user;

processing the first inbound message with the neural network to compute a first message embedding;

selecting a first intent of the first subset of intents by comparing the first message embedding with prototype vectors for the first subset of intents;

selecting a second state using the first intent, wherein the second state corresponds to a second outbound message and a second subset of intents for possible responses to the second outbound message, and wherein the second subset of intents is a subset of the plurality of intents;

transmitting the second outbound message to the user;

receiving a second inbound message from the user;

processing the second inbound message with the neural network to compute a second message embedding; and

selecting a second intent of the second subset of intents by comparing the second message embedding with prototype vectors for the second subset of intents.

2. The computer-implemented method of claim 1, wherein the first inbound message comprises text or an audio signal comprising speech.

3. The computer-implemented method of claim 1, wherein processing the first inbound message with the neural network to compute the first message embedding comprises:

obtaining word embeddings for words of the first inbound message, wherein a word embedding is a vector representing a corresponding word in a vector space;

processing the word embeddings with the neural network to compute the first message embedding.

4. The computer-implemented method of claim 1, wherein a first prototype vector for the first intent is computed as an average of message embeddings of messages corresponding to the first intent.

5. The computer-implemented method of claim 1, wherein comparing the first message embedding with the prototype vectors for the first subset of intents comprises computing a distance between the first message embedding and each of the prototype vectors for the first subset of intents.

6. The computer-implemented method of claim 5, wherein the distance is a Euclidean distance or a hyperbolic distance.

7. The computer-implemented method of claim 1, wherein the second state is connected to the first state in a graph.

8. The computer-implemented method of claim 1, wherein the first state is a root of a graph.

9. A system for processing messages using a neural network, the system comprising:

at least one server computer comprising at least one processor and at least one memory, the at least one server computer configured to;

obtain the neural network, wherein the neural network is configured to process a message to compute a message embedding that represents a meaning of the message;

obtain prototype vectors for a plurality of intents, wherein a respective prototype vector for a respective intent was computed using a respective message embedding of a message corresponding to the respective intent, and wherein the respective message embedding was computed using the neural network;

select a first state, wherein the first state corresponds to a first outbound message and a first subset of intents for possible responses to the first outbound message, and wherein the first subset of intents is a subset of the plurality of intents;

transmit the first outbound message to a user;

receive a first inbound message from the user;

process the first inbound message with the neural network to compute a first message embedding;

select a first intent of the first subset of intents by comparing the first message embedding with prototype vectors for the first subset of intents;

select a second state using the first intent, wherein the second state corresponds to a second outbound message and a second subset of intents for possible responses to the second outbound message, and wherein the second subset of intents is a subset of the plurality of intents;

transmit the second outbound message to the user;

receive a second inbound message from the user;

process the second inbound message with the neural network to compute a second message embedding; and

select a second intent of the second subset of intents by comparing the second message embedding with prototype vectors for the second subset of intents.

10. The system of claim 9, wherein the system is implemented by a company for providing customer support to its customers.

11. The system of claim 9, wherein the system is implemented by a third-party company for providing customer support to customers of a company.

12. The system of claim 9, wherein the at least one server computer is configured to:

select a third state, wherein the third state corresponds to a third outbound message and a third subset of intents for possible responses to the third outbound message, and wherein the third subset of intents is a subset of the plurality of intents;

transmit the third outbound message to the user;

receive a third inbound message from the user;

process the third inbound message with the neural network to compute a third message embedding; and

select a third intent by comparing the third message embedding to a prototype vector for the third intent, wherein the third intent is in the plurality of intents but is not in the third subset of intents.

13. The system of claim 12, wherein the at least one server computer is configured to:

selecting a fourth state using the third intent; and

wherein the fourth state is not connected to the third state in a graph.

14. The system of claim 12, wherein the at least one server computer is configured to select the third intent by comparing the third message embedding to a prototype vector for each intent of the plurality of intents.

15. The system of claim 9, wherein a first prototype vector for the first intent is computed as an average of message embeddings of messages corresponding to the first intent.

16. The system of claim 9, wherein the first subset of intents and the second subset of intents comprise at least one intent in common.

17. One or more non-transitory, computer-readable media comprising computer-executable instructions that, when executed, cause at least one processor to perform actions comprising:

obtaining a neural network, wherein the neural network is configured to process a message to compute a message embedding that represents a meaning of the message;

obtaining prototype vectors for a plurality of intents, wherein a respective prototype vector for a respective intent was computed using a respective message embedding of a message corresponding to the respective intent, and wherein the respective message embedding was computed using the neural network;

selecting a first state, wherein the first state corresponds to a first outbound message and a first subset of intents for possible responses to the first outbound message, and wherein the first subset of intents is a subset of the plurality of intents;

transmitting the first outbound message to a user;

receiving a first inbound message from the user;

processing the first inbound message with the neural network to compute a first message embedding;

selecting a first intent of the first subset of intents by comparing the first message embedding with prototype vectors for the first subset of intents;

selecting a second state using the first intent, wherein the second state corresponds to a second outbound message and a second subset of intents for possible responses to the second outbound message, and wherein the second subset of intents is a subset of the plurality of intents;

transmitting the second outbound message to the user;

receiving a second inbound message from the user;

processing the second inbound message with the neural network to compute a second message embedding; and

selecting a second intent of the second subset of intents by comparing the second message embedding with prototype vectors for the second subset of intents.

18. The one or more non-transitory, computer-readable media of claim 17, wherein comparing the first message embedding with the prototype vectors for the first subset of intents comprises computing a distance between the first message embedding and each of the prototype vectors for the first subset of intents.

19. The one or more non-transitory, computer-readable media of claim 17, wherein a first prototype vector for the first intent is computed as an average of message embeddings of messages corresponding to the first intent.

20. The one or more non-transitory, computer-readable media of claim 17, wherein the neural network comprises a recurrent neural network.