CROWDSOURCED TRAINING OF TEXTUAL NATURAL LANGUAGE UNDERSTANDING SYSTEMS

• US 20180144046A1
• Filed: 11/16/2017
• Published: 05/24/2018
• Est. Priority Date: 11/18/2016
• Status: Active Grant

• Current Assignee: Definedcrowd Corporation
• Original Assignee: Definedcrowd Corporation
• Inventors: Braga, Daniela, Freitas, Joao, Baldewijns, Daan
• Granted Patent: US 10,528,605 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06N 5/025 Extracting rules from data

Abstract

A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.

Claims

1. A method in a computing system for adapting a virtual assistant to operate with respect to a plurality of user intents, comprising:

for each user intent of the plurality of user intents, receiving (1) a sample expression of the user intent, and (2) an enumeration of entities of direct relevance to the user intent;

for each worker in a first pool crowdsourced workers;

providing the sample expression of at least a portion of the plurality of user intents;

for each provided sample expression, obtaining one or more alternative expressions of user intent that each use a linguistic approach to express the user intent that differs from the sample expression of the distinguished user intent;

from each worker in a third pool crowdsourced workers;

for each of at least a portion of the obtained alternative expressions of user intent;

providing alternative expression;

obtaining a selection of the user intent expressed by the alternative expression;

obtaining a selection of the entities included in the alternative expression; and

training a virtual assistant using the alternative expressions and their selected user intents and included entities.

2. The method of claim 1, further comprising performing at least one form of data cleaning on the obtained alternative expressions of user intent.

3. The method of claim 2, wherein the at least one form of data cleaning is performed by a second pool of crowdsourced workers.

4. The method of claim 1, further comprising:

for each worker in a fourth set of crowdsourced users;

providing at least a portion of the selection of the user intents expressed by the alternative expression;

obtaining expression validation indicators that each represent whether the at least a portion of the selection of the user intents expressed by the alternative expression are correct;

providing at least a portion of the selection of the entities included in the alternative expression; and

obtaining entities validation indicators that each represent whether the at least a portion of the selection of the entities included in the alternative expression are correct;

5. The method of claim 1, wherein at least one worker in the first pool crowdsourced workers and the third pool crowdsourced workers is the same.

6. The method of claim 1, wherein at least one of the first pool of crowdsourced workers and the third pool of crowdsourced workers comprise a user, a bot, or a combination thereof.

7. The method of claim 1, wherein the virtual assistant is a rule-based virtual assistant, and wherein training the virtual assistant comprises adding the alternative expressions to a pool of expressions that are recognized as expressing their selected user intents and included entities.

8. The method of claim 1, wherein the virtual assistant is a statistical model-based virtual assistant, and wherein training the virtual assistant comprises using the alternative expressions and their selected user intents and included entities as observations.

9. A method in a computing system for adapting a virtual assistant to operate with respect to a plurality of user intents, comprising:

for each user intent of the plurality of user intents;

receiving a set of entities of direct relevance to the user intent;

receiving, from a first set of crowdsourced users, a set of queries of direct relevance to the user intent;

for each query in the set of queries of direct relevance to the user intent;

presenting the query to a third set of crowdsourced users;

receiving, from the third set of crowdsourced users, a selection of the user intents expressed by the query; and

receiving, from the third set of crowdsourced users, a selection of the set of entities included in the query;

for each user in a fourth set of crowdsourced users;

presenting at least a portion of the selection of the user intents received from the third set of crowdsourced users and at least a portion of the selection of the set of entities included in the query received from the third set of crowdsourced users; and

receiving validating indicators that represent whether the selection of the user intents received from the third set of crowdsourced users and the selection of the set of entities included in the query received from the third set of crowdsourced users are correct; and

training a virtual assistant using the set of queries and their selected user intents and the set of entities.

10. The method of claim 9, further comprising performing at least one form of data cleaning on queries in the set of queries of direct relevance to the user intent.

11. The method of claim 10, wherein the at least one form of data cleaning is performed by a second set of crowdsourced users.

12. The method of claim 9, wherein a size of the set of queries of direct relevance to the user intent is based at least in part on a complexity of the user intent.

13. The method of claim 9, further comprising for at least one user intent of the plurality of user intents:

receiving a sample query of the at least one user intent, wherein the sample query of the at least one user intent is different than queries in the set of queries of direct relevance to the user intent; and

presenting the sample query of the at least one user intent to the first set of crowdsourced users prior to receiving the set of queries of direct relevance to the user intent.

14. The method of claim 9, further comprising for at least one user intent of the plurality of user intents:

determining a similarity between the at least one user intent and a second user intent of the plurality of user intents;

identifying a sample query of the second user intent of the plurality of intents;

presenting the sample query of the second user intent of the plurality of intents to the first set of crowdsourced users prior to receiving the set of queries of direct relevance to the user intent.

15. The method of claim 9, wherein at least one user in at least two of the first set of users, the third set of users, and the fourth set of users is the same.

16. The method of claim 9, wherein at least one user intent of the plurality of user intents is associated with a project and a scenario, wherein the project represents a solution or a theme for which the virtual assistant provides virtual assistant support, and the scenario represents capabilities of the project.

17. The method of claim 9, wherein the virtual assistant is a rule-based virtual assistant, and wherein training the virtual assistant comprises adding the set of queries to a pool of queries that are recognized as expressing their selected user intents and set of entities.

18. The method of claim 9, wherein the virtual assistant is a statistical model-based virtual assistant, and wherein training the virtual assistant comprises using the set of queries and their selected user intents and the set of entities as observations.

19. A computer-readable storage medium whose contents are capable of causing a computing system to perform a method adapting a virtual assistant to operate with respect to a plurality of user intents, the method comprising:

for each user intent of the plurality of user intents, receiving (1) a sample expression of the user intent, and (2) an enumeration of entities of direct relevance to the user intent;

for each worker in a first pool crowdsourced workers;

providing the sample expression of at least a portion of the plurality of user intents;

for each provided sample expression, obtaining one or more alternative expressions of user intent that each use a linguistic approach to express the user intent that differs from the sample expression of the distinguished user intent;

from each worker in a third pool crowdsourced workers;

for each of at least a portion of the obtained alternative expressions of user intent;

providing alternative expression;

obtaining a selection of the user intent expressed by the alternative expression;

obtaining a selection of the entities included in the alternative expression; and

training a virtual assistant using the alternative expressions and their selected user intents and included entities.

20. The computer-readable storage medium of claim 19, wherein the method further comprises performing at least one form of data cleaning on the obtained alternative expressions of user intent.

21. The computer-readable storage medium of claim 20, wherein the at least one form of data cleaning is performed by a second pool of crowdsourced workers.

22. The computer-readable storage medium of claim 19, wherein the method further comprises:

for each worker in a fourth set of crowdsourced users;

providing at least a portion of the selection of the user intents expressed by the alternative expression;

obtaining expression validation indicators that represent whether the at least a portion of the selection of the user intents expressed by the alternative expression are correct;

providing at least a portion of the selection of the entities included in the alternative expression; and

obtaining entities validation indicators that represent whether the at least a portion of the selection of the entities included in the alternative expression are correct;

23. The computer-readable storage medium of claim 19, wherein at least one worker in the first pool crowdsourced workers and the third pool crowdsourced workers is the same.

24. The computer-readable storage medium of claim 19, wherein at least one of the first pool of crowdsourced works and the third pool of crowdsourced workers comprise a user, a bot, or a combination thereof.

25. The computer-readable storage medium of claim 19, wherein the virtual assistant is a rule-based virtual assistant, and wherein training the virtual assistant comprises adding the alternative expressions to a pool of expressions that are recognized as expressing their selected user intents and included entities.

26. The computer-readable storage medium of claim 19, wherein the virtual assistant is a statistical model-based virtual assistant, and wherein training the virtual assistant comprises using the alternative expressions and their selected user intents and included entities as observations.