Systems and methods for intelligently curating machine learning training data and improving machine learning model performance

• US 10,303,978 B1
• Filed: 09/27/2018
• Issued: 05/28/2019
• Est. Priority Date: 03/26/2018
• Status: Active Grant

• Current Assignee: Clinc, Inc.
• Original Assignee: Clinc, Inc.
• Inventors: Kang, Yiping, Zhang, Yunqi, Kummerfeld, Jonathan K., Hill, Parker, Hauswald, Johann, Laurenzano, Michael A., Tang, Lingjia, Mars, Jason
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US16/143,773
• Time in Patent Office: 243 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.

Claims

1. A system for intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system, the system comprising:

one or more remote sources of machine learning training data;

one or more hardware computing servers implementing an artificially intelligent dialogue platform that;

constructs a corpora of machine learning test corpus that comprise a plurality of historical queries and/or historical commands test sampled from one or more production logs of a deployed dialogue system;

configures one or more training data sourcing parameters to source a corpora of raw machine learning training data from one or more remote sources of machine learning training data;

transmits, via one or more communication networks, the one or more training data sourcing parameters to the one or more remote sources of machine learning training data and collects, via the one or more communication networks, the corpora of raw machine learning training data;

calculates, using the one or more hardware computing servers, one or more efficacy metrics of the corpora of raw machine learning training data, wherein calculating the one or more efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data;

identifies whether to train at least one machine learning classifier of the artificially intelligent dialogue system based on one or more the coverage metric value and the diversity metric value of the corpora of raw machine learning;

uses the corpora of raw machine learning training data, as machine learning training input, to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold; and

responsive to training the at least one machine learning classifier using the corpora of raw machine learning training data, deploys the at least one machine learning classifier into a live implementation of the artificially intelligent dialogue system.

2. The system according to claim 1, wherein

calculating the one or more efficacy metrics of the corpora of raw machine learning training data includes:

calculating a coverage metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data, wherein the coverage metric value relates to a measure indicating how well the corpora of raw machine learning training data covers different ways an intent can be expressed by a user of an automated dialogue system;

calculating the coverage metric value for the corpora of raw machine learning training data based on the coverage metric value for each of the plurality of distinct corpus of machine learning training data within the corpora.

3. The system according to claim 2, wherein

the artificially intelligent dialogue platform calculates the coverage metric value of the corpora of raw machine learning training data according to the following equations:

4. The system according to claim 1, wherein

analyzing the one or more efficacy metrics of the corpora of raw machine learning training data includes:

calculating a diversity metric value for each of a plurality of distinct corpus of machine learning training data within the corpora of raw machine learning training data, wherein the diversity metric value relates to a measure indicating a level of heterogeneity among machine learning data within a distinct corpus of machine learning training data; and

calculating an aggregated diversity metric value for the corpora of raw machine learning training data based on the diversity metric value for each of the plurality of distinct corpus of machine learning training data within the corpora.

5. The system according to claim 4, wherein

the artificially intelligent dialogue platform calculates the diversity metric value of the corpora of raw machine learning training data according to the following equations:

6. The system according to claim 1, wherein:

the corpora of machine learning test corpus is defined by a plurality of distinct machine learning test corpus,

each of the plurality of distinct machine learning test corpus is associated with a distinct intent classification task of the artificially intelligent dialogue platform, and

each of the plurality of distinct machine learning test corpus includes at least one subset of the plurality of historical queries and/or historical commands obtained from a deployed dialogue system.

7. The system according to claim 1, wherein

the artificially intelligent dialogue platform further constructs the corpora of machine learning test corpus using a plurality of engineered queries and/or engineered commands, and

each of the plurality of engineered queries and/or engineered commands is artificially generated for one or more identified intent classification tasks.

8. The system according to claim 7, wherein

the artificially intelligent dialogue platform further constructs a composition of the corpora of machine learning test corpus to include a first predetermined ratio of historical queries and/or historical commands and a second predetermined ratio of engineered queries and/or engineered commands, and

the first predetermined ratio of historical queries and/or historical commands has a value greater than a value of the second predetermined ratio of engineered queries and/or engineered commands in the composition.

9. The system according to claim 1, wherein:

the corpora of raw machine learning training data comprises a plurality of distinct corpus of machine learning training data;

each of the plurality of distinct corpus of machine learning training data.

10. The system according to claim 1, wherein

configuring the one or more training data sourcing parameters includes:

generating a plurality of distinct sets of prompts for sourcing raw machine learning training data for each of a plurality of intent classification tasks of the artificially intent dialogue system.

11. The system according to claim 10, wherein:

generating the plurality of distinct sets of prompts is based on a plurality of historical user queries and/or a plurality of historical user commands,

generating the plurality of distinct sets of prompts includes;

test sampling by the artificially intelligent dialogue platform the plurality of historical user queries and/or the plurality of historical user commands from one or more production logs of a deployed dialogue system, and

converting the plurality of historical user queries and/or the plurality of historical user commands into the set of prompts for sourcing raw machine learning training data.

12. The system according to claim 10, wherein

the plurality of distinct sets of prompts comprises a combination of:

a plurality of scenario-driven prompts, wherein each of the plurality of scenario-driven prompts describes a real-world circumstance for which a suitable response is required; and

a plurality of paraphrasing requests, wherein each of the plurality of paraphrasing requests includes an instruction to rephrase and/or paraphrase a given prompt or a given statement.

13. The system according to claim 12, wherein:

the artificially intelligent dialogue platform constructs a composition of the plurality of distinct sets of prompts to include a first predetermined ratio of scenario-driven prompts and a second predetermined ratio of paraphrasing requests, and

the first predetermined ratio of scenario-driven prompts is greater than the second predetermined ratio of paraphrasing requests in the composition.

14. The system according to claim 2, wherein

calculating the coverage metric value for each of the plurality of distinct corpus of machine learning training data includes:

[i] selecting a subject test corpus datum from within a subject distinct machine learning test corpus of the corpora of machine learning test corpus;

[ii] constructing a plurality of diversity pairwise comprising the subject test corpus datum and each training data within a subject distinct corpus of machine learning training data of the corpora of raw machine learning training data;

[iii] calculating a semantic similarities value of each of the plurality of diversity pairwise involving the subject test corpus training datum;

[iv] identifying a minimum diversity metric value for the subject test corpus datum based on the semantic similarities value of each of the plurality of diversity pairwise involving the subject test corpus training datum;

[v] calculating a minimum diversity metric value for each remaining test corpus datum within the subject distinct machine learning test corpus; and

[vi] calculating the coverage metric value for the subject distinct corpus of machine learning training data based on the minimum diversity metric value for the subject test corpus datum and for each of the remaining test corpus datum of the subject distinct machine learning test corpus.

15. The system according to claim 14, further comprising:

calculating by the artificially intelligent dialogue platform;

an aggregated coverage metric value for the corpora of raw machine learning training data, wherein calculating the aggregated diversity metric includes;

calculating an average coverage metric value by calculating a sum of the coverage metric value for each of the plurality of distinct corpus of machine learning training data that defines the corpora and dividing the sum by a number of the distinct corpus of machine learning training data within the corpora.

16. The system according to claim 4, wherein

calculating the diversity metric value for each of the plurality of distinct corpus of machine learning training data includes:

[i] selecting a subject training datum from training data within a subject distinct corpus of machine learning training data of the plurality of distinct corpus of machine learning training data;

[ii] constructing a plurality of diversity pairwise comprising the subject training datum and each of a remaining training data within the subject distinct corpus of machine learning training data;

[iii] calculating a semantic difference value of each of the plurality of diversity pairwise involving the subject training datum;

[iv] calculating a specific diversity metric value for the subject training datum based on an average of the semantic difference value of each of the plurality of diversity pairwise involving the subject training datum;

[v] calculating a specific diversity metric value for each of the remaining training data within the subject distinct corpus of machine learning training data; and

[vi] calculating the diversity metric value for the subject distinct corpus of machine learning training data based on the specific diversity metric value for the subject training datum and for each of the remaining training data of the subject distinct corpus of the machine learning training data.

17. The system according to claim 15, further comprising:

calculating an aggregated diversity metric value for the corpora of raw machine learning training data, wherein calculating the aggregated diversity metric includes;

calculating an average diversity metric value by calculating a sum of the diversity metric value for each of the plurality of distinct corpus of machine learning training data that defines the corpora and dividing the sum by a number of the distinct corpus of machine learning training data within the corpora.

18. The system according to claim 1, wherein

if the coverage metric value of the corpora of raw machine learning training data does not satisfy the minimum coverage metric threshold, the artificially intelligent dialogue platform automatically reconfigures the composition of the plurality of distinct sets of prompts by changing both the first predetermined ratio of scenario-driven prompts and the second predetermined ratio of paraphrasing requests.

19. A method of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system, the method comprising:

an artificially intelligent dialogue platform implemented by one or more hardware computing servers;

constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and/or historical commands test sampled from one or more production logs of a deployed dialogue system;

configuring one or more training data sourcing parameters to source a corpora of raw machine learning training data from one or more remote sources of machine learning training data;

transmitting the one or more training data sourcing parameters to the one or more remote sources of machine learning training data and collecting the corpora of raw machine learning training data;

calculating one or more efficacy metrics of the corpora of raw machine learning training data, wherein calculating the one or more efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data;

identifying whether to train at least one machine learning classifier of the artificially intelligent dialogue system based on one or more the coverage metric value and the diversity metric value of the corpora of raw machine learning;

using the corpora of raw machine learning training data to train at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold; and

once the at least one machine learning classifier is trained using the corpora of raw machine learning training data, deploying the at least one machine learning classifier into an online implementation of the artificially intelligent dialogue system.