×

Systems and method for automatically configuring machine learning models

  • US 10,296,848 B1
  • Filed: 03/05/2018
  • Issued: 05/21/2019
  • Est. Priority Date: 03/05/2018
  • Status: Active Grant
First Claim
Patent Images

1. A system that rapidly improves a classification accuracy of a machine learning classification model of an artificially intelligent conversational system, the system comprising:

  • a machine learning configuration and management console that enables an administrator of an artificially intelligent conversational service to configure updates to the machine learning classification model of the artificially intelligent conversational system, wherein the machine learning configuration and management console comprises one or more computer processors and a non-transitory computer-readable medium storing computer instructions then when executed by the one or more computer processors performs the steps of;

    detecting that an accuracy level of the machine learning classification model does not satisfy a predetermined threshold;

    in response to detecting that the accuracy level does not satisfy the predetermined threshold, automatically generating a notification requiring an update for improving a classification accuracy of the machine learning classification model;

    implementing one or more user interfaces that receive input for configuring a machine learning training data request based on the notification, wherein a machine learning training data request includes a plurality of seed machine learning data samples and a request, from the artificially intelligent conversational service, to a plurality of remote third-party training data sources to generate machine learning training data using the plurality of seed machine learning data samples, wherein at least one of the plurality of remote third-party training data sources includes a remote crowdsourcing platform;

    transmitting, by the artificially intelligent conversational service, via a network, the machine learning training data request to each of the plurality of remote third-party training data sources, wherein each of the plurality of remote third-party training data sources is different from each other;

    collecting and storing the machine learning training data produced by each of the plurality of remote third-party training data sources, wherein the machine learning training data comprises a plurality of training data samples proliferated based on the seed machine learning data samples of the machine learning data request, and wherein each of the plurality of training data samples of the machine learning training data is distinct from each of the plurality of seed machine learning data samples;

    processing the machine learning training data collected from the plurality of remote training data sources using a predefined training data processing algorithm, wherein the processing the machine learning training data includes;

    [i] calculating a fit score value for each of the plurality of training data samples, wherein the fit score value relates to how well each of the plurality of training data samples fits one or more of the plurality of seed training data samples of the training data request,[ii] after the fit score value is calculated for each training data sample, applying a pruning threshold to each of the plurality of training data samples, wherein the pruning threshold comprises a minimum required fit score value for a given training data sample, and[iii] pruning from the plurality of training data samples any training data sample that does not satisfy the pruning threshold; and

    in response to processing the collected machine learning training data;

    [a] simulating a performance of the machine learning classification model using the plurality of training data samples remaining after the pruning;

    [b] identifying a simulated accuracy level of the machine learning classification model;

    updating the machine learning classification model based on the simulated accuracy level by training the machine learning classification model with the plurality of training data samples remaining after the pruning; and

    after the updating, deploying the machine learning classification model into a live use by the artificially intelligent conversational system.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×