×

Multi-task learning using knowledge distillation

  • US 10,635,977 B2
  • Filed: 07/01/2019
  • Issued: 04/28/2020
  • Est. Priority Date: 12/30/2016
  • Status: Active Grant
First Claim
Patent Images

1. A computer implemented method comprising:

  • obtaining a respective set of training data for each of a plurality of machine learning tasks;

    for each of the machine learning tasks, configuring a respective teacher machine learning model to perform the machine learning task by training the teacher machine learning model on the training data for the task; and

    training a single student machine learning model having a plurality of student machine learning model parameters to perform all of the plurality of machine learning tasks using (i) the configured teacher machine learning models, and (ii) the obtained training data, wherein training the single student machine learning model comprises;

    for each of the plurality of machine learning tasks;

    selecting one or more subsets from the set of training data for the machine learning task;

    processing the selected subsets using the respective teacher machine learning model to generate respective teacher machine learning model outputs; and

    training the single student machine learning model to perform the machine learning task using (i) the selected one or more subsets, and (ii) respective generated teacher machine learning model outputs, comprising, for each subset;

    augmenting the subset with an identifier for the machine learning task;

    processing the augmented subset using the student machine learning model to generate a student machine learning model output; and

    adjusting values of the student machine learning model parameters to match the generated student machine learning model output to the respective generated teacher machine learning model output for the subset.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×