Adjusting training set combination based on classification accuracy

• US 10,402,691 B1
• Filed: 10/04/2018
• Issued: 09/03/2019
• Est. Priority Date: 10/04/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Abdi Taghi Abad, Fardin, Goodsitt, Jeremy Edward, Walters, Austin Grant
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US16/151,825
• Time in Patent Office: 334 Days
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

Various embodiments are generally directed to techniques of adjusting the combination of the samples in a training batch or training set. Embodiments include techniques to determine an accuracy for each class of a classification model, for example. Based on the determined accuracies, the combination of the samples in the training batch may be adjusted or modified to improve the training of the classification model.

Claims

1. A system comprising:

one or more computing devices, wherein the one or more computing devices comprises;

a memory to store instructions; and

processing circuitry, coupled with the memory, operable to execute the instructions, that when executed, cause the processing circuitry to;

access or receive a plurality of samples associated with one or more classes of a classification model;

generate at least one training batch, wherein the at least one training batch includes the plurality of samples associated with the one or more classes;

train the classification model for a number of iterations using the at least one training batch;

determine an accuracy of each class based on the training;

determine whether the accuracy of each of the one or more classes meets or exceeds an accuracy threshold value;

increase a number of the samples associated with the one or more classes having accuracies that fall below the accuracy threshold value in order to generate an adjusted training batch; and

train the classification model for a subsequent number of iterations using the adjusted training batch.

2. The system of claim 1, wherein the plurality of samples and/or the classification model is provided by a user via a network of the system.

3. The system of claim 1, wherein one or more of:

(i) the one or more classes, (ii) the plurality of samples, (iii) the number of iterations, (v) the accuracy threshold value, and (vi) the subsequent number of iterations are changeable via user input.

4. The system of claim 1, wherein the accuracy threshold value is lower than a target accuracy value.

5. An apparatus, comprising:

a memory to store instructions; and

processing circuitry, coupled with the memory, operable to execute the instructions, that when executed, cause the processing circuitry to;

train a classification model for a number of iterations using at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of the classification model;

determine an accuracy of each of the one or more classes based on the training;

compare the determined accuracy of one class relative to the determined accuracies of at least one other class to generate one or more weighted values;

determine, based on the one or more weighted values, how many samples associated with each class are to be included in the at least one training batch; and

perform an adjustment of the at least one training batch for a subsequent training of the classification model.

6. The apparatus of claim 5, wherein the plurality of samples includes a minimum predetermined number of samples associated with each class.

7. The apparatus of claim 5, wherein the processing circuitry is further caused to train the classification model using the adjusted training batch.

8. The apparatus of claim 5, wherein the processing circuitry is further caused to perform a subsequent training of the classification model for a subsequent number of iterations based on the adjusted training batch.

9. The apparatus of claim 8, wherein the processing circuitry is further caused to:

determine a subsequent accuracy of each class based on the adjusted training batch; and

perform a subsequent adjustment of the adjusted training batch based on the determined subsequent accuracy of each class.

10. The apparatus of claim 9, wherein the processing circuitry to is further caused to perform additional training and additional adjustments of the training batch after the subsequent training and the subsequent adjustment until a predefined level of accuracy is achieved for the one or more classes of the classification model.

11. The apparatus of claim 5, wherein the determination of the accuracy of each class comprises determining one or more of the following:

(i) a classification accuracy, (ii) a precision metric, (iii) a recall metric, and (iv) a F1 score.

12. The apparatus of claim 11, wherein the precision metric is a number of true positives divided by a number of false positives.

13. The apparatus of claim 11, wherein the recall metric is a number of true positives divided by the number of true positives and a number of false negatives.

14. The apparatus of claim 5, wherein the at least one training batch has a size that is constant and includes a set number of the plurality of samples.

15. The apparatus of claim 14, wherein the set number is the size divided by a number of the one or more classes.

16. The apparatus of claim 15, wherein the adjustment of the at least one training batch comprises increasing a number of the plurality of samples that are associated with the one or more classes that fall below a predetermined level of accuracy.

17. The apparatus of claim 5, wherein a size of the at least one training batch is variably adjusted based on the determined accuracy of each class.

18. The apparatus of claim 17, wherein the adjustment of the at least one training batch comprises adding to the plurality of samples of the at least one training batch at least an additional sample associated with the one or more classes that fall below a predetermined level of accuracy.

19. A method comprising:

training, via at least one processor, a classification model for a number of iterations using at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of the classification model;

determining, via the at least one processor, an accuracy of each of the one or more classes based on the training;

comparing, via the at least one processor, the determined accuracy of one class relative to the determined accuracies of at least one other class to generate one or more weighted values;

determining, via the at least one processor, how many samples associated with each class are to be included in the at least one training batch based on the one or more weighted values; and

performing, via the at least one processor, an adjustment of the at least one training batch for a subsequent training of the classification model.

20. A non-transitory computer-readable storage medium storing computer-readable program code executable by at least one processor to:

train a classification model for a number of iterations using at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of the classification model;

determine an accuracy of each of the one or more classes based on the training;

compare the determined accuracy of one class relative to the determined accuracies of at least one other class to generate one or more weighted values;

determine, based on the one or more weighted values, how many samples associated with each class are to be included in the at least one training batch; and

perform an adjustment of the at least one training batch for a subsequent training of the classification model.