Adjusting training set combination based on classification accuracy

• US 10,534,984 B1
• Filed: 06/14/2019
• Issued: 01/14/2020
• Est. Priority Date: 10/04/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Abdi Taghi Abad, Fardin, Goodsitt, Jeremy Edward, Walters, Austin Grant
• Primary Examiner(s): Liew, Alex Kok S
• Application Number:
  US16/441,649
• Time in Patent Office: 214 Days
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

Various embodiments are generally directed to techniques of adjusting the combination of the samples in a training batch or training set. Embodiments include techniques to determine an accuracy for each class of a classification model, for example. Based on the determined accuracies, the combination of the samples in the training batch may be adjusted or modified to improve the training of the classification model.

Claims

1. An apparatus, comprising:

memory to store instructions; and

processing circuitry, coupled with the memory, operable to execute the instructions, that when executed, cause the processing circuitry to;

generate at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of a classification model;

train the classification model for a number of iterations using the at least one training batch;

determine an accuracy of each class based on the training; and

adjust a number of the plurality of samples corresponding to the one or more classes in the at least one training batch that meet one or more criteria related to the adjustment based on the determined accuracy of each class.

2. The apparatus of claim 1, wherein the processing circuitry is further caused to perform a subsequent training of the classification model for a subsequent number of iterations based on the performed adjustment of the at least one training batch.

3. The apparatus of claim 2, wherein the processing circuitry is further caused to:

determine a subsequent accuracy of each class after the adjustment of the at least one training batch; and

perform a subsequent adjustment of the at least one training batch based on the determined subsequent accuracy of each class.

4. The apparatus of claim 3, wherein the processing circuitry to is further caused to perform additional training and additional adjustments of the training batch after the subsequent training and the subsequent adjustment until a predefined level of accuracy is achieved for the one or more classes of the classification model.

5. The apparatus of claim 1, wherein the plurality of samples includes a minimum predetermined number of samples associated with each class.

6. The apparatus of claim 1, wherein the determination of the accuracy of each class comprises determining one or more of the following:

(i) a classification accuracy, (ii) a precision metric, (iii) a recall metric, and (iv) a F1 score.

7. The apparatus of claim 6, wherein the precision metric is a number of true positives divided by a number of false positives.

8. The apparatus of claim 6, wherein the recall metric is a number of true positives divided by the number of true positives and a number of false negatives.

9. The apparatus of claim 1, wherein the at least one training batch has a size that is constant and includes a set number of the plurality of samples, wherein the set number is the size divided by a number of the one or more classes, and wherein the adjustment of the at least one training batch comprises increasing a number of the plurality of samples that are associated with the one or more classes that fall below a predetermined level of accuracy.

10. The apparatus of claim 1, wherein a size of the at least one training batch is variably adjusted based on the determined accuracy of each class, and wherein the adjustment of the at least one training batch comprises adding to the plurality of samples of the at least one training batch at least an additional sample associated with the one or more classes that fall below a predetermined level of accuracy.

11. A method comprising:

generating, via one or more processors, at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of a classification model;

training, via the one or more processors, the classification model for a number of iterations using the at least one training batch;

determining, via the one or more processors, an accuracy of each class based on the training; and

adjusting, via the one or more processors, a number of the plurality of samples corresponding to the one or more classes in the at least one training batch that meet one or more criteria related to the adjusting based on the determined accuracy of each class.

12. The method of claim 11, further comprising performing, via the one or more processors, additional training and additional adjustments of the at least one training batch until a predefined level of accuracy is achieved for each of the one or more classes of the classification model.

13. The method of claim 11, wherein the plurality of samples includes a minimum predetermined number of samples associated with each class.

14. The method of claim 11, wherein the determining of the accuracy of each class comprises determining one or more of the following:

(i) a classification accuracy, (ii) a precision metric, (iii) a recall metric, and (iv) a F1 score.

15. The method of claim 11, wherein the at least one training batch has a size that is:

(i) constant and includes a set number of the plurality of samples or (ii) variably adjustable based on the determined accuracy of each class.

16. A non-transitory computer-readable storage medium storing computer-readable program code executable by a processor to:

generate at least one training batch, wherein the at least one training batch includes a plurality of samples associated with one or more classes of a classification model;

train the classification model for a number of iterations using the at least one training batch;

determine an accuracy of each class based on the training; and

adjust a number of the plurality of samples corresponding to the one or more classes in the at least one training batch that meet one or more criteria related to the adjustment based on the determined accuracy of each class.

17. The non-transitory computer-readable storage medium of claim 16, wherein the computer-readable program code further causes the processor to perform additional training and additional adjustments of the at least one training batch until a predefined level of accuracy is achieved for each of the one or more classes of the classification model.

18. The non-transitory computer-readable storage medium of claim 16, wherein the plurality of samples includes a minimum predetermined number of samples associated with each class.

19. The non-transitory computer-readable storage medium of claim 16, wherein the determination of the accuracy of each class comprises determining one or more of the following:

(i) a classification accuracy, (ii) a precision metric, (iii) a recall metric, and (iv) a F1 score.

20. The non-transitory computer-readable storage medium of claim 16, wherein the at least one training batch has a size that is:

(i) constant and includes a set number of the plurality of samples or (ii) variably adjustable based on the determined accuracy of each class.