Systems and methods for quality assurance of image recognition model

• US 10,332,245 B1
• Filed: 12/11/2018
• Issued: 06/25/2019
• Est. Priority Date: 12/11/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Price, Micah, Ho, Chi-San, Charania, Aamer, Vasisht, Sunil
• Primary Examiner(s): Fujita, Katrina R
• Application Number:
  US16/216,982
• Time in Patent Office: 196 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

Disclosed embodiments provide systems, methods, and computer-readable storage media for stress testing a vehicle identification model by determining image quality parameters optimizing model confidence values. The embodiments comprise one or more memory devices storing instructions and one or more processors configured to execute instructions. The one or more processors perform operations comprising receiving a first set of images of the vehicle, wherein the images contain preset values for parameters comprising at least one of a rotation, a focus, a contrast, and a color. The system further analyzes the first set of images with an identification model and determining an initial identification confidence of the vehicle. The system further modifies the first set of images, compares the modified images to the first set of images, and determines whether the model should be modified based on at least the comparison.

Claims

1. A system for stress testing a vehicle identification model, comprising:

one or more memory devices storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising;

receiving a plurality of first images of a vehicle, wherein the first images contain preset values of attributes, the attributes comprising at least one of a rotation attribute, a focus attribute, a contrast attribute, or a color attribute;

analyzing the first images using the vehicle identification model;

determining a first identification confidence value of the vehicle output from the model based on the first images;

modifying the first images;

analyzing the modified images using the model;

determining a second identification confidence value of the vehicle output from the model based on the modified images;

comparing the first and second identification confidence values; and

adjusting the model based on at least the comparison.

2. The system of claim 1, wherein modifying the first images comprises modifying values of a first one of the attributes of the first images.

3. The system of claim 1, wherein modifying the first images further comprises modifying the rotation, focus, contrast, and color attribute values of the first images.

4. The system of claim 3, wherein modifying the rotation attribute value of the first images further comprises:

adjusting the preset rotation attribute value of the of images to a second rotation attribute value;

analyzing the modified images, with the second rotation attribute value, with the vehicle identification model and determining a rotation identification confidence value; and

comparing the rotation identification confidence value to the first identification confidence value.

5. The system of claim 4, the operations further comprising:

adjusting the preset rotation attribute value of the first images to a third rotation attribute value;

analyzing the modified images, with the third rotation attribute value, with the vehicle identification model and determining a second rotation identification confidence value; and

comparing the second rotation identification confidence value to the first identification confidence value.

6. The system of claim 5, wherein modifying the focus attribute value of the first images further comprises:

adjusting the preset focus attribute value of the first images to a second focus attribute value;

analyzing the modified images, with the second focus attribute value, with the identification model and determining a focus identification confidence; and

comparing the focus identification confidence to the first identification confidence.

7. The system of claim 6, the operations further comprising:

adjusting the preset focus value of the first set of images to a third focus value;

analyzing the modified images, with the third focus value, with the identification model and determining a second focus identification confidence; and

comparing the second focus identification confidence to the first identification confidence.

8. The system of claim 7, wherein modifying the contrast attribute value of the first images further comprises:

adjusting the preset contrast attribute value of the first images to a second contrast attribute value;

analyzing the modified images, with the second contrast attribute value, with the identification model and determining a contrast identification confidence; and

comparing the contrast identification confidence to the first identification confidence.

9. The system of claim 8, the operations further comprising:

adjusting the preset contrast attribute value of the first set of images to a third contrast attribute value;

analyzing the modified images, with the third contrast attribute value, with the identification model and determining a second contrast identification confidence; and

comparing the second contrast identification confidence to the initial identification confidence.

10. The system of claim 9, wherein modifying the color attribute value of the first images further comprises:

adjusting the preset color attribute value of the first images to a second contrast attribute value;

analyzing the modified images, with the second color attribute value, with the identification model and determining a color identification confidence; and

comparing the color identification confidence to the first identification confidence.

11. The system of claim 10, the operations further comprising:

adjusting the preset color attribute value of the first images to a third color attribute value;

analyzing the modified images, with the third color attribute value, with the identification model and determining a second color identification confidence; and

comparing the second color identification confidence to the first identification confidence.

12. The system of claim 11, wherein comparing the modified images further comprises determining a set of image quality parameters for each respective compared rotation identification confidence values, the compared focus identification confidence values, the compared contrast identification confidences, and the compared color identification confidence values.

13. A method for stress testing a vehicle identification model, comprising:

one or more memory devices storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising;

receiving a plurality of first images of the vehicle, wherein the first images contain preset values of attributes, the attributes comprising at least one of a rotation attribute, a focus attribute, a contrast attribute, and a color attribute;

analyzing the first images using the vehicle identification model;

determining a first identification confidence value of the vehicle;

modifying the first set of images;

determining a second identification confidence value of the vehicle;

comparing the first and second identification confidence values; and

adjusting the model based on at least the comparison.

14. The method of claim 13, wherein modifying the first images comprises modifying values of a first one of the attributes of the first images.

15. The method of claim 14, wherein modifying the rotation attribute value of the first images further comprises:

adjusting the preset rotation attribute value of the first images to a second rotation attribute value;

analyzing the modified images, with the second rotation attribute value, with the vehicle identification model and determining a rotation identification confidence value; and

comparing the rotation identification confidence value to the first identification confidence value.

16. The method of claim 15, wherein modifying the focus attribute value of the first images further comprises:

adjusting the preset focus attribute value of the first images to a second focus attribute value;

analyzing the modified images, with the second focus attribute value, with the identification model and determining a focus identification confidence; and

comparing the focus identification confidence to the first identification confidence.

17. The method of claim 16, wherein modifying the contrast attribute value of the first images further comprises:

adjusting the preset contrast attribute value of the first images to a second contrast attribute value;

analyzing the modified images, with the second contrast attribute value, with the identification model and determining a contrast identification confidence; and

comparing the contrast identification confidence to the first identification confidence.

18. The method of claim 17, wherein modifying the color attribute value of the first images further comprises:

adjusting the preset color attribute value of the first images to a second contrast attribute value;

analyzing the modified images, with the second color attribute value, with the identification model and determining a color identification confidence; and

comparing the color identification confidence to the first identification confidence.

19. The method of claim 18, wherein comparing the modified images further comprises determining a set of image quality parameters for each respective compared rotation identification confidences, the compared focus identification confidences, the compared contrast identification confidences, and the compared color identification confidences.

20. The method of claim 19, wherein modifying the model further comprises incorporating the set of image quality parameters into the model.