Generating a machine learning model for objects based on augmenting the objects with physical properties

• US 10,565,475 B2
• Filed: 04/24/2018
• Issued: 02/18/2020
• Est. Priority Date: 04/24/2018
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Lecue, Freddy, Oliveira Antonino, Victor, Hamiti, Sofian, Kaila, Gaurav
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US15/961,392
• Publication Number:
  US 20190325265A1
• Time in Patent Office: 665 Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/0464 Convolutional networks [CNN, ConvNet]
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/126 Evolutionary algorithms, e.g. genetic algorithms or genetic programming
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

A device receives images of a video stream, models for objects in the images, and physical property data for the objects, and maps the models and the physical property data to the objects in the images to generate augmented data sequences. The device applies different physical properties to the objects in the augmented data sequences to generate augmented data sequences with different applied physical properties, and trains a machine learning (ML) model based on the images to generate a first trained ML model. The device trains the ML model, based on the augmented data sequences with the different applied physical properties, to generate a second trained ML model, and compares the first trained ML model and the second trained ML model. The device determines whether the second trained ML model is optimized based on the comparison, and provides the second trained ML model when optimized.

Claims

1. A device, comprising:

one or more memories; and

one or more processors, communicatively coupled to the one or more memories, to;

receive images of a video stream, three-dimensional models for objects in the images, and physical property data for the objects;

map the three-dimensional models and the physical property data to the objects in the images to generate augmented data sequences with the objects;

apply different physical properties, of the physical property data, to the objects in the augmented data sequences, based on an augmentation policy, to generate augmented data sequences with different applied physical properties;

train a machine learning model based on the images of the video stream to generate a first trained machine learning model;

train the machine learning model, based on the augmented data sequences with the different applied physical properties, to generate a second trained machine learning model;

compare the first trained machine learning model and the second trained machine learning model;

determine whether the second trained machine learning model is optimized based on a result of comparing the first trained machine learning model and the second trained machine learning model; and

provide the second trained machine learning model and the different applied physical properties when the second trained machine learning model is optimized.

2. The device of claim 1, wherein the one or more processors are further to:

modify the different applied physical properties when the second trained machine learning model is not optimized;

retrain the machine learning model, based on the modified different applied physical properties, to generate the second trained machine learning model; and

repeat the modifying the different applied physical properties and the retraining until the second trained machine learning model is optimized.

3. The device of claim 1, wherein the one or more processors are further to:

utilize the second trained machine learning model and the different applied physical properties, when the second trained machine learning model is optimized, to predict an unknown object.

4. The device of claim 1, wherein the one or more processors are further to:

receive the machine learning model and the augmentation policy,

wherein the augmentation policy includes information indicating how the different physical properties are to be applied to each of the augmented data sequences.

5. The device of claim 1, wherein the machine learning model includes one or more of:

a single shot multibox detector (SSD) model,

a region-based fully convolutional network (R-FCN) model,

a region-based convolution network (R-CNN) model,

a fast R-CNN model, or

a faster R-CNN model.

6. The device of claim 1, wherein the one or more processors are further to:

modify the different applied physical properties, when the second trained machine learning model is not optimized, based on a hyperparameter optimization technique,

wherein the hyperparameter optimization technique includes one or more of;

a grid search technique,

a random search technique,

a Bayesian optimization technique,

a gradient-based optimization technique, or

an evolutionary optimization technique.

7. The device of claim 1, wherein, the one or more processors are further to:

test the first trained machine learning model to generate first test results;

test the second trained machine learning model to generate second test results;

compare the first test results and the second test results; and

determine whether the second trained machine learning model is optimized based on a result of comparing the first test results and the second test results.

8. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed by one or more processors, cause the one or more processors to;

receive images of a video stream, three-dimensional models for objects in the images, and physical property data for the objects,

the images of the video stream including metadata that identifies at least two of;

the images of the video stream,

the objects in the images,

classes associated with the objects,

boundary boxes for the images,

coordinates associated with the objects in the images, or

names of the objects,

the three-dimensional models including at least two of;

three-dimensional representations of the objects,

three-dimensional coordinates associated with the objects,

normal vectors associated with the objects, or

the names of the objects,

the physical property data including at least two of;

the names of the objects,

information associated with deformations of the objects,

information associated with gravities for the objects,

information associated with rotations of the objects,

information associated with renderings of the objects, or

information associated with collisions of the objects;

map the three-dimensional models and the physical property data to the objects in the images to generate augmented data sequences with the objects;

apply different physical properties, of the physical property data, to the objects in the augmented data sequences to generate augmented data sequences with different applied physical properties;

train a machine learning model based on the images of the video stream to generate a first machine learning model;

train the machine learning model, based on the augmented data sequences with the different applied physical properties, to generate a second machine learning model;

test the first machine learning model and the second machine learning model to generate first test results and second test results, respectively;

determine whether the second machine learning model is optimized based on comparing the first test results and the second test results; and

utilize the second machine learning model and the different applied physical properties, when the second machine learning model is optimized, to make a prediction.

9. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise:

one or more instructions that, when executed by the one or more processors, cause the one or more processors to;

provide the second machine learning model and the different applied physical properties when the second machine learning model is optimized.

10. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise:

one or more instructions that, when executed by the one or more processors, cause the one or more processors to;

modify the different applied physical properties when the second machine learning model is not optimized;

retrain the machine learning model, based on the modified different applied physical properties, to generate the second machine learning model;

retest the second machine learning model to generate the second test results; and

repeat the modifying the different applied physical properties, the retraining, and the retesting until the second machine learning model is optimized.

11. The non-transitory computer-readable medium of claim 8, wherein the different applied physical properties are configurable.

12. The non-transitory computer-readable medium of claim 8, wherein each of the first machine learning model and second machine learning model includes one or more of:

a single shot multibox detector (SSD) model,

a region-based fully convolutional network (R-FCN) model,

a region-based convolution network (R-CNN) model,

a fast R-CNN model, or

a faster R-CNN model.

13. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise:

one or more instructions that, when executed by the one or more processors, cause the one or more processors to;

modify the different applied physical properties, when the second machine learning model is not optimized, based on one or more of;

a grid search technique,

a random search technique,

a Bayesian optimization technique,

a gradient-based optimization technique, or

an evolutionary optimization technique.

14. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise:

one or more instructions that, when executed by the one or more processors, cause the one or more processors to;

determine that the second machine learning model is optimized when the second test results are within a predetermined threshold of the first test results.

15. A method, comprising:

receiving, by a device, images of a video stream, three-dimensional models for objects in the images, and physical property data for the objects;

associating, by the device, the three-dimensional models and the physical property data with the objects in the images to generate augmented data sequences with the objects;

receiving, by the device, an augmentation policy;

applying, by the device and based on an augmentation policy, different physical properties, of the physical property data, to the objects in the augmented data sequences in order to generate augmented data sequences with different applied physical properties;

training, by the device, a machine learning model based on the images of the video stream to generate a first trained machine learning model;

training, by the device, the machine learning model, based on the augmented data sequences with the different applied physical properties, to generate a second trained machine learning model;

testing, by the device, the first trained machine learning model and the second trained machine learning model to generate first test results and second test results, respectively;

determining, by the device, whether the second trained machine learning model is optimized based on whether the second test results are within a predetermined threshold of the first test results; and

providing, by the device, the second trained machine learning model and the different applied physical properties when the second trained machine learning model is optimized.

16. The method of claim 15, further comprising:

receiving an unknown image with an unknown object; and

utilizing the second trained machine learning model and the different applied physical properties, when the second trained machine learning model is optimized, to identify the unknown object.

17. The method of claim 15, further comprising:

modifying the different applied physical properties when the second trained machine learning model is not optimized;

retraining the machine learning model, based on the modified different applied physical properties, to generate an updated second trained machine learning model;

retesting the updated second trained machine learning model to generate updated second test results; and

repeating the modifying the different applied physical properties, the retraining, and the retesting until the second trained machine learning model is optimized.

18. The method of claim 15, wherein the augmentation policy includes configurable information indicating how the different physical properties are to be applied to each of the augmented data sequences.

19. The method of claim 15, wherein the machine learning model includes an object detection deep learning model.

20. The method of claim 15, further comprising:

modifying the different applied physical properties, when the second trained machine learning model is not optimized, based on a hyperparameter optimization technique.