Object Classification in Image Data Using Machine Learning Models

• US 20180150713A1
• Filed: 11/29/2016
• Published: 05/31/2018
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Farooqi, Waqas Ahmad, Lipps, Jonas, Schmidt, Eckehard, Fricke, Thomas, Verzano, Nemrude
• Granted Patent: US 10,289,925 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Combined color and depth data for a field of view is received. Thereafter, using at least one bounding polygon algorithm, at least one proposed bounding polygon is defined for the field of view. It can then be determined, using a binary classifier having at least one machine learning model trained using a plurality of images of known objects, whether each proposed bounding polygon encapsulates an object. The image data within each bounding polygon that is determined to encapsulate an object can then be provided to a first object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon. Further, the image data within each bounding polygon that is determined to encapsulate an object is provided to a second object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon. A final classification for each bounding polygon is then determined based on the output of the first classifier machine learning model and the output of the second classifier machine learning model.

Claims

1. A method for implementation by one or more data processors forming part of at least one computing system, the method comprising:

receiving combined color and depth data for a field of view;

defining, using at least one bounding polygon algorithm, at least one proposed bounding polygon for the field of view;

determining, using a binary classifier having at least one machine learning model trained using a plurality of images of known objects, whether each proposed bounding polygon encapsulates an object;

providing the image data within each bounding polygon that is determined to encapsulate an object to a first object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon;

providing the image data within each bounding polygon that is determined to encapsulate an object to a second object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon;

determining a final classification for each bounding polygon based on the output of the first classifier machine learning model and the output of the second classifier machine learning model; and

providing data characterizing the final classification for each bounding polygon.

2. The method of claim 1, wherein the at least one first classifier machine learning model is a region and measurements-based convolutional neural network.

3. The method of claim 1, wherein the combined color and depth image data is RGB-D data.

4. The method of claim 1, wherein the first object classifier uses metadata characterizing each object.

5. The method of claim 4, wherein the metadata is extracted from the combined color and image data.

6. The method of claim 1, wherein the at least one machine learning model of the binary classifier is one or more of:

a neural network, a convolutional neural network, a logistic regression model, a support vector machine, decision trees, ensemble model, k-nearest neighbors model, linear regression model, naï

ve Bayes model, a logistic regression model, and/or a perceptron model.

7. The method of claim 1, wherein the at least one machine learning model of the first object classifier is one or more of:

a neural network, a convolutional neural network, a logistic regression model, a support vector machine, decision trees, ensemble model, k-nearest neighbors model, linear regression model, naï

ve Bayes model, a logistic regression model, and/or a perceptron model.

8. The method of claim 1, wherein the at least one machine learning model of the second object classifier is one or more of:

a neural network, a convolutional neural network, a logistic regression model, a support vector machine, decision trees, ensemble model, k-nearest neighbors model, linear regression model, naï

ve Bayes model, a logistic regression model, and/or a perceptron model.

9. The method of claim 1 further comprising:

discarding proposed bounding polygons determined, by the binary classifier, to not include an object.

10. The method of claim 1, wherein at least one of the binary classifier, the first object classifier, or the second object classifier utilizes a plurality of machine learning models which are selected and utilized based on a type of object encapsulated within the corresponding bounding polygon.

11. The method of claim 1, wherein the providing data characterizing the final classification for each bounding polygon comprises at least one of:

displaying the data characterizing the final classification for each bounding polygon in an electronic visual display, loading the data characterizing the final classification for each bounding polygon into memory, storing the data characterizing the final classification for each bounding polygon in persistence, or transmitting the data characterizing the final classification for each bounding polygon to a remote computing device.

12. A method for implementation by one or more data processors forming part of at least one computing device, the method comprising:

receiving RGB-data for a field of view;

defining, using at least one bounding polygon algorithm, at least one bounding polygon for the field of view;

determining, using a binary classifier machine learning model trained using a plurality of images of known objects, whether each bounding polygon encapsulates one of the known objects;

providing the image data within each bounding polygon that is determined to encapsulate one of the known objects to a plurality of classifier machine learning models trained using a plurality of images of known objects, to classify the known objects; and

providing data characterizing the classification of the known objects.

13. The method of claim 12, wherein the plurality of classifier machine learning models to which the image data is provided are selected based on metadata associated with the RGB-data.

14. The method of claim 12, wherein the metadata associated with the RGB-data acts as a pre-classifier.

15. The method of claim 12, wherein the RGB data is RGB-D data.

16. A system comprising:

at least one data processor; and

memory storing instructions which, when executed by the at least one data processor, result in operations comprising;

receiving combined color and depth data for a field of view;

defining, using at least one bounding polygon algorithm, at least one proposed bounding polygon for the field of view;

determining, using a binary classifier having at least one machine learning model trained using a plurality of images of known objects, whether each proposed bounding polygon encapsulates an object;

providing the image data within each bounding polygon that is determined to encapsulate an object to a first object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon;

providing the image data within each bounding polygon that is determined to encapsulate an object to a second object classifier having at least one machine learning model trained using a plurality of images of known objects, to classify the object encapsulated within the respective bounding polygon;

determining a final classification for each bounding polygon based on the output of the first classifier machine learning model and the output of the second classifier machine learning model; and

providing data characterizing the final classification for each bounding polygon.

17. The system of claim 16, wherein the at least one first classifier machine learning model is a region and measurements-based convolutional neural network.

18. The system of claim 16, wherein the combined color and depth image data is RGB-D data.

19. The system of claim 16, wherein the first object classifier uses metadata characterizing each object, the metadata being extracted from the combined color and image data.

20. The system of claim 16, wherein the at least one machine learning model of the binary classifier is one or more of:

a neural network, a convolutional neural network, a logistic regression model, a support vector machine, decision trees, ensemble model, k-nearest neighbors model, linear regression model, naï

ve Bayes model, a logistic regression model, and/or a perceptron model.