Real-time multiclass driver action recognition using random forests

• US 9,501,693 B2
• Filed: 10/09/2013
• Issued: 11/22/2016
• Est. Priority Date: 10/09/2013
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Sarratt, Trevor, Fujimura, Kikuo
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): LEMIEUX, IAN L
• Application Number:
  US14/050,259
• Publication Number:
  US 20150098609A1
• Time in Patent Office: 1,140 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An action recognition system recognizes driver actions by using a random forest model to classify images of the driver. A plurality of predictions is generated using the random forest model. Each prediction is generated by one of the plurality of decision trees and each prediction comprises a predicted driver action and a confidence score. The plurality of predictions is regrouped into a plurality of groups with each of the plurality of groups associated with one of the driver actions. The confidence scores are combined within each group to determine a combined score associated with each group. The driver action associated with the highest combined score is selected.

Claims

1. A method for performing action recognition on an image of a driver in a vehicle, the method comprising:

receiving, by a computing system, an image of the driver in the vehicle;

accessing a random forest model comprising a plurality of decision trees;

generating a plurality of predictions of the action being performed by the driver in the image through the random forest model, each prediction generated by one of the plurality of decision trees, each of the plurality of predictions comprising a predicted driver action and a confidence score comprising a ratio or percentage;

grouping the plurality of generated predictions into a plurality of groups by the predicted driver action, such that each group of the plurality of groups is associated with a single predicted driver action;

combining the confidence scores of the generated predictions for each group to determine a single combined score for each group relating to the predicted driver action associated with each group; and

selecting the driver action associated with a highest combined confidence score from the plurality of groups.

2. The method of claim 1, wherein combining the confidence scores comprises adding the confidence scores.

3. The method of claim 1, wherein generating the plurality of predictions through the random forest model comprises:

for a first decision tree in the plurality of decision trees, applying a first test to the image based on first test parameters of a parent branch node of the first decision tree in the random forest, the parent branch node having a plurality of child nodes;

selecting one of the child nodes of the parent branch node based on a result of the test;

responsive to the selected one of the child nodes being a branch node, applying a second test to the image based on second test parameters associated with the selected one of the child nodes;

responsive to the selected one of the child nodes being a leaf node, generating the prediction, the generated prediction comprising the driver action and the confidence score associated with the leaf node.

4. The method of claim 3, wherein applying the first test based on the first test parameters comprises:

selecting a plurality of spatial regions of the image;

selecting a feature channel representing features of the image; and

selecting a threshold value.

5. The method of claim 4, wherein the image comprises a three-dimensional image and wherein the selected feature channel comprises three-dimensional depth data.

6. The method of claim 4, wherein applying the first test comprises:

determining a difference between average values of the selected feature channels of at

least two of the selected plurality of spatial regions; and

comparing the difference to the threshold value.

7. The method of claim 1, comprising:

delaying a notification provided from an in-vehicle system based on the predicted driver action.

8. The method of claim 1, wherein the predicted driver action comprises at least one of:

normal driving, reaching for the center compartment, reaching for a glove compartment, reaching for an overhead compartment, adjusting a radio, talking on a phone, and adjusting a mirror.

9. The method of claim 1, wherein the random forest model is learned based on a set of labeled training images.

10. A non-transitory computer-readable storage medium storing instructions for performing action recognition on an image of a driver in a vehicle, the instructions when executed by a processor causing the processor to perform steps including:

receiving, by a computing system, an image of the driver in the vehicle;

accessing a random forest model comprising a plurality of decision trees;

generating a plurality of predictions through the random forest model, each prediction generated by one of the plurality of decision trees, each of the plurality of predictions comprising a predicted driver action and a confidence score comprising a ratio or percentage;

grouping the plurality of generated predictions into a plurality of groups by the predicted driver action, such that each group of the plurality of groups is associated with a single predicted driver action;

combining the confidence scores of the generated predictions for each group to determine a single combined score for each group relating to the predicted driver action associated with each group; and

selecting the driver action associated with a highest combined confidence score from the plurality of groups.

11. The non-transitory computer-readable storage medium of claim 10, wherein combining the confidence scores comprises adding the confidence scores.

12. The non-transitory computer-readable storage medium of claim 10, wherein generating the plurality of predictions through the random forest model comprises:

for a first decision tree in the plurality of decision trees, applying a first test to the image based on first test parameters of a parent branch node of the first decision tree in the random forest, the parent branch node having a plurality of child nodes;

selecting one of the child nodes of the parent branch node based on a result of the test;

responsive to the selected one of the child nodes being a branch node, applying a second test to the image based on second test parameters associated with the selected one of the child nodes;

responsive to the selected one of the child nodes being a leaf node, generating the prediction, the generated prediction comprising the driver action and the confidence score associated with the leaf node.

13. The non-transitory computer-readable storage medium of claim 12, wherein applying the first test based on the first test parameters comprises:

selecting a plurality of spatial regions of the image;

selecting a feature channel representing features of the image; and

selecting a threshold value.

14. The non-transitory computer-readable storage medium of claim 13,

wherein the image comprises a three-dimensional image and wherein the selected feature channel comprises three-dimensional depth data.

15. The non-transitory computer-readable storage medium of claim 13, wherein applying the first test comprises:

determining a difference between average values of the selected feature channels of at least two of the selected plurality of spatial regions; and

comparing the difference to the threshold value.

16. A method for learning a random forest model for action recognition, the random forest model comprising a plurality of decision trees, the method comprising:

receiving, by a computing system, a plurality of training images, each training image depicting a driver action being performed inside a vehicle and each training image having a label identifying the driver action being performed;

generating a test corresponding to a parent node of one of the plurality of decision trees, the test comprising one or more test parameters;

applying the test to each training image to classify each training image into a plurality of image groups including at least a first image group and a second image group;

determining if an entropy value of the first image group is below a threshold value;

responsive to a determination that the entropy value of the first image group is below the threshold value, generating a prediction based on the labels associated with the first image group, the prediction comprising a driver action and a confidence score comprising a ratio or percentage, and generating a leaf node associated with the prediction as a child node of the parent node; and

responsive to determining that the entropy value of the first image group is not below the threshold value, generating a branch node associated with the first image group as a child node of the parent node;

wherein

the generated prediction is grouped into one of a plurality of groups by the prediction, such that each group of the plurality of groups is associated with a single prediction.

17. A method as claimed in claim 16, wherein generating the test comprises:

selecting a plurality of spatial regions;

selecting a feature channel; and

selecting a threshold value.

18. The method of claim 17, wherein applying the test comprises:

for a given training image, determining an average value for the feature channel of each of the plurality of spatial regions;

determining a difference between the average values of a first spatial region and a second spatial region in the plurality of spatial regions;

comparing the difference to the threshold value to generate a comparison result; and

classifying the given training image into one of the plurality of image groups based on the comparison result.

19. The method as claimed in claim 18, wherein generating the leaf node comprises:

determining a driver action associated with a highest number of images in the first image group; and

calculating the confidence score as a percentage of the images in the first image group associated with the driver action.

20. The method of claim 16, wherein the generating the test comprises:

applying a first sample test;

determining a first information gain value for the first sample test, the information gain value representing a difference between an entropy value of the plurality of training images and a combined entropy value of the first and second image groups;

applying a second sample test;

determining a second information gain value for the second sample test; and

identifying the test having a highest information gain value.