System and method of predicting human interaction with vehicles

• US 10,402,687 B2
• Filed: 12/04/2017
• Issued: 09/03/2019
• Est. Priority Date: 07/05/2017
• Status: Active Grant

• Current Assignee: Perceptive Automata, Inc.
• Original Assignee: Perceptive Automata, Inc.
• Inventors: Anthony, Samuel English, Misra, Kshitij, Faller, Avery Wagner
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US15/830,549
• Publication Number:
  US 20190012574A1
• Time in Patent Office: 638 Days
• Field of Search: 382104, 382103, 382110, 382128, 702 19, 738654, 348135, 348139, 386278
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 60/00274 considering possible movement changes
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06V 10/7784 based on feedback from supervisors
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08G 1/0112 from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/012 from other sources than vehicle or roadside beacons, e.g. mobile networks
• CPC Class Codes: G08G 1/0129 for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0133 for classifying traffic situation
• CPC Class Codes: G08G 1/0141 for traffic information dissemination
• CPC Class Codes: G08G 1/04 using optical or ultrasonic detectors
• CPC Class Codes: G08G 1/164 : Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Systems and methods for predicting user interaction with vehicles. A computing device receives an image and a video segment of a road scene, the first at least one of an image and a video segment being taken from a perspective of a participant in the road scene and then generates stimulus data based on the image and the video segment. Stimulus data is transmitted to a user interface and response data is received, which includes at least one of an action and a likelihood of the action corresponding to another participant in the road scene. The computing device aggregates a subset of the plurality of response data to form statistical data and a model is created based on the statistical data. The model is applied to another image or video segment and a prediction of user behavior in the another image or video segment is generated.

Claims

1. A computerized method of predicting user interaction with vehicles enabling motorists, cyclists, and pedestrians to anticipate the actions of other motorists, cyclists, and pedestrians, the computerized method comprising:

receiving, by a computing device, a first at least one of an image and a video segment of a road scene, the first at least one of an image and a video segment being taken from a perspective of a participant in the road scene, the first at least one of an image and a video segment including at least one of a pedestrian, a cyclist, and a motor vehicle;

generating, by the computing device, stimulus data based on the first at least one of the image and the video segment, the stimulus data comprising at least one of;

the first at least one of the image and the video segment, and

an altered version of the first at least one of the image and the video segment;

transmitting, by the computing device, the stimulus data to a user interface;

receiving, by the computing device, a plurality of response data from the user interface, the response data including user inputted data by associated with the stimulus data, the user inputted data including at least one of an action and a likelihood of the action corresponding to at least one of the pedestrian, the cyclist, and the motor vehicle associated with the road scene;

aggregating, by the computing device, a subset of the plurality of response data corresponding to one of the first at least one of the image and the video segment to form statistical data;

creating, by the computing device, a model based on the statistical data;

applying, by the computing device, the model to a second at least one image or video segment; and

generating, by the computing device, a prediction of user behavior in the second at least one image or video segment based on the application of the model to the second at least one image or video segment.

2. The computerized method of claim 1, wherein creating the model further comprises training, by the computing device, a supervised learning algorithm, the supervised learning algorithm including at least one of a random forest regressor, a support vector regressor, a simple neural network, a deep convolutional neural network, a recurrent neural network, and a long-short-term memory (LSTM) neural network.

3. The computerized method of claim 1, wherein generating the stimulus data further comprises:

manipulating at least one of pixel data or array data associated with the first at least one of the image and the video segment; and

creating a manipulated data file including the manipulated at least one of pixel data or array data associated with the first at least one of the image and the video segment.

4. The computerized method of claim 3, wherein the manipulated data file includes a looped version of extracted frames from the first at least one of the image and the video segment.

5. The computerized method of claim 1, wherein:

the action includes one of the at least one of the pedestrian, the cyclist, and the motor vehicle staying in place, changing lanes, and crossing a street; and

the likelihood of the action includes an ordinal value associated with a probability of the action.

6. The computerized method of claim 1, wherein the statistical data is associated with a parameter of the subset of the plurality of response data, the parameter including at least one of a content of a response, a time associated with entering a response, and a position of an eye of a human observer associated with the response, the position being measured with respect to a display associated with the user interface.

7. The computerized method of claim 6, wherein the parameter is further associated with at least one of a central tendency, a variance, a skew, a kurtosis, a scale, and a histogram.

8. The computerized method of claim 1, wherein the second at least one image or video segment is associated with a live road scene.

9. The computerized method of claim 8, further comprising receiving, by the computing device, the second at least one image or video segment from another computing device associated with a vehicle participating in the live road scene.

10. The computerized method of claim 1, wherein the second at least one image or video segment is the same as the first at least one image or video segment, the method further comprising:

comparing, by the computing device, the prediction of user behavior with the statistical data to generate an error signal; and

adjusting, by the computing device, at least one weight associated with the model based on the error signal.

11. A computing system for predicting user interaction with vehicles enabling motorists, cyclists, and pedestrians to anticipate the actions of other motorists, cyclists, and pedestrians, the computing system comprising a memory containing instructions for execution by a processor, the processor configured to:

receive a first at least one of an image and a video segment of a road scene, the first at least one of an image and a video segment being taken from a perspective of a participant in the road scene, the first at least one of an image and a video segment including at least one of a pedestrian, a cyclist, and a motor vehicle;

generate stimulus data based on the first at least one of the image and the video segment, the stimulus data comprising at least one of;

the first at least one of the image and the video segment, and

an altered version of the first at least one of the image and the video segment;

transmit the stimulus data to a user interface;

receive a plurality of response data from the user interface, the response data including user inputted data by associated with the stimulus data, the user inputted data including at least one of an action and a likelihood of the action corresponding to at least one of the pedestrian, the cyclist, and the motor vehicle associated with the road scene;

aggregate a subset of the plurality of response data corresponding to one of the first at least one of the image and the video segment to form statistical data;

create a model based on the statistical data;

apply the model to a second at least one image or video segment; and

generate a prediction of user behavior in the second at least one image or video segment based on the application of the model to the second at least one image or video segment.

12. The computing system of claim 11, wherein to create the model the processor is further configured to train a supervised learning algorithm, the supervised learning algorithm including at least one of a random forest regressor, a support vector regressor, a simple neural network, a deep convolutional neural network, a recurrent neural network, and a long-short-term memory (LSTM) neural network.

13. The computing system of claim 11, wherein to generate the stimulus data, the processor is further configured to:

manipulate at least one of pixel data or array data associated with the first at least one of the image and the video segment; and

create a manipulated data file including the manipulated at least one of pixel data or array data associated with the first at least one of the image and the video segment.

14. The computing system of claim 11, wherein the manipulated data file includes a looped version of extracted frames from the first at least one of the image and the video segment.

15. The computing system of claim 11, wherein:

the action includes one of the at least one of the pedestrian, the cyclist, and the motor vehicle staying in place, changing lanes, and crossing a street; and

the likelihood of the action includes an ordinal value associated with a probability of the action.

16. The computing system of claim 11, wherein the statistical data is associated with a parameter of the subset of the plurality of response data, the parameter including at least one of a content of a response, a time associated with entering a response, and a position of an eye of a human observer associated with the response, the position being measured with respect to a display associated with the user interface.

17. The computing system of claim 16, wherein the parameter is further associated with at least one of a central tendency, a variance, a skew, a kurtosis, a scale, and a histogram.

18. The computing system of claim 11, wherein the second at least one image or video segment is associated with a live road scene.

19. The computing system of claim 18, wherein the processor is further configured to receive the second at least one image or video segment from a computing device associated with a vehicle participating in the live road scene.

20. The computing system of claim 11, wherein the second at least one image or video segment is the same as the first at least one image or video segment, and wherein the processor is further configured to:

compare the prediction of user behavior with the statistical data to generate an error signal; and

adjust at least one weight associated with the model based on the error signal.