Integrative cognition of driver behavior

• US 10,611,379 B2
• Filed: 08/16/2016
• Issued: 04/07/2020
• Est. Priority Date: 08/16/2016
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Inventors: Olabiyi, Oluwatobi, Yalla, Veeraganesh, Martinson, Eric
• Primary Examiner(s): Vincent, David R
• Application Number:
  US15/238,646
• Publication Number:
  US 20180053093A1
• Time in Patent Office: 1,330 Days
• Field of Search: 706 15, 706 45
• CPC Class Codes: B60W 2050/143 Alarm means
• CPC Class Codes: B60W 2050/146 Display means
• CPC Class Codes: B60W 40/09 Driving style or behaviour
• CPC Class Codes: B60W 50/0097 Predicting future conditions
• CPC Class Codes: B60W 50/14 Means for informing the driver, warning the driver or prompting a driver intervention
• CPC Class Codes: B60W 50/16 Tactile feedback to the driver, e.g. vibration or force feedback to the driver on the steering wheel or the accelerator pedal
• CPC Class Codes: G01C 21/3697 Output of additional, non-guidance related information, e.g. low fuel level G01C21/3679 takes precedence
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

By way of example, the technology disclosed by this document is capable of receiving signal data from one or more sensors; inputting the signal data into an input layer of a deep neural network (DNN), the DNN including one or more layers; generating, using the one or more layers of the DNN, one or more spatial representations of the signal data; generating, using one or more hierarchical temporal memories (HTMs) respectively associated with the one or more layers of the DNNs, one or more temporal predictions by the DNN based on the one or more spatial representations; and generating an anticipation of a future outcome by recognizing a temporal pattern based on the one or more temporal predictions.

Claims

1. A computer-implemented method comprising:

receiving, at one or more computing devices, signal data from one or more sensors, the one or more computing devices being electronically communicatively coupled to the one or more sensors to receive the signal data;

inputting the signal data into an input layer of a deep neural network (DNN), the DNN including one or more layers;

generating, using the one or more layers of the DNN, one or more spatial representations of the signal data;

generating, using one or more hierarchical temporal memories (HTMs) respectively associated with the one or more layers of the DNNs, one or more temporal predictions by the DNN based on the one or more spatial representations; and

generating an anticipation of a future outcome by recognizing a temporal pattern based on the one or more temporal predictions.

2. The computer-implemented method of claim 1, wherein recognizing the temporal pattern includes fusing the one or more temporal predictions.

3. The computer-implemented method of claim 1, wherein

the DNN includes a plurality of hierarchical layers including an input layer and one or more intermediate layers,

the input layer is coupled to an input layer HTM,

the one or more intermediate layers are coupled to one or more intermediate layer HTMs,

generating the one or more spatial representations of the signal data and generating the one or more temporal predictions based on the one or more spatial representations further include

generating, by the input layer of the DNN, a first spatial representation of the signal data,

generating, by the input layer HTM, a first temporal prediction using the first spatial representation of the signal data,

receiving, by the one or more intermediate layers of the DNN, the first temporal prediction and the first spatial representation of the signal data,

generating, by the one or more intermediate layers of the DNN, one or more intermediate spatial representations of the signal data, and

generating, by the one or more intermediate layer HTMs, one or more intermediate temporal predictions based on the one or more intermediate spatial representations of the signal data, and

recognizing the temporal pattern includes fusing the first temporal prediction and the one or more intermediate temporal predictions.

4. The computer-implemented method of claim 3, wherein the one or more intermediate layers of the DNN are hidden layers.

5. The computer-implemented method of claim 3, wherein

the DNN includes a recognition layer coupled to a recognition layer HTM, and

fusing the first temporal prediction and the one or more intermediate temporal predictions includes

fusing, at each HTM of the one or more intermediate layer HTMs and at the recognition layer, a current temporal prediction generated by the HTM and one or more prior temporal predictions generated by one or more prior intermediate layer HTMs and the input layer HTM.

6. The computer-implemented method of claim 3, wherein

receiving, at the one or more computing devices, the signal data from the one or more sensors includes receiving, at the one or more computing devices, two or more sets of signal data from two or more sets of sensors, respectively,

the DNN includes two or more input layers, the two or more input layers being associated with two or more input layer HTMs, respectively,

the one or more intermediate layers include two or more first intermediate layers associated with the two or more input layers, respectively, and a second intermediate layer after the two or more first intermediate layers,

the one or more intermediate layer HTMs include two or more intermediate layer HTMs coupled to the two or more first intermediate layers of the DNN,

generating, by the input layer of the DNN, the first spatial representation of the signal data includes generating, by the two or more input layers of the DNN, two or more first spatial representations of the signal data,

generating, by the input layer HTM, the first temporal prediction using the first spatial representation of the signal data includes generating, by the two or more input layer HTMs, two or more first temporal predictions,

generating, by the one or more intermediate layers of the DNN, the one or more intermediate spatial representations of the signal data includes fusing the two or more first intermediate spatial representations,

generating, by the one or more intermediate layer HTMs, the one or more intermediate temporal predictions based on the one or more intermediate spatial representations of the signal data includes fusing the two or more first temporal predictions, and recognizing the temporal pattern includes fusing a present temporal prediction with the fused two or more first temporal predictions.

7. The computer-implemented method of claim 1, wherein generating, using the one or more layers of the DNN, the one or more spatial representations of the signal data includes performing, by a successive layer of the DNN, a nonlinear transformation of each spatial representation generated by one or more prior layers of the DNN.

8. The computer-implemented method of claim 1, further comprising:

recognizing a spatial pattern correlated with a present outcome based on the one or more spatial representations.

9. The computer-implemented method of claim 8, wherein the present outcome is a present human user action.

10. The computer-implemented method of claim 8, wherein the signal data includes data describing characteristics of a vehicle in motion at a present time.

11. The computer-implemented method of claim 1, wherein the future outcome is a predicted future human user action.

12. The computer-implemented method of claim 1, further comprising:

computing, at a future time, an HTM prediction error for using

     1) the one or more temporal predictions; and

     2) one or more spatial representations of sensor data received at the future time; and

training the HTM based on the HTM prediction error.

13. The computer-implemented method of claim 1, further comprising:

computing, at a future time, a DNN prediction error for a recognition layer of the DNN using

     1) the anticipation; and

     2) a recognized spatial pattern recognized by the recognition layer of the DNN based on the one or more spatial representations of sensor data received at a future time; and

training the DNN based on the DNN prediction error.

14. The computer-implemented method of claim 1, wherein the signal data includes one or more of image data, audio data, vehicle data from a Controlled Area Network (CAN) of the vehicle, electronic messaging data, and navigation data associated with a moving platform.

15. The computer-implemented method of claim 1, further comprising:

generating and providing one or more of an auditory instruction, a visual instruction, and a tactile instruction to the user via one or more output devices of a moving platform based on the anticipation of the outcome.

16. A computer system comprising:

one or more sensors providing sensor data;

one or more non-transitory computer memories for storing and providing access to data;

one or more computer processors coupled to the non-transitory computer memories to store and receive data;

a deep neural network (DNN) storable by the one or more non-transitory computer memories and executable by the one or more computer processors, the DNN including a plurality of hierarchical layers, the plurality of hierarchical layers including an input layer, one or more intermediate layers, and a recognition layer, the input layer of the DNN receiving sensor data from the one or more sensors; and

a plurality of hierarchical temporal memories (HTMs) storable by the one or more non-transitory computer memories and executable by the one or more computer processors, the HTMs including an input layer HTM, one or more intermediate layer HTMs, and a recognition layer HTM,

the input layer HTM being coupled to the input layer of the DNN and executable by the one or more computer processors to receive a spatial representation of the signal data,

the input layer HTM is executable by the one or more computer processors to generate an input layer temporal prediction using the spatial representation of the signal data,

the one or more intermediate layer HTMs being coupled to the one or more intermediate layers of the DNN and executable by the one or more computer processors to receive one or more intermediate layer spatial representations from the one or more intermediate layers of the DNN,

the one or more intermediate layer HTMs executable by the one or more computer processors to generate one or more intermediate layer temporal predictions based on the one or more intermediate layer spatial representations,

the recognition layer HTM being coupled to the recognition layer of the DNN and executable by the one or more computer processors to receive a recognition layer spatial representation from the recognition layer of the DNN,

the recognition layer HTM executable by the one or more computer processors to generate a recognition layer temporal prediction based on the recognition layer spatial representation, and

the recognition layer HTM executable by the one or more computer processors to generate an anticipation of a future outcome by recognizing a temporal pattern based on the input layer temporal prediction, the one or more intermediate layer temporal predictions, and the recognition layer temporal prediction.

17. A computer system comprising:

one or more computer processors; and

one or more non-transitory memories storing instructions that, when executed by the one or more computer processors, cause the computer system to perform operations comprising;

receiving signal data from one or more sensors, the one or more computing devices being electronically communicatively coupled to the one or more sensors to receive the signal data,

inputting the signal data into an input layer of a deep neural network (DNN), the DNN including one or more layers,

generating, using the one or more layers of the DNN, one or more spatial representations of the signal data,

generating, using one or more hierarchical temporal memories (HTMs) respectively associated with the one or more layers of the DNNs, one or more temporal predictions by the DNN based on the one or more spatial representations, and

generating an anticipation of a future outcome by recognizing a temporal pattern based on the one or more temporal predictions.

18. The system of claim 17, wherein recognizing the temporal pattern includes fusing the one or more temporal predictions.

19. The system of claim 17, wherein

the DNN includes a plurality of hierarchical layers including an input layer and one or more intermediate layers,

the input layer is coupled to an input layer HTM;

the one or more intermediate layers are coupled to one or more intermediate layer HTMs,

generating the one or more spatial representations of the signal data and generating the one or more temporal predictions based on the one or more spatial representations further include

generating, by the input layer of the DNN, a first spatial representation of the signal data,

passing the first spatial representation to the input layer HTM,

generating, by the input layer HTM, a first temporal prediction using the first spatial representation of the signal data,

receiving, by the one or more intermediate layers of the DNN, the first temporal prediction and the first spatial representation of the signal data, and

generating, by the one or more intermediate layers of the DNN, one or more intermediate spatial representations of the signal data,

generating, by the one or more intermediate layer HTMs, one or more intermediate temporal predictions based on the one or more intermediate spatial representations of the signal data, and

recognizing the temporal pattern includes fusing the first temporal prediction and the one or more intermediate temporal predictions.

20. The system of claim 19, wherein the one or more intermediate layers of the DNN are hidden layers.

21. The system of claim 19, wherein

the DNN includes a recognition layer coupled to a recognition layer HTM, and

fusing the first temporal prediction and the one or more intermediate temporal predictions includes

fusing, at each HTM of the one or more intermediate layer HTMs and at the recognition layer, a current temporal prediction generated by the HTM and one or more prior temporal predictions generated by one or more prior intermediate layer HTMs and the input layer HTM.

22. The system of claim 19, wherein

receiving, at the one or more computing devices, the signal data from the one or more sensors includes receiving, at the one or more computing devices, two or more sets of signal data from two or more sets of sensors, respectively,

the DNN includes two or more input layers, the two or more input layers being associated with two or more input layer HTMs, respectively,

the one or more intermediate layers include two or more first intermediate layers associated with the two or more input layers, respectively, and a second intermediate layer after the two or more first intermediate layers,

the one or more intermediate layer HTMs include two or more intermediate layer HTMs coupled to the two or more first intermediate layers of the DNN,

generating, by the input layer of the DNN, a first spatial representation of the signal data includes generating, by the two or more input layers of the DNN, two or more first spatial representations of the signal data,

generating, by the input layer HTM, a first temporal prediction using the first spatial representation of the signal data includes generating, by the two or more input layer HTMs, two or more first temporal predictions,

generating, by the one or more intermediate layers of the DNN, one or more intermediate spatial representations of the signal data includes fusing the two or more first intermediate spatial representations,

generating, by the one or more intermediate layer HTMs, one or more intermediate temporal predictions based on the one or more intermediate spatial representations of the signal data includes fusing the two or more first temporal predictions, and

recognizing the temporal pattern includes fusing a present temporal prediction with the fused two or more first temporal predictions.

23. The system of claim 17, wherein generating, using the one or more layers of the DNN, the one or more spatial representations of the signal data includes performing, by a successive layer of the DNN, a nonlinear transformation of each spatial representation generated by one or more prior layers of the DNN.

24. The system of claim 17, further comprising:

recognizing a spatial pattern correlated with a present outcome based on the one or more spatial representations.

25. The system of claim 24, wherein the present outcome is a present human user action.

26. The system of claim 24, wherein the signal data includes data describing characteristics of a vehicle in motion at a present time.

27. The system of claim 17, wherein the future outcome is a predicted future human user action.

28. The system of claim 17, further comprising:

computing, at a future time, an HTM prediction error for using

     1) the one or more temporal predictions; and

     2) one or more spatial representations of sensor data received at the future time; and

training the HTM based on the HTM prediction error.

29. The system of claim 17, further comprising:

computing, at a future time, an DNN prediction error for a recognition layer of the DNN using

     1) the anticipation; and

     2) a recognized spatial pattern recognized by the recognition layer of the DNN based on the one or more spatial representations of sensor data received at a future time; and

training the DNN based on the DNN prediction error.

30. The system of claim 17, wherein the signal data includes one or more of image data, audio data, vehicle data from a Controlled Area Network (CAN) of the vehicle, electronic messaging data, and navigation data associated with a moving platform.

31. The system of claim 17, further comprising:

generating and providing one or more of an auditory instruction, a visual instruction, and a tactile instruction to the user via one or more output devices of a vehicle based on the anticipation of the outcome.