Method and device for producing vehicle operational data based on deep learning techniques

• US 10,139,823 B2
• Filed: 09/13/2016
• Issued: 11/27/2018
• Est. Priority Date: 09/13/2016
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Prokhorov, Danil V., Derry, Matthew O.
• Primary Examiner(s): Ramesh, Krishnan
• Application Number:
  US15/263,963
• Publication Number:
  US 20180074493A1
• Time in Patent Office: 805 Days
• Field of Search: 701 25, 701 23
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2520/10 Longitudinal speed
• CPC Class Codes: B60W 2520/105 Longitudinal acceleration
• CPC Class Codes: B60W 2540/18 Steering angle
• CPC Class Codes: B60W 2556/05 Big data
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 30/00 Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G08G 1/096811 where the route is computed offboard
• CPC Class Codes: G08G 1/0969 having a display in the form of a map
• CPC Class Codes: G08G 1/164 Centralised systems, e.g. external to vehicles
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

A method and device for effecting vehicle control is presented. The method includes receiving driving map data, which includes vehicle target data relative to a vehicle location data. The driving map data is processed to produce desired vehicle operational data, the desired vehicle operational data facilitates a vehicle to traverse a travel route. From the desired vehicle operational data vehicle, corresponding actuator control data is produced, and transmitted to effect vehicle control in either of the autonomous or driver-assisted modes.

Claims

1. A method in a vehicle control unit for controlling a vehicle, the method comprising:

receiving driving map data that is a composition of separate temporal observations within a single composite image including vehicle target data relative to a vehicle location data and obstacle data that characterizes temporal aspects of dynamic obstacles;

processing, using a machine learning model, the driving map data to produce driving controls for traversing a travel route through the location, wherein the machine learning model identifies vehicle operational data correlating with the driving map data to produce the driving controls;

and

transmitting the driving controls to effect control of the vehicle over the travel route.

2. The method of claim 1, wherein receiving the driving map data includes generating the driving map data by combining the temporal observations to form the driving map data as a composite image, wherein generating the temporal observations includes encoding the observations on separate color channels of the composite image, wherein the temporal observations include at least perceptions by sensors of the vehicle, wherein the desired vehicle operational data includes at least one of:

vehicle velocity control data;

vehicle acceleration control data;

steering wheel angle control data;

vehicle braking data; and

vehicle engine braking data.

3. The method of claim 1, wherein the machine learning model is an autonomous decision device further comprising:

a deep neural network being trained on a plurality of temporal data sets, wherein each of the plurality of temporal data sets including a plurality of sequenced data having a corresponding vehicle control signal.

4. The method of claim 3, wherein each of the plurality of sequenced data being spaced with respect to one another by a predetermined time interval.

5. The method of claim 4, wherein the plurality of sequenced data comprising:

time-series data operating to convey spatial information relating to the vehicle;

subsequent time-series data operating to convey velocity information relating to the vehicle; and

further subsequent time-series data operating to convey acceleration information relating to the vehicle.

6. The method of claim 3, wherein the deep neural network comprising a multi-layer artificial neural network.

7. The method of claim 1, wherein the processing the driving map data to produce desired vehicle operation data further comprising:

segmenting the driving map data to produce a plurality of map segments having dimensions that correspond to an input size of the machine learning algorithm, each of the map segments including an indicator to the vehicle target data;

determine whether one of the map segments includes the vehicle location data;

when the one of the map segments includes the vehicle location data;

identifying a relationship of an obstacle of the one of the plurality of segmented driving map data; and

producing the desired vehicle operational data based on the relationship.

8. A vehicle control unit for controlling a vehicle, the vehicle control unit comprising:

a wireless communication interface to service communication with a vehicle network;

a processor coupled to the wireless communication interface, the processor for controlling operations of the vehicle control unit; and

a memory coupled to the processor, the memory for storing data and program instructions used by the processor, the processor configured to execute the program instructions stored in the memory to;

receive driving map data that is a composition of separate temporal observations within a single composite image including vehicle target data relative to a vehicle location data and obstacle data that characterizes temporal aspects of dynamic obstacles;

process the driving map data using a machine learning model to produce driving controls for traversing a travel route, wherein the machine learning model identifies vehicle operational data correlating with the driving map data to produce the driving controls;

and

transmit, via the wireless communication interface, the driving controls to effect control of the vehicle over the travel route.

9. The vehicle control unit of claim 8, wherein the program instructions to receive the driving map data include program instructions to generate the driving map data by combining the temporal observations to form the driving map data as a composite image, wherein generating the temporal observations includes encoding the observations on separate color channels of the composite image, wherein the temporal observations include at least perceptions by sensors of the vehicle, and wherein the desired vehicle operational data includes at least one of:

vehicle velocity data;

vehicle acceleration data;

steering wheel angle;

vehicle braking data; and

vehicle engine braking data.

10. The vehicle control unit of claim 8, wherein

the machine learning model is a deep neural network trained on a temporal sequence of data for a plurality of temporal data sets, wherein each of the plurality of temporal data sets including a plurality of time-series data being accompanied by vehicle sensor data.

11. The vehicle control unit of claim 10, wherein the plurality of time-series data being separately spaced according to a predetermined time interval.

12. The vehicle control unit of claim 11, wherein:

a time-series data of the plurality of time-series data operates to convey spatial information relating to the vehicle;

a subsequent time-series data of the plurality of time-series data operates to convey velocity information relating to the vehicle; and

a further subsequent time-series data of the plurality of time-series data operates to convey acceleration information relating to the vehicle.

13. The vehicle control unit of claim 10, wherein the deep neural network comprising a multi-layer artificial neural network.

14. The vehicle control unit of claim 8, wherein the program instructions to process the driving map data include program instructions to produce the vehicle operation data by:

retrieving obstacle mapping data related to the driving map data;

segmenting the driving map data to produce a plurality of map segments having dimensions that correspond to an input size of the machine learning algorithm, each of the map segments including an indicator to the vehicle target data;

determine whether one of the map segments includes the vehicle location data;

when the one of the map segments includes the vehicle location data;

identifying a relationship of an obstacle of the one of the map segments; and

producing the desired vehicle operational data based, at least in part, on the relationship.