System and method for controlling vehicle using neural network

• US 9,989,964 B2
• Filed: 11/03/2016
• Issued: 06/05/2018
• Est. Priority Date: 11/03/2016
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Berntorp, Karl, Liu, Ming-Yu, Weiss, Avishai
• Primary Examiner(s): FREJD, RUSSELL WARREN
• Application Number:
  US15/342,216
• Publication Number:
  US 20180120843A1
• Time in Patent Office: 579 Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2420/54 Audio sensitive means, e.g. ultrasound
• CPC Class Codes: B60W 30/0956 the prediction being responsive to traffic or environmental parameters
• CPC Class Codes: B60W 30/18163 Lane change; Overtaking manoeuvres
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G05B 13/027 using neural networks only
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method and a system generate a time-series signal indicative of a variation of the environment in vicinity of the vehicle with respect to a motion of the vehicle and submit the time-series signal to the neural network to produce a reference trajectory as a function of time that satisfies time and spatial constraints on a position of the vehicle. The neural network is trained in to transform time-series signals to reference trajectories of the vehicle. The motion trajectory tracking the reference trajectory while satisfying constraints on the motion of the vehicle is determined and the motion of the vehicle is controlled to follow the motion trajectory.

Claims

1. A method for controlling a vehicle, comprising:

generating, using at least one sensor, a time-series signal indicative of a variation of the environment in vicinity of the vehicle with respect to a motion of the vehicle;

selecting from a memory a neural network trained to transform time-series signals to reference trajectories of the vehicle;

submitting the time-series signal to the neural network to produce a reference trajectory as a function of time that satisfies time and spatial constraints on a position of the vehicle;

determining a motion trajectory tracking the reference trajectory while satisfying constraints on the motion of the vehicle; and

controlling the motion of the vehicle to follow the motion trajectory, wherein at least some steps of the method are performed by a processor operatively connected to the memory and the sensor.

2. The method of claim 1, wherein the time-series signal includes a derivative of measurements of the sensor.

3. The method of claim 2, wherein the neural network is trained to receive as an input a vector of a predetermined size, the input vector including elements representing a combination of the time-series signal, first information indicative of a current position of the vehicle, second information indicative of a target position of the vehicle, and third information indicative of the spatial constraints on the current position of the vehicle, further comprising:

determining the current position of the vehicle;

retrieving from the memory the target position of the vehicle and information indicative of the spatial constraints on the current position of the vehicle;

filtering the measurements of the sensor to produce the time-series signal, such that the combination of the time-series signal with the first information, the second information, and the third information results in the vector of the predetermined size;

combining of the time-series signal with the first information, the second information, and the third information to produce an input vector of the predetermined size; and

inputting the input vector into the neural network to produce the reference trajectory.

4. The method of claim 3, wherein the filtering comprises:

merging and synchronizing measurements of different sensors of the vehicle.

5. The method of claim 1, wherein the memory stores a set of neural networks, each neural network in the set is trained to consider a different driving style for mapping the time-series signals to reference trajectories of the vehicle, further comprising:

determining a driving style for controlling the vehicle; and

selecting from the memory the neural network corresponding to the determined driving style.

6. The method of claim 5, wherein the driving style is determined based on an input from a user of the vehicle.

7. The method of claim 5, wherein the vehicle is a semi-autonomous vehicle, further comprising

learning the driving style of a driver of the vehicle while the vehicle is driven in a manual mode.

8. The method of claim 1, wherein the neural network includes an encoding subnetwork and a decoding subnetwork trained concurrently with the encoding subnetwork,

wherein the encoding subnetwork is trained to transform the time-series signal to information indicative of the constraints on the motion of the vehicle, and

wherein the decoding subnetwork is trained to produce the reference trajectory from the information indicative of the constraints on the motion of the vehicle.

9. The method of claim 8, wherein the encoding and decoding subnetworks are recurrent neural networks.

10. The method of claim 8, wherein the encoding and decoding subnetworks are trained using simulation data for different driving patterns.

11. The method of claim 1, wherein the reference trajectory defines an objective for the motion of the vehicle, and wherein the motion trajectory includes a set of commands for the actuators of the vehicle to move the vehicle according the objective.

12. A system for controlling a vehicle, comprising:

at least one sensor for sensing the environment in vicinity of the vehicle to generate a time-series signal indicative of a variation of the environment with respect to a motion of the vehicle;

a memory storing a neural network trained to transform time-series signals to reference trajectories of the vehicle;

at least one processor configured for submitting the time-series signal to the neural network selected from the memory to produce a reference trajectory as a function of time that satisfies time and spatial constraints on a position of the vehicle and for determining a motion trajectory tracking the reference trajectory while satisfying constraints on the motion of the vehicle; and

a controller for controlling the motion of the vehicle to follow the motion trajectory.

13. The system of claim 12, wherein the memory stores a set of neural networks, each neural network in the set is trained to consider a different driving style for mapping the time-series signals to reference trajectories of the vehicle, wherein the processor selects from the memory the neural network corresponding to a determined driving, further comprising:

an input interface for accepting a selection of the driving style from a user of the vehicle.

14. The system of claim 12, wherein the neural network includes an encoding subnetwork and a decoding subnetwork trained concurrently with the encoding subnetwork,

wherein the encoding subnetwork is trained to transform the time-series signal to information indicative of the constraints on the motion of the vehicle, and

wherein the decoding subnetwork is trained to produce the reference trajectory from the information indicative of the constraints on the motion of the vehicle.

15. The system of claim 14, wherein the encoding and decoding subnetworks are recurrent neural networks.

16. The system of claim 14, wherein the encoding and decoding subnetworks are trained using simulation data for different driving patterns.

17. The system of claim 12, wherein the reference trajectory defines an objective for the motion of the vehicle, and wherein the motion trajectory includes a set of commands for actuators of the vehicle to move the vehicle according the objective.

18. A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, the method comprising:

generating a time-series signal indicative of a variation of the environment in vicinity of the vehicle with respect to a motion of the vehicle;

selecting a neural network trained to transform time-series signals to reference trajectories of the vehicle;

submitting the time-series signal to the neural network to produce a reference trajectory as a function of time that satisfies time and spatial constraints on a position of the vehicle;

determining a motion trajectory tracking the reference trajectory while satisfying constraints on the motion of the vehicle; and

controlling the motion of the vehicle to follow the motion trajectory.