VEHICLE LANE PLACEMENT

• US 20170259801A1
• Filed: 03/08/2016
• Published: 09/14/2017
• Est. Priority Date: 03/08/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Abou-Nasr, Mahmoud A., Filev, Dimitar Petrov
• Granted Patent: US 9,802,599 B2
• Time in Patent Office: Days
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2510/18 Braking system
• CPC Class Codes: B60W 2510/20 Steering systems
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road

Abstract

A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. One general aspect includes sends video data and light detecting and ranging (LIDAR) data to a recurrent neural network (rnn) that includes feedback elements to identify a roadway feature. The system also sends the data to a dynamic convolutional neural network (dcnn) to identify the feature. Output values are sent to a softmax decision network to aggregate the rnn and the dcnn output values and determine a vehicle positional location on the roadway.

Claims

1. A system, comprising a computer having a processor and a memory, the memory storing instructions executable by the processor including instructions to:

receive video data from a video data source affixed to a vehicle;

receive light detecting and ranging (LIDAR) data from a LIDAR data source affixed to the vehicle;

send the video data to a recurrent neural network (RNN) that includes feedback elements to provide feedback from a first layer to a second layer to identify a roadway feature;

send the video data to a dynamic convolutional neural network (DCNN) to apply a convolution filter to the video data to generate a convolution image and a pooling filter to the convolution image to develop a feature map to identify the feature;

send the LIDAR data to the RNN to identify the feature;

send the LIDAR data to the DCNN to identify the feature;

send a RNN output value and a DCNN output value to a softmax decision network to aggregate the RNN output value and the DCNN output value;

determine a softmax output based on whether the RNN output value and the DCNN output value are within a range;

determine a vehicle positional location on the roadway of the vehicle based on the softmax output;

compare the vehicle positional location on the roadway relative to an edge of the roadway and an image from a set of training data images to determine an error rate;

determine a change in a weight value for the RNN and the DCNN from the error rate;

apply the change in the weight value to the RNN and the DCNN; and

control at least a steering, a braking and an acceleration of the vehicle from at least the vehicle positional location of the vehicle.

2. The system of claim 1, wherein the video data source is at least a camera affixed to the vehicle or the set of training data images.

3. The system of claim 1, wherein the LIDAR data source is at least a LIDAR image capture device affixed to the vehicle and the set of training data images.

4. The system of claim 1, wherein the feature includes at least one of a lane marker, a curb, and a roadway edge.

5. The system of claim 1, wherein the computer aggregates the RNN output value and the DCNN output to produce a softmax output value.

6. The system of claim 1, the computer extracts a video frame image from the video data and a LIDAR frame image from the LIDAR data.

7. The system of claim 6, the computer converts the video frame image and the LIDAR frame image into a machine readable image.

8. The system of claim 7, the computer sends the machine readable image to a RNN input and a DCNN input.

9. The system of claim 1, the memory further storing instructions to store the weight value for the RNN and the DCNN in a memory.

10. A system, comprising a computer having a processor and a memory, the memory storing instructions executable by the processor including instructions to:

receive video data from a camera affixed to a vehicle;

receive light detecting and ranging (LIDAR) data from a LIDAR image capture device affixed to the vehicle;

send the video data to a recurrent neural network (RNN) that includes feedback elements that enable signals from a first layer to be fed back to a second layer to identify a roadway feature;

send the video data to a dynamic convolutional neural network (DCNN) to apply a convolution filter to the video data to generate a convolution image and a pooling filter to the convolution image to develop a feature map to identify the roadway feature;

send the LIDAR data to the RNN to identify the feature;

send the LIDAR data to the DCNN to identify the feature;

send a RNN output value and a DCNN output value to a softmax decision network to determine a softmax output from at least the RNN output value and the DCNN output value;

determine a vehicle position on the roadway of the vehicle from the softmax output; and

control at least a steering, a braking and an acceleration of the vehicle from at least the vehicle positional location of the vehicle.

11. The system of claim 10, wherein the feature includes at least a lane marker, a curb and a roadway edge.

12. A method, comprising:

receiving video data from a video data source on a vehicle;

receiving light detecting and ranging (LIDAR) data from a LIDAR data source on the vehicle;

sending the video data to a recurrent neural network (RNN) that includes feedback elements that enable signals from a first layer to be fed back to a second layer to identify a roadway feature;

sending the video data to a dynamic convolutional neural network (DCNN) to apply a convolution filter to the video data to generate a convolution image and a pooling filter to the convolution image to develop a feature map to identify the roadway feature;

sending the LIDAR data to the RNN to identify the roadway feature;

sending the LIDAR data to the DCNN to identify the roadway feature;

sending a RNN output value and a DCNN output value to a softmax decision network to determine a softmax output from at least the RNN output value and the DCNN output value;

determining a vehicle positional location on the roadway of the vehicle from the softmax output;

comparing the vehicle positional location on the roadway and an image from a set of training data images to determine an error rate;

determining a change in a weight value for the RNN and the DCNN from the error rate;

applying the change in the weight value to the RNN and the DCNN; and

control at least a steering of the vehicle from at least the vehicle positional location of the vehicle.

13. The method of claim 12, further comprising controlling at least an acceleration and a braking of the vehicle.

14. The method of claim 12, wherein the video data source is at least a camera affixed to the vehicle or the set of training data images.

15. The method of claim 12, wherein the LIDAR data source is at least a LIDAR image capture device affixed to the vehicle and the set of training data images.

16. The method of claim 12, wherein the roadway feature includes at least a lane marker, a curb and a roadway edge.

17. The method of claim 12, further comprising determining the softmax output by averaging the RNN output value and the DCNN output.

18. The method of claim 12, further comprising extracting a video frame image from the video data and a LIDAR frame image from the LIDAR data.

19. The method of claim 18, further comprising converting the video frame image and the LIDAR frame image into a machine readable images.

20. The method of claim 12, further comprising storing the weight value for the RNN and the DCNN in a memory.