Vehicle heading prediction neural network

• US 10,366,502 B1
• Filed: 12/09/2016
• Issued: 07/30/2019
• Est. Priority Date: 12/09/2016
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Li, Congcong
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US15/374,884
• Time in Patent Office: 963 Days
• Field of Search: None
• CPC Class Codes: B60R 2300/30 characterised by the type of image processing
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0202 specially adapted to aircraft
• CPC Class Codes: G05D 1/0206 specially adapted to water vehicles
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating vehicle heading predictions from point cloud data using a neural network. One of the methods includes receiving a plurality of different projections of point cloud data, wherein the point cloud data represents different sensor measurements of electromagnetic radiation reflected off a vehicle. Each of the plurality of projections of point cloud data is provided as input to a neural network subsystem trained to receive projections of point cloud data for a vehicle and to generate one or more vehicle heading classifications as an output. At the output of the neural network subsystem, one or more vehicle heading predictions is received.

Claims

1. A computer-implemented method comprising:

receiving, by a first vehicle including on-board sensors, point cloud data representing sensor measurements of electromagnetic radiation reflected off a second vehicle and captured by a first on-board sensor of the first vehicle;

generating a plurality of different projections of the point cloud data representing the sensor measurements captured by the first on-board sensor, wherein each projection of the plurality of different projections is a mapping of the point cloud data onto a different projection plane of a plurality of projection planes, and wherein each different projection plane represents a different view of the second vehicle;

providing each of the plurality of different projections of the point cloud data as input to a neural network subsystem trained to receive projections of point cloud data for the second vehicle and to generate one or more second vehicle classifications as an output; and

receiving, at the output of the neural network subsystem, the one or more second vehicle classifications for the second vehicle, wherein each second vehicle classification indicates a current direction in which the second vehicle is likely to be oriented at a current time in a particular coordinate system.

2. The method of claim 1, wherein the neural network subsystem comprises a neural network having a plurality of point cloud subnetworks that each operate on a different respective projection of the point cloud data.

3. The method of claim 2, wherein each point cloud subnetwork includes a convolutional stack having one or more convolutional layers.

4. The method of claim 3, wherein the neural network comprises a camera subnetwork that operates on camera image data.

5. The method of claim 4, wherein the camera subnetwork includes a convolutional stack having one or more convolutional layers.

6. The method of claim 4, wherein the neural network includes a concatenation layer that combines the outputs of each of the point cloud subnetworks and the camera subnetwork.

7. The method of claim 1, wherein the one or more second vehicle classifications include a respective score for each of one or more directions.

8. The method of claim 1, wherein the one or more second vehicle classifications identify multiple components of a single direction.

9. The method of claim 1, wherein the point cloud data comprises a plurality of points representing an intensity and a position of electromagnetic radiation reflected off the second vehicle.

10. The method of claim 1, wherein the neural network subsystem is trained to generate the one or more second vehicle classifications from learned physical shape information for a plurality of different vehicle types.

11. A system comprising:

one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving, by a first vehicle including on-board sensors, point cloud data representing sensor measurements of electromagnetic radiation reflected off a second vehicle and captured by a first on-board sensor of the first vehicle;

generating a plurality of different projections of the point cloud data representing the sensor measurements captured by the first on-board sensor, wherein each projection of the plurality of different projections is a mapping of the point cloud data onto a different projection plane of a plurality of projection planes, and wherein each different projection plane represents a different view of the second vehicle;

providing each of the plurality of different projections of the point cloud data as input to a neural network subsystem trained to receive projections of point cloud data for the second vehicle and to generate one or more second vehicle classifications as an output; and

receiving, at the output of the neural network subsystem, the one or more second vehicle classifications for the second vehicle, wherein each second vehicle classification indicates a current direction in which the second vehicle is likely to be oriented at a current time in a particular coordinate system.

12. The system of claim 11, wherein the neural network subsystem comprises a neural network having a plurality of point cloud subnetworks that each operate on a different respective projection of the point cloud data.

13. The system of claim 12, wherein each point cloud subnetwork includes a convolutional stack having one or more convolutional layers.

14. The system of claim 13, wherein the neural network comprises a camera subnetwork that operates on camera image data.

15. The system of claim 14, wherein the camera subnetwork includes a convolutional stack having one or more convolutional layers.

16. The system of claim 14, wherein the neural network includes a concatenation layer that combines the outputs of each of the point cloud subnetworks and the camera subnetwork.

17. The system of claim 11, wherein the one or more second vehicle classifications include a respective score for each of one or more directions.

18. The system of claim 11, wherein the one or more second vehicle classifications identify multiple components of a single direction.

19. The system of claim 11, wherein the point cloud data comprises a plurality of points representing an intensity and a position of electromagnetic radiation reflected off the second vehicle.

20. The system of claim 11, wherein the neural network subsystem is trained to generate the one or more second vehicle classifications from learned physical shape information for a plurality of different vehicle types.

21. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving, by a first vehicle including on-board sensors, point cloud data representing sensor measurements of electromagnetic radiation reflected off a second vehicle and captured by a first on-board sensor of the first vehicle;

generating a plurality of different projections of the point cloud data representing the sensor measurements captured by the first on-board sensor, wherein each projection of the plurality of different projections is a mapping of the point cloud data onto a different projection plane of a plurality of projection planes, and wherein each different projection plane represents a different view of the second vehicle;

providing each of the plurality of different projections of the point cloud data as input to a neural network subsystem trained to receive projections of point cloud data for the second vehicle and to generate one or more second vehicle classifications as an output; and

receiving, at the output of the neural network subsystem, the one or more second vehicle classifications for the second vehicle, wherein each second vehicle classification indicates a current direction in which the second vehicle is likely to be oriented at a current time in a particular coordinate system.