Methods and Systems for Object Detection using Multiple Sensors

• US 20150310281A1
• Filed: 07/06/2015
• Published: 10/29/2015
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Agarwal, Pratik
• Granted Patent: US 9,607,226 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Methods and systems for object detection using multiple sensors are described herein. In an example embodiment, a vehicle'"'"'s computing device may receive sensor data frames indicative of an environment at different rates from multiple sensors. Based on a first frame from a first sensor indicative of the environment at a first time period and a portion of a first frame that corresponds to the first time period from a second sensor, the computing device may estimate parameters of objects in the vehicle'"'"'s environment. The computing device may modify the parameters in response to receiving subsequent frames or subsequent portions of frame of sensor data from the sensors even if the frames arrive at the computing device out of order. The computing device may provide the parameters of the objects to systems of the vehicle for object detection and obstacle avoidance.

Claims

1. A method comprising:

receiving, at a computing system from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, at the computing system from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information corresponding to one or more objects in the environment;

modifying, using a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information corresponding to the one or more objects; and

providing, by the computing system, the modified information corresponding to the one or more objects to one or more systems of the vehicle.

2. The method of claim 1, wherein the first sensor is LIDAR and the second sensor is a camera system.

3. The method of claim 1, wherein the computing system completes receiving the first data segment of the first sensor data prior to completely receiving the first data segment of the second sensor data.

4. The method of claim 1, wherein providing the modified information corresponding to the one or more objects to one or more systems of the vehicle includes providing obstacle avoidance instructions.

5. The method of claim 1, wherein the second rate is less than the first rate.

6. The method of claim 1, wherein determining information corresponding to the one or more objects in the environment comprises:

estimating a position and size of respective objects of the one or more objects in the environment.

7. The method of claim 1, wherein modifying the information corresponding to the one or more objects comprises:

estimating a change in a velocity and an acceleration of respective objects of the one or more objects in the environment.

8. The method of claim 1, wherein determining information corresponding to the one or more objects in the environment comprises:

estimating one or more trajectories of respective objects of the one or more objects, wherein the one or more trajectories are indicative of position patterns of the respective objects based on a near-term time period.

9. The method of claim 8, wherein modifying the information corresponding to the one or more objects comprises:

adjusting the one or more trajectories of respective objects based on one or more changes in the information corresponding to the one or more objects.

10. The method of claim 1, wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information corresponding to one or more objects in the environment comprises:

performing a data analysis process on the first data segment of the first sensor data and on the first data segment of the second sensor data, wherein the data analysis process comprises using probabilistic gating to associate measurements within a given data segment of sensor data to respective objects in the environment; and

based on the data analysis of the first data segment of the first sensor data and the data analysis of the portion of the first data segment of the second sensor data, determining information indicative of a plurality of parameters of one or more objects in the environment of the vehicle.

11. The method of claim 10, wherein modifying the information corresponding to the one or more objects comprises:

performing a subsequent data analysis process on the second data segment of the first sensor data corresponding to the environment at a second time period, wherein the subsequent data analysis process comprises the computing device using probabilistic gating to associate measurements within the subsequent data segment to respective objects in the environment; and

based on the subsequent data analysis of the second data segment of the first sensor data and the data analysis of the subsequent portion of the first data segment of the second sensor data that corresponds to the second time period, modifying the information indicative of a plurality of parameters of the one or more objects.

12. The method of claim 1, further comprising:

based on the modified information corresponding to the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

13. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information corresponding to one or more objects in the environment;

modifying, using a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information corresponding to the one or more objects; and

providing the modified information corresponding to the one or more objects to one or more systems of the vehicle.

14. The system of claim 13, wherein receiving, from the first sensor at the first rate, first sensor data corresponding to the environment of a vehicle comprises:

receiving respective data segments of the first sensor data from the first sensor at a first latency level; and

wherein receiving, from the second sensor at the second rate, sensor data corresponding to the environment comprises;

receiving respective data segments of second sensor data from the second sensor at a second latency level, wherein the second latency level is a threshold higher than the first latency level.

15. The system of claim 13, wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information corresponding to one or more objects in the environment comprises:

estimating one or more trajectories of respective objects in the one or more objects responsive to receiving the first data segment of the first sensor data from the first sensor, wherein the one or more trajectories are indicative of near-term position patterns of the respective objects; and

adjusting the one or more trajectories of the respective objects in the one or more objects responsive to receiving the portion of the first data segment of the second sensor data from the second sensor.

16. The system of claim 15, wherein providing the modified information corresponding to the one or more objects to one or more systems of the vehicle includes providing the one or more adjusted trajectories of the respective objects in the one or more objects to the one or more systems of the vehicle.

17. The system of claim 13, wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information corresponding to one or more objects in the environment comprises:

using a Global Nearest Neighbor (GNN) process to select the portion of the first data segment of the second sensor data that corresponds to the environment at a same time of the first data segment of the first sensor data.

18. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information corresponding to one or more objects in the environment;

modifying, using a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information corresponding to the one or more objects; and

providing the modified information corresponding to the one or more objects to one or more systems of the vehicle.

19. The non-transitory computer readable medium of claim 18, further comprising:

based on the modified information corresponding to the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

20. The non-transitory computer readable medium of claim 18, further comprising:

receiving, from a third sensor at a third rate, third sensor data corresponding to the environment; and

wherein determining information corresponding to one or more objects in the environment further comprises using at least a portion of a first data segment of the third sensor data.