Methods and systems for object detection using multiple sensors

• US 9,607,226 B2
• Filed: 07/06/2015
• Issued: 03/28/2017
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Agarwal, Pratik
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/791,725
• Publication Number:
  US 20150310281A1
• Time in Patent Office: 631 Days
• Field of Search: 382/103, 382/104, 382/107
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Methods and systems for object detection using multiple sensors are described herein. In an example embodiment, a vehicle'"'"'s computing device may receive sensor data frames indicative of an environment at different rates from multiple sensors. Based on a first frame from a first sensor indicative of the environment at a first time period and a portion of a first frame that corresponds to the first time period from a second sensor, the computing device may estimate parameters of objects in the vehicle'"'"'s environment. The computing device may modify the parameters in response to receiving subsequent frames or subsequent portions of frame of sensor data from the sensors even if the frames arrive at the computing device out of order. The computing device may provide the parameters of the objects to systems of the vehicle for object detection and obstacle avoidance.

Claims

1. A method comprising:

receiving, at a computing system from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, at the computing system from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic gating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing, by the computing system, the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

2. The method of claim 1, wherein the first sensor is LIDAR and the second sensor is a camera system.

3. The method of claim 1, wherein the computing system completes receiving the first data segment of the first sensor data prior to completely receiving the first data segment of the second sensor data.

4. The method of claim 1, wherein providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle includes providing obstacle avoidance instructions.

5. The method of claim 1, wherein the second rate is less than the first rate.

6. The method of claim 1, wherein determining information indicative of the parameters of the one or more objects in the environment comprises:

estimating a position and size of respective objects of the one or more objects in the environment.

7. The method of claim 1, wherein modifying the information indicative of the parameters of the one or more objects comprises:

estimating a change in a velocity and an acceleration of respective objects of the one or more objects in the environment.

8. The method of claim 1, wherein determining information indicative of the parameters of the one or more objects in the environment comprises:

estimating one or more trajectories of respective objects of the one or more objects, wherein the one or more trajectories are indicative of position patterns of the respective objects based on a near-term time period.

9. The method of claim 8, wherein modifying the information indicative of the parameters of the one or more objects comprises:

adjusting the one or more trajectories of respective objects based on one or more changes in the information corresponding to the one or more objects.

10. The method of claim 1, further comprising:

based on the modified information indicative of the parameters of the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

11. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic gating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

12. The system of claim 11, wherein receiving, from the first sensor at the first rate, first sensor data corresponding to the environment of a vehicle comprises:

receiving respective data segments of the first sensor data from the first sensor at a first latency level; and

wherein receiving, from the second sensor at the second rate, sensor data corresponding to the environment comprises;

receiving respective data segments of second sensor data from the second sensor at a second latency level, wherein the second latency level is a threshold higher than the first latency level.

13. The system of claim 11, wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information indicative of parameters of one or more objects in the environment comprises:

estimating one or more trajectories of respective objects in the one or more objects responsive to receiving the first data segment of the first sensor data from the first sensor, wherein the one or more trajectories are indicative of near-term position patterns of the respective objects; and

adjusting the one or more trajectories of the respective objects in the one or more objects responsive to receiving the portion of the first data segment of the second sensor data from the second sensor.

14. The system of claim 13, wherein providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle includes providing the one or more adjusted trajectories of the respective objects in the one or more objects to the one or more systems of the vehicle.

15. The system of claim 11, wherein determining, using the first data segment of the first sensor data and the portion of the first data segment of the second sensor data that correspond to the environment at the first time period, information indicative of the parameters of one or more objects in the environment comprises:

using a Global Nearest Neighbor (GNN) process to select the portion of the first data segment of the second sensor data that corresponds to the environment at a same time of the first data segment of the first sensor data.

16. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving, from a first sensor at a first rate, first sensor data corresponding to an environment of a vehicle;

receiving, from a second sensor at a second rate, second sensor data corresponding to the environment;

determining, using probabilistic gating on a first data segment of the first sensor data and a portion of a first data segment of the second sensor data that correspond to the environment at a first time period, information indicative of parameters of one or more objects in the environment, wherein the probabilistic gating associates measurements within a given data segment of sensor data to respective objects in the environment;

modifying, using probabilistic dating on a second data segment of the first sensor data and a subsequent portion of the first data segment of the second sensor data that correspond to the environment at a second time period, the information indicative of the parameters of the one or more objects; and

providing the modified information indicative of the parameters of the one or more objects to one or more systems of the vehicle.

17. The non-transitory computer readable medium of claim 16, further comprising:

based on the modified information indicative of the parameters of the one or more objects, providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment to the one or more systems of the vehicle.

18. The non-transitory computer readable medium of claim 16, further comprising:

receiving, from a third sensor at a third rate, third sensor data corresponding to the environment; and

wherein determining information indicative of parameters of one or more objects in the environment further comprises using at least a portion of a first data segment of the third sensor data.