Methods and systems for object detection using multiple sensors

• US 9,098,753 B1
• Filed: 04/25/2014
• Issued: 08/04/2015
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhu, Jiajun, Agarwal, Pratik
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US14/262,269
• Time in Patent Office: 466 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

Methods and systems for object detection using multiple sensors are described herein. In an example embodiment, a vehicle'"'"'s computing device may receive sensor data frames indicative of an environment at different rates from multiple sensors. Based on a first frame from a first sensor indicative of the environment at a first time period and a portion of a first frame that corresponds to the first time period from a second sensor, the computing device may estimate parameters of objects in the vehicle'"'"'s environment. The computing device may modify the parameters in response to receiving subsequent frames or subsequent portions of frame of sensor data from the sensors even if the frames arrive at the computing device out of order. The computing device may provide the parameters of the objects to systems of the vehicle for object detection and obstacle avoidance.

Claims

1. A method comprising:

receiving, at a computing device, a first plurality of frames of sensor data indicative of an environment of a vehicle at a first frame rate;

receiving, at the computing device, a second plurality of frames of sensor data indicative of the environment of the vehicle at a second frame rate;

based on a first frame of the first plurality of frames indicative of the environment at a first time period and a portion of a first frame of the second plurality of frames that corresponds to the first time period, determining a plurality of parameters of one or more objects in the environment of the vehicle;

based on a subsequent frame of the first plurality of frames indicative of the environment at a second time period and at least a subsequent portion of the first frame of the second plurality of frames that corresponds to the second time period, modifying the plurality of parameters of the one or more objects; and

providing the modified plurality of parameters of the one or more objects to one or more systems of the vehicle.

2. The method of claim 1, wherein the computing device receives the first plurality of frames of sensor data from a first sensor and the second plurality of frames of sensor data from a second sensor.

3. The method of claim 2, wherein the first sensor is a camera system and the second sensor is LIDAR.

4. The method of claim 1, further comprising:

providing obstacle avoidance instructions to the one or more systems of the vehicle based on the modified plurality of parameters of the one or more objects.

5. The method of claim 1, wherein the second frame rate is less than the first frame rate.

6. The method of claim 1, wherein determining the plurality of parameters of the one or more objects in the environment of the vehicle comprises:

estimating information indicative of one or more of a position, a velocity, a size, and an acceleration associated with respective objects of the one or more objects in the environment.

7. The method of claim 1, wherein modifying the plurality of parameters of the one or more objects comprises:

estimating information indicative of a change in one or more of a position, a velocity, a size, and an acceleration associated with respective objects of the one or more objects in the environment.

8. The method of claim 1, wherein determining the plurality of parameters of the one or more objects in the environment of the vehicle comprises:

estimating one or more trajectories of respective objects in the one or more objects, wherein the one or more trajectories are indicative of position patterns of the respective objects based on a near-term time period.

9. The method of claim 8, wherein modifying the plurality of parameters of the one or more objects further comprises:

adjusting the one or more trajectories of respective objects based on one or more changes in the plurality of parameters of the one or more objects.

10. The method of claim 1, wherein based on the first frame of the first plurality of frames indicative of the environment at the first time period and the portion of the first frame of the second plurality of frames that corresponds to the first time period, determining the plurality of parameters of one or more objects in the environment of the vehicle comprises:

performing a data analysis process on the first frame of the first plurality of frames and on the first frame of the second plurality of frames, wherein the data analysis process comprises the computing device using probabilistic gating to associate measurements within a given frame to respective objects in the environment; and

based on the data analysis of the first frame of the first plurality of frames and a portion of the data analysis of the first frame of the second plurality of frames that corresponds to the first time period, determining the plurality of parameters of one or more objects in the environment of the vehicle.

11. The method of claim 1, wherein modifying the plurality of parameters of the one or more objects comprises:

performing a subsequent data analysis process on a subsequent frame of the first plurality of frames indicative of the environment at a second time period, wherein the subsequent data analysis process comprises the computing device using probabilistic gating to associate measurements within the subsequent frame to respective objects in the environment; and

based on the subsequent data analysis of the subsequent frame of the first plurality of frames and a subsequent portion of the data analysis of the first frame of the second plurality of frames that corresponds to the second time period, modifying the plurality of parameters of the one or more objects.

12. The method of claim 1, wherein providing the modified plurality of parameters of the one or more objects to the one or more systems of the vehicle includes providing one or more conceptual gates that substantially outline respective objects of the one or more objects in the environment.

13. A system comprising:

at least one processor; and

a memory having stored thereon instructions that, upon execution by the at least one processor, cause the system to perform functions comprising;

receiving a first plurality of frames of sensor data indicative of an environment of a vehicle at a first frame rate;

receiving a second plurality of frames of sensor data indicative of the environment of the vehicle at a second frame rate;

based on a first frame of the first plurality of frames indicative of the environment at a first time period and a portion of a first frame of the second plurality of frames that corresponds to the first time period, determining a plurality of parameters of one or more objects in the environment of the vehicle;

based on a subsequent frame of the first plurality of frames indicative of the environment at a second time period and at least a subsequent portion of the first frame of the second plurality of frames that corresponds to the second time period, modifying the plurality of parameters of the one or more objects; and

providing the modified plurality of parameters of the one or more objects to one or more systems of the vehicle.

14. The system of claim 13, wherein receiving the first plurality of frames of sensor data indicative of the environment of the vehicle at the first frame rate comprises receiving respective frames of the first plurality of frames at a first latency level;

and

wherein receiving the second plurality of frames of sensor data indicative of the environment of the vehicle at the second frame rate comprises receiving respective frames of the second plurality of frames at a second latency level a threshold higher than the first latency level.

15. The system of claim 13, wherein based on the first frame of the first plurality of frames indicative of the environment at the first time period and the portion of the first frame of the second plurality of frames that corresponds to the first time period, determining the plurality of parameters of one or more objects in the environment of the vehicle further comprises:

estimating one or more trajectories of respective objects in the one or more objects responsive to receiving the first frame of the first plurality of frames, wherein the one or more trajectories are indicative of near-term position patterns of the respective objects; and

adjusting the one or more trajectories of the respective objects in the one or more objects responsive to receiving the first frame of the second plurality of frames.

16. The system of claim 15, wherein providing the modified plurality of parameters of the one or more objects to one or more systems of the vehicle comprises:

providing the one or more adjusted trajectories of the respective objects in the one or more objects to the one or more systems of the vehicle.

17. The system of claim 13, wherein based on the first frame of the first plurality of frames indicative of the environment at the first time period and the portion of a first frame of the second plurality of frames that corresponds to the first time period, determining the plurality of parameters of one or more objects in the environment of the vehicle comprises:

responsive to receiving the first frame of the first plurality of frames, determining the plurality of parameters of the one or more objects in the environment of the vehicle; and

responsive to receiving the first frame of the second plurality of frames after receiving the first frame of the first plurality of frames, adjusting the plurality of parameters of the one or more objects in the environment of the vehicle.

18. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions comprising:

receiving a first plurality of frames of sensor data indicative of an environment of a vehicle at a first frame rate;

receiving a second plurality of frames of sensor data indicative of the environment of the vehicle at a second frame rate;

based on a first frame of the first plurality of frames indicative of the environment at a first time period and a portion of a first frame of the second plurality of frames that corresponds to the first time period, determining a plurality of parameters of one or more objects in the environment of the vehicle;

based on a subsequent frame of the first plurality of frames indicative of the environment at a second time period and at least a subsequent portion of the first frame of the second plurality of frames that corresponds to the second time period, modifying the plurality of parameters of the one or more objects; and

providing the modified plurality of parameters of the one or more objects to one or more systems of the vehicle.

19. The non-transitory computer readable medium of claim 18, wherein at least the subsequent portion of the first frame of the second plurality of frames that corresponds to the second time period comprises:

a remainder of the first frame of the second plurality of frames; and

a portion of a subsequent frame of the second plurality of frames that corresponds to the second time period.

20. The non-transitory computer readable medium of claim 18, wherein determining the plurality of parameters of one or more objects in the environment of the vehicle is further based on at least a portion of a first frame of a third plurality of frames indicative of the environment at the first time period, wherein the third plurality of frames correspond to data received from a third sensor.