Planning for unknown objects by an autonomous vehicle

• US 10,281,920 B2
• Filed: 03/07/2017
• Issued: 05/07/2019
• Est. Priority Date: 03/07/2017
• Status: Active Grant

• Current Assignee: Motional AD, LLC (Aptiv PLC)
• Original Assignee: nuTonomy, Inc. (BorgWarner Incorporated)
• Inventors: Frazzoli, Emilio, Qin, Baoxing
• Primary Examiner(s): Melton, Todd M
• Application Number:
  US15/451,734
• Publication Number:
  US 20180259968A1
• Time in Patent Office: 791 Days
• CPC Class Codes: B60W 10/184 with wheel brakes
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/00 Input parameters relating to infrastructure
• CPC Class Codes: B60W 2552/10 Number of lanes
• CPC Class Codes: B60W 2552/20 Road profile, i.e. the change in elevation or curvature of a plurality of continuous road segments
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2554/4026 Cycles
• CPC Class Codes: B60W 2554/4029 Pedestrians
• CPC Class Codes: B60W 2554/406 Traffic density
• CPC Class Codes: B60W 2555/20 Ambient conditions, e.g. wind or rain
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/00 Input parameters relating to data
• CPC Class Codes: B60W 2556/10 Historical data
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: B60W 60/0016 of the vehicle or its occupants
• CPC Class Codes: G05D 1/0219 ensuring the processing of the whole working surface
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G08G 1/166 : for active traffic, e.g. moving vehicles, pedestrians, bikes

Abstract

Among other things, a world model is maintained of an environment of a vehicle. A hypothetical object in the environment that cannot be perceived by sensors of the vehicle is included in the world model.

Claims

1. A computer implemented method comprising:

receiving data collected by one or more sensors of an autonomous vehicle, the data representing an observable part of an environment of the autonomous vehicle;

generating, based on the received data from the one or more sensors, data representing a non-observable part of the environment, including hypothetical object data representing at least one hypothetical object at a particular location in the non-observable part of the environment; and

generating commands for operation of the vehicle within the environment, the commands being dependent on the data representing the observable part of the environment and on the generated data representing the at least one hypothetical object in the non-observable part of the environment.

2. The method of claim 1, in which the hypothetical object comprises a moving object.

3. The method of claim 1, in which the hypothetical object comprises an object that uses a path of travel from which the vehicle is excluded.

4. The method of claim 1, in which the hypothetical object comprises at least one of:

a vehicle, a bicycle, a bus, a train, a pedestrian, and an animal.

5. The method of claim 1, in which generating data representing a non-observable part of the environment comprises selecting a type of the hypothetical object and an attribute of the hypothetical object probabilistically based on objects previously observed in the environment.

6. The method of claim 5, in which the hypothetical object comprises a vehicle and the attribute comprises a size and a speed.

7. The method of claim 1, in which the observable part and the non-observable part are separated by a boundary.

8. The method of claim 7, comprising detecting of the boundary.

9. The method of claim 8, in which detecting the boundary comprises using data from the one or more sensors to distinguish an observable ground from a foreground that obscures a portion of the ground.

10. The method of claim 1, in which generating data representing the non-observable part of the environment comprises using stored data to infer a possible location of the hypothetical object.

11. The method of claim 1, in which generating data representing the non-observable part of the environment comprises querying traffic lane information from a road network database.

12. The method of claim 1, in which generating data representing the non-observable part of the environment comprises determining a location of the vehicle based on a road network database and the one or more sensors.

13. The method of claim 1, in which generating data representing the non-observable part of the environment comprises querying traffic lane information from a database and discretizing the traffic lane into discretized points.

14. The method of claim 1, in which generating data representing the non-observable part of the environment comprises generating an unknown skeleton of discretized points of a lane that cannot be perceived by the one or more sensors.

15. The method of claim 14, in which generating data representing the non-observable part of the environment comprises (a) generating a representative shape at a discretized point of the unknown skeleton, and (b) evaluating if the representative shape is completely within the non-observable part.

16. The method of claim 1, in which generating data representing the non-observable part of the environment comprises treating a representative shape as the hypothetical object.

17. The method of claim 1, in which generating data representing the non-observable part of the environment comprises applying temporal filtering to determine a location of the hypothetical object.

18. The method of claim 17, in which applying the temporal filtering comprises smoothing an unknown skeleton by a forward propagated unknown skeleton, wherein the forward propagated unknown skeleton is generated by moving forward an old unknown skeleton along a traffic lane.

19. The method of claim 1, in which including data representing at least one hypothetical object in the non-observable part of the environment comprises associating one or more attributes with the hypothetical object.

20. The method of claim 19 in which the one or more of the attributes are related to a possible motion state of the hypothetical object.

21. The method of claim 20, in which the motion state comprises a stationary condition.

22. The method of claim 20, in which the motion state comprises a moving condition.

23. The method of claim 20, in which the motion state comprises a speed and a moving direction.

24. The method of claim 23, in which the speed is set to less than or equal to a predetermined maximum value.

25. The method of claim 24, in which the predetermined maximum value comprises a speed limit.

26. The method of claim 24, in which the predetermined maximum value comprises a quantity derived from other objects concurrently or previously observed in the environment.

27. The method of claim 24, in which the predetermined maximum value comprises a quantity derived from historical data, road configuration, traffic rules, an event, a time, a weather condition, or a combination of two or more of them.

28. The method of claim 1, comprising accessing a database comprising road network information.

29. The method of claim 28, comprises using data from a second set of sensors.

30. The method of claim 29, in which the one or more sensors or the second set of sensors comprises a radar sensor.

31. The method of claim 29, in which the one or more sensors or the second set of sensors comprises a lidar sensor.

32. The method of claim 29, in which the one or more sensors or the second set of sensors comprises a camera sensor.

33. The method of claim 29, in which the one or more sensors or the second set of sensors comprises a stereo camera sensor.

34. The method of claim 29, in which the one or more sensors or the second set of sensors comprises a monocular camera sensor.

35. The method of claim 1, in which generating the commands comprises updating a trajectory for the vehicle.

36. The method of claim 35, comprising executing the trajectory for the vehicle.

37. The method of claim 1, in which the vehicle comprises an autonomous vehicle.

38. The method of claim 1, wherein the hypothetical object data is conservatively generated without saturating the non-observable part of the environment with hypothetical objects.

39. A computer-implemented method comprising:

generating commands to cause an autonomous vehicle to drive on a road network at specified speeds and making specified turns to reach a goal position, the autonomous vehicle comprising one or more sensors configured to collect data of an environment of the autonomous vehicle;

generating, based on data received from the one or more sensors, current data representing an observable part and non-observable part of the environment, including hypothetical object data representing at least one hypothetical vehicle at a particular location in a non-observable part of the environment; and

updating the commands in response to the current data representing a hypothetical speed and moving direction of a hypothetical vehicle also being driven on the road network, the commands being updated to reduce a risk of the autonomous vehicle colliding with another vehicle on the road network.

40. The method of claim 39, in which the hypothetical speed and moving direction is probabilistically derived based on vehicles previously observed in the environment.

41. The method of claim 39, in which the observable part and the non-observable part are separated by a boundary.

42. The method of claim 41, comprising detecting of the boundary.

43. The method of claim 42, in which detecting of the boundary comprises using data from the one or more sensors to distinguish an observable ground from a foreground that obscures a portion of the ground.

44. The method of claim 43, in which the one or more sensors comprise sensors onboard the autonomous vehicle.

45. The method of claim 43, in which the one or more sensors comprise sensors offboard the autonomous vehicle.

46. The method of claim 39, comprising generating of the current data based on known objects perceived by the one or more sensors.

47. The method of claim 46, in which generating of the current data comprises querying traffic lane information from a road network database.

48. The method of claim 46, in which generating of the current data comprises using stored data to infer a possible location of the hypothetical vehicle.

49. The method of claim 46, in which generating of the current data comprises determining a location of the autonomous vehicle based on a road network database and the one or more sensors.

50. The method of claim 46, in which generating of the current data comprises querying traffic lane information from a database and discretizing the traffic lane into discretized points.

51. The method of claim 46, in which generating of the current data comprises generating an unknown skeleton of discretized points of a lane that cannot be perceived by sensors.

52. The method of claim 51, in which generating of the current data comprises (a) generating a representative shape at a discretized point of the unknown skeleton, and (b) evaluating if the representative shape is completely within the non-observable part of the environment.

53. The method of claim 52, in which generating of the current data comprises treating the representative shape as the hypothetical vehicle.

54. The method of claim 46, in which generating of the current data comprises applying temporal filtering to determine a location of the hypothetical vehicle.

55. The method of claim 54, in which applying the temporal filtering comprises smoothing an unknown skeleton by a forward propagated unknown skeleton, wherein the forward propagated unknown skeleton is generated by moving forward an old unknown skeleton along a traffic lane.

56. The method of claim 46, in which generating of the current data comprises associating one or more attributes with the hypothetical vehicle.

57. The method of claim 56, in which the one or more of the attributes are related to a possible motion state of the hypothetical vehicle.

58. The method of claim 57, in which the motion state comprises a stationary condition.

59. The method of claim 39, in which the hypothetical speed is set to less than or equal to a predetermined maximum value.

60. The method of claim 59, in which the predetermined maximum value comprises a speed limit.

61. The method of claim 59, in which the predetermined maximum value comprises a quantity derived from other objects concurrently or previously observed in the environment.

62. The method of claim 59, in which the predetermined maximum value comprises a quantity derived from historical data, road configuration, traffic rules, an event, a time, a weather condition, or a combination of two or more of them.

63. The method of claim 39, comprising accessing a database comprising road network information.

64. The method of claim 39, in which the one or more sensors comprises a radar sensor.

65. The method of claim 39, in which the one or more sensors comprises a lidar sensor.

66. The method of claim 39, in which the one or more sensors comprises a camera sensor.

67. The method of claim 66, in which the camera sensor comprises a stereo camera sensor.

68. The method of claim 66, in which the camera sensor comprises a monocular camera sensor.

69. The method of claim 39, wherein the hypothetical object data is conservatively generated without saturating the non-observable part of the environment with hypothetical objects.