Intelligent ladar system with low latency motion planning updates

• US 10,495,757 B2
• Filed: 08/21/2018
• Issued: 12/03/2019
• Est. Priority Date: 09/15/2017
• Status: Active Grant

• Current Assignee: AEye, Inc.
• Original Assignee: AEye, Inc.
• Inventors: Dussan, Luis Carlos, Steinhardt, Allan, Benscoter, Joel David, Greene, Jordan Spencer
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US16/106,350
• Publication Number:
  US 20190086550A1
• Time in Patent Office: 469 Days
• Field of Search: None
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4811 common to transmitter and receiver
• CPC Class Codes: G01S 7/4814 of transmitters alone
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/484 Transmitters
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G02B 19/0028 refractive and reflective surfaces, e.g. non-imaging catadioptric systems
• CPC Class Codes: G02B 19/0085 for use with both a detector and a source , e.g. in a transceiver, G02B19/009, G02B19/0095 take precedence
• CPC Class Codes: G02B 26/108 having one or more prisms as scanning elements

Abstract

Systems and methods are disclosed for vehicle motion planning where a sensor, such as a ladar system, is used to detect threatening or anomalous conditions within the sensor'"'"'s field of view so that priority warning data about such conditions can be inserted at low latency into the motion planning loop of a motion planning computer system for the vehicle. The ladar system can perform compressive sensing to target the field of view with a plurality of ladar pulses.

Claims

1. A system for vehicle motion planning, the system comprising:

a sensor; and

a motion planning computer system in communication with the sensor;

wherein the sensor has a field of view and is configured to detect a threatening or anomalous condition in the field of view;

wherein the motion planning computer system is configured with a motion planning loop; and

wherein the sensor and motion planning computer system are configured to cooperate such that the sensor is configured to provide a priority flag to the motion planning computer system that is indicative of the detected condition to thereby insert priority data about the detected condition into the motion planning loop.

2. The system of claim 1 wherein the sensor comprises a ladar system.

3. The system of claim 2 wherein the ladar system comprises a ladar transmitter configured with a compressive sensing capability with respect to range point targeting.

4. The system of claim 3 wherein the ladar system is configured to re-task its targeting of range points based on the detected condition.

5. The system of claim 4 wherein the ladar system includes memory for storing a point cloud of range point return data, and wherein the ladar system is further configured to exploit the point cloud to detect the condition.

6. The system of claim 3 wherein the ladar transmitter comprises a plurality of scanable mirrors for targeting the ladar transmitter.

7. The system of claim 3 wherein the ladar system comprises a ladar receiver configured with selective readout from a detector array based on range point targeting.

8. The system of claim 1 wherein the motion planning computer system is configured to treat the priority flag as a vector interrupt that alters a data stack used for decision-making by the motion planning loop.

9. The system of claim 2 wherein the sensor further comprises a camera.

10. The system of claim 9 wherein the camera is co-bore sited with respect to the ladar system.

11. The system of claim 9 wherein the camera includes a video controller configured to (1) read out image data corresponding to at least a portion of a field of view for the camera, and (2) detect motion within the field of view based on the read out image data.

16. The system of claim 2 wherein the sensor further comprises a radar.

17. The system of claim 2 wherein the sensor further comprises an acoustic sensor.

18. The system of claim 2 wherein the sensor further comprises a vehicle telematics sensor.

19. The system of claim 2 wherein the sensor is configured to detect the condition via at least one of comparative, selective, and/or objective sensing.

22. The system of claim 1 wherein the sensor includes a field programmable gate array (FPGA), the FPGA configured to detect the condition.

12. A system for vehicle motion planning, the system comprising:

a sensor; and

a motion planning computer system in communication with the sensor;

wherein the sensor has a field of view and is configured to detect a threatening or anomalous condition in the field of view;

wherein the motion planning computer system is configured with a motion planning loop;

wherein the sensor and motion planning computer system are configured to cooperate such that the sensor is configured to insert priority data indicative of the detected condition into the motion planning loop;

wherein the sensor comprises a ladar system and a camera; and

wherein the camera is configured to detect a shimmer event as the detected condition.

45. The system of claim 12 wherein the ladar system comprises a ladar transmitter configured with a compressive sensing capability with respect to range point targeting.

46. The system of claim 45 wherein the ladar system is configured to re-task its targeting of range points based on the detected condition.

47. The system of claim 46 wherein the ladar system includes memory for storing a point cloud of range point return data, and wherein the ladar system is further configured to exploit the point cloud to detect the condition.

48. The system of claim 12 wherein the sensor is configured to provide a priority flag to the motion planning computer system that is indicative of the detected condition.

49. The system of claim 48 wherein the motion planning computer system is configured to treat the priority flag as a vector interrupt that alters a data stack used for decision-making by the motion planning loop.

50. The system of claim 12 wherein the camera is co-bore sited with respect to the ladar system.

51. The system of claim 12 wherein the sensor includes a field programmable gate array (FPGA), the FPGA configured to detect the condition.

13. A system for vehicle motion planning, the system comprising:

a sensor; and

a motion planning computer system in communication with the sensor;

wherein the sensor has a field of view and is configured to detect a threatening or anomalous condition in the field of view;

wherein the motion planning computer system is configured with a motion planning loop;

wherein the sensor and motion planning computer system are configured to cooperate such that the sensor is configured to insert priority data indicative of the detected condition into the motion planning loop;

wherein the sensor comprises a ladar system and a camera; and

wherein the camera is configured to detect a shiny object event as the detected condition.

52. The system of claim 13 wherein the ladar system comprises a ladar transmitter configured with a compressive sensing capability with respect to range point targeting.

53. The system of claim 52 wherein the ladar system is configured to re-task its targeting of range points based on the detected condition.

54. The system of claim 53 wherein the ladar system includes memory for storing a point cloud of range point return data, and wherein the ladar system is further configured to exploit the point cloud to detect the condition.

55. The system of claim 13 wherein the sensor is configured to provide a priority flag to the motion planning computer system that is indicative of the detected condition.

56. The system of claim 55 wherein the motion planning computer system is configured to treat the priority flag as a vector interrupt that alters a data stack used for decision-making by the motion planning loop.

57. The system of claim 13 wherein the camera is co-bore sited with respect to the ladar system.

58. The system of claim 13 wherein the sensor includes a field programmable gate array (FPGA), the FPGA configured to detect the condition.

14. A system for vehicle motion planning, the system comprising:

a sensor; and

a motion planning computer system in communication with the sensor;

wherein the sensor has a field of view and is configured to detect a threatening or anomalous condition in the field of view;

wherein the motion planning computer system is configured with a motion planning loop;

wherein the sensor and motion planning computer system are configured to cooperate such that the sensor is configured to insert priority data indicative of the detected condition into the motion planning loop;

wherein the sensor comprises a ladar system and a camera; and

wherein the ladar system and the camera are configured to cooperate via cross-cueing.

15. The system of claim 14 wherein the camera is configured to generate a shot list for use by the ladar system in response to the detected condition.

59. The system of claim 14 wherein the ladar system comprises a ladar transmitter configured with a compressive sensing capability with respect to range point targeting.

60. The system of claim 59 wherein the ladar system is configured to re-task its targeting of range points based on the detected condition.

61. The system of claim 60 wherein the ladar system includes memory for storing a point cloud of range point return data, and wherein the ladar system is further configured to exploit the point cloud to detect the condition.

62. The system of claim 14 wherein the sensor is configured to provide a priority flag to the motion planning computer system that is indicative of the detected condition.

63. The system of claim 62 wherein the motion planning computer system is configured to treat the priority flag as a vector interrupt that alters a data stack used for decision-making by the motion planning loop.

64. The system of claim 14 wherein the camera is co-bore sited with respect to the ladar system.

65. The system of claim 14 wherein the sensor includes a field programmable gate array (FPGA), the FPGA configured to detect the condition.

20. A system for vehicle motion planning, the system comprising:

a sensor; and

a motion planning computer system in communication with the sensor;

wherein the sensor has a field of view and is configured to detect a threatening or anomalous condition in the field of view;

wherein the motion planning computer system is configured with a motion planning loop; and

wherein the sensor and motion planning computer system are configured to cooperate such that the sensor is configured to insert priority data indicative of the detected condition into the motion planning loop; and

wherein the sensor comprises a ladar system, and wherein the ladar system includes a ladar receiver that comprises a protection circuit configured to reduce saturation from high energy interference.

21. The system of claim 20 wherein the protection circuit includes a metallization layer that allows customization of the protection circuit at time of manufacturing.

23. A vehicle motion planning method comprising:

a sensor detecting a threatening or anomalous condition within a field of view, wherein the sensor includes a ladar system;

the sensor interrupting a motion planning loop of a motion planning computer system based on the detected condition;

the ladar system performing compressive sensing to target the field of view with a plurality of ladar pulses; and

the motion planning computer system making a decision about vehicle motion in response to the interrupting; and

wherein the interrupting includes the ladar system asserting a priority flag for receipt by the motion planning computer system in conjunction with a ladar frame.

24. The method of claim 23 wherein the detecting and interrupting are performed within one millisecond.

44. The method of claim 23 further comprising:

the motion planning computer system treating the priority flag as a vector interrupt that alters a data stack used for decision-making by a motion planning loop of the motion planning computer system.

25. A vehicle motion planning method comprising:

a sensor detecting a threatening or anomalous condition within a field of view, wherein the sensor includes a ladar system;

the sensor interrupting a motion planning loop of a motion planning computer system based on the detected condition;

the ladar system performing compressive sensing to target the field of view with a plurality of ladar pulses;

the motion planning computer system making a decision about vehicle motion in response to the interrupting; and

the ladar system re-tasking itself to probe a region within the field of view corresponding to the detected condition with a plurality of ladar pulses.

26. The method of claim 25 wherein the detecting and interrupting are performed within one millisecond.

27. The method of claim 25 wherein the interrupting includes the ladar system asserting a priority flag for receipt by the motion planning computer system in conjunction with a ladar frame.

28. The method of claim 27 further comprising:

the motion planning computer system treating the priority flag as a vector interrupt that alters a data stack used for decision-making by a motion planning loop of the motion planning computer system.

29. The method of claim 25 further comprising:

the ladar system storing a point cloud of range point return data; and

the ladar system exploiting the point cloud to detect the condition.

30. The method of claim 25 wherein the ladar system comprises a ladar transmitter, the ladar transmitter comprising a plurality of scanable mirrors for targeting the ladar transmitter.

31. The method of claim 25 wherein the ladar system comprises a ladar receiver, wherein the ladar receiver includes a detector array, the method further comprising:

the ladar receiver selectively reading out from the detector array based on range point targeting to detect ladar pulse returns.

32. The method of claim 25 wherein the sensor further includes a camera.

33. The method of claim 32 wherein the camera is co-bore sited with respect to the ladar system.

34. The method of claim 32 wherein the camera includes a video controller, the method further comprising:

the video controller reading out image data corresponding to at least a portion of a field of view for the camera; and

the video controller detecting motion within the field of view based on the read out image data.

35. The method of claim 32 wherein the detecting step comprises the camera detecting a shimmer event as the detected condition.

36. The method of claim 32 wherein the detecting step comprises the camera detecting a shiny object event as the detected condition.

37. The method of claim 32 further comprising:

the ladar system and the camera cooperating via cross-cueing.

38. The method of claim 37 further comprising:

the camera generating a shot list for use by the ladar system in response to the detected condition.

39. The method of claim 25 wherein the sensor further comprises a radar.

40. The method of claim 25 wherein the sensor further comprises an acoustic sensor.

41. The method of claim 25 wherein the sensor further comprises a vehicle telematics sensor.

42. The method of claim 25 wherein the detecting step comprises the sensor detecting the condition via at least one of comparative, selective, and/or objective sensing.

43. The method of claim 25 wherein the sensor includes a field programmable gate array (FPGA), wherein the FPGA performs the detecting step.