Adaptive control of Ladar shot selection using spatial index of prior Ladar return data

• US 10,598,788 B1
• Filed: 03/18/2019
• Issued: 03/24/2020
• Est. Priority Date: 10/25/2018
• Status: Active Grant

• Current Assignee: AEye, Inc.
• Original Assignee: AEye, Inc.
• Inventors: Dussan, Luis Carlos, Steinhardt, Allan, Presutti, Federico, Benscoter, Joel David
• Primary Examiner(s): Bolda, Eric L
• Application Number:
  US16/356,089
• Time in Patent Office: 372 Days
• CPC Class Codes: G01S 17/006 Theoretical aspects
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/93 for anti-collision purposes
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4814 of transmitters alone
• CPC Class Codes: G01S 7/4816 of receivers alone
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/484 Transmitters
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates

Abstract

Disclosed herein are examples of ladar systems and methods where data about a plurality of ladar returns from prior ladar pulse shots gets stored in a spatial index that associates ladar return data with corresponding locations in a coordinate space to which the ladar return data pertain. This spatial index can then be accessed by a processor to retrieve ladar return data for locations in the coordinate space that are near a range point to be targeted by the ladar system with a new ladar pulse shot. This nearby prior ladar return data can then be analyzed by the ladar system to help adapt a shot selection for use by the ladar system with respect to new ladar pulse shots.

Claims

1. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a ladar receiver configured to receive incident light and detect ladar returns based on the received incident light;

a memory configured to store data about a plurality of ladar returns from prior ladar pulse shots in a spatial index, the spatial index associating the ladar return data with a plurality of locations in the coordinate space; and

a processor configured to adapt a shot selection for the ladar transmitter based on an analysis of the ladar return data in the spatial index associated with locations within a defined vicinity of an identified location in the coordinate space, wherein the adapted shot selection includes a new ladar pulse shot that targets a defined range point in the coordinate space;

wherein the ladar transmitter is further configured to transmit the new ladar pulse shot toward the defined range point; and

wherein the processor is further configured to add synthetic ladar return data to a plurality of range points in a ladar frame based on the analysis, and wherein the adapted shot selection comprises an elimination of a plurality of ladar shots that were targeted toward range points corresponding to the added synthetic ladar return data.

2. The system of claim 1 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

3. The system of claim 2 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

4. The system of claim 3 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) adapt the shot selection based on the interpolation.

5. The system of claim 2 wherein the processor is further configured to repeat the adapt operation for a plurality of new ladar pulse shots.

6. The system of claim 5 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

7. The system of claim 1 wherein the spatial index comprises a quad tree index.

8. The system of claim 1 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

9. The system of claim 1 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

10. The system of claim 1 wherein the processor is further configured to compute a static scene intensity map based on the analysis.

11. The system of claim 10 further comprising:

a camera that generates camera data of the coordinate space over time; and

wherein the processor is further configured to (1) predict a future location of a moving object in the coordinate space based on the camera data, and (2) adapt the shot selection so that new ladar pulse shots will target the predicted future location of the moving object.

12. The system of claim 1 wherein the adapted shot selection comprises an adapted shot location for the new ladar pulse shot.

13. The system of claim 1 wherein the adapted shot selection comprises an adapted shot time for the new ladar pulse shot.

14. The system of claim 1 wherein the ladar transmitter includes a plurality of scanable mirrors;

wherein the processor is further configured to dynamically schedule a plurality of ladar pulse shots for the ladar transmitter to target a plurality of range points in the coordinate space;

wherein the ladar transmitter is further configured to (1) controllably scan the scanable mirrors to target the ladar transmitter at the targeted range points in accordance with the scheduled ladar pulse shots, and (2) transmit a plurality of ladar pulse shots toward the targeted range points via the controllably scanned mirrors.

15. The system of claim 14 wherein the processor is further configured to perform the dynamic scheduling on a shot-by-shot basis.

16. The system of claim 1 wherein the processor comprises a field programmable gate array (FPGA).

17. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a ladar receiver configured to receive incident light and detect ladar returns based on the received incident light;

a memory configured to store data about a plurality of ladar returns from prior ladar pulse shots in a spatial index, the spatial index associating the ladar return data with a plurality of locations in the coordinate space; and

a processor configured to adapt a shot selection for the ladar transmitter based on an analysis of the ladar return data in the spatial index associated with locations within a defined vicinity of an identified location in the coordinate space, wherein the adapted shot selection includes a new ladar pulse shot that targets a defined range point in the coordinate space;

wherein the ladar transmitter is further configured to transmit the new ladar pulse shot toward the defined range point;

wherein the processor is further configured to compute a static scene intensity map based on the analysis; and

wherein the adaptive shot selection comprises a reduction in a number of ladar shots that are targeted toward elements of the static scene intensity map.

18. The system of claim 17 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

19. The system of claim 18 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

20. The system of claim 19 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) adapt the shot selection based on the interpolation.

21. The system of claim 18 wherein the processor is further configured to repeat the adapt operation for a plurality of new ladar pulse shots.

22. The system of claim 21 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

23. The system of claim 17 wherein the spatial index comprises a quad tree index.

24. The system of claim 17 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

25. The system of claim 17 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

26. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a ladar receiver configured to receive incident light and detect ladar returns based on the received incident light;

a memory configured to store data about a plurality of ladar returns from prior ladar pulse shots in a spatial index, the spatial index associating the ladar return data with a plurality of locations in the coordinate space; and

a processor configured to adapt a shot selection for the ladar transmitter based on an analysis of the ladar return data in the spatial index associated with locations within a defined vicinity of an identified location in the coordinate space, wherein the adapted shot selection includes a new ladar pulse shot that targets a defined range point in the coordinate space;

wherein the ladar transmitter is further configured to transmit the new ladar pulse shot toward the defined range point; and

wherein the ladar transmitter and the processor are configured to generate a rolling ladar frame based on a rolling shutter technique.

27. The system of claim 26 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

28. The system of claim 27 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

29. The system of claim 28 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) adapt the shot selection based on the interpolation.

30. The system of claim 27 wherein the processor is further configured to repeat the adapt operation for a plurality of new ladar pulse shots.

31. The system of claim 30 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

32. The system of claim 26 wherein the spatial index comprises a quad tree index.

33. The system of claim 26 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

34. The system of claim 26 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

35. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a ladar receiver configured to receive incident light and detect ladar returns based on the received incident light;

a memory configured to store data about a plurality of ladar returns from prior ladar pulse shots in a spatial index, the spatial index associating the ladar return data with a plurality of locations in the coordinate space; and

a processor configured to adapt a shot selection for the ladar transmitter based on an analysis of the ladar return data in the spatial index associated with locations within a defined vicinity of an identified location in the coordinate space, wherein the adapted shot selection includes a new ladar pulse shot that targets a defined range point in the coordinate space;

wherein the ladar transmitter is further configured to transmit the new ladar pulse shot toward the defined range point;

wherein the adapted shot selection comprises an adapted shot location for the new ladar pulse shot; and

wherein the processor is further configured to (1) compute optical flow data for a plurality of ladar frames, and (2) use the spatial index to interpolate intensity data with respect to the computed optical flow data to build a static object intensity map for a scene.

36. The system of claim 35 wherein the processor is further configured to define a plurality of ladar shots that target moving objects in the scene.

37. The system of claim 36 further comprising a camera that generates camera data of the scene over time;

and

wherein the processor is further configured to (1) predict a future location of a moving object in the scene based on the camera data, and (2) define a ladar shot for a subsequent ladar frame that targets the predicted future location of the moving object.

38. The system of claim 35 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

39. The system of claim 38 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

40. The system of claim 39 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) adapt the shot selection based on the interpolation.

41. The system of claim 38 wherein the processor is further configured to repeat the adapt operation for a plurality of new ladar pulse shots.

42. The system of claim 41 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

43. The system of claim 35 wherein the spatial index comprises a quad tree index.

44. The system of claim 35 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

45. The system of claim 35 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

46. The system of claim 35 wherein the processor is further configured to add synthetic ladar return data to a plurality of range points in a ladar frame based on the analysis, and wherein the adapted shot selection comprises an elimination of a plurality of ladar shots that were targeted toward range points corresponding to the added synthetic ladar return data.

47. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a ladar receiver configured to receive incident light and detect ladar returns based on the received incident light;

a memory configured to store data about a plurality of ladar returns from prior ladar pulse shots in a spatial index, the spatial index associating the ladar return data with a plurality of locations in the coordinate space; and

a processor configured to adapt a shot selection for the ladar transmitter based on an analysis of the ladar return data in the spatial index associated with locations within a defined vicinity of an identified location in the coordinate space, wherein the adapted shot selection includes a new ladar pulse shot that targets a defined range point in the coordinate space;

wherein the ladar transmitter is further configured to transmit the new ladar pulse shot toward the defined range point

wherein the adapted shot selection comprises an adapted shot time for the new ladar pulse shot; and

wherein the processor is further configured to (1) compute optical flow data for a plurality of ladar frames, and (2) use the spatial index to interpolate intensity data with respect to the computed optical flow data to build a static object intensity map for a scene.

48. The system of claim 47 wherein the processor is further configured to define a plurality of ladar shots that target moving objects in the scene.

49. The system of claim 48 further comprising a camera that generates camera data of the scene over time;

and

wherein the processor is further configured to (1) predict a future location of a moving object in the scene based on the camera data, and (2) define a ladar shot for a subsequent ladar frame that targets the predicted future location of the moving object.

50. The system of claim 47 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

51. The system of claim 50 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

52. The system of claim 51 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) adapt the shot selection based on the interpolation.

53. The system of claim 50 wherein the processor is further configured to repeat the adapt operation for a plurality of new ladar pulse shots.

54. The system of claim 53 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

55. The system of claim 47 wherein the spatial index comprises a quad tree index.

56. The system of claim 47 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

57. The system of claim 47 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

58. A method for adaptively controlling a ladar system that interrogates a coordinate space with a plurality of ladar pulse shots, the method comprising:

spatially indexing ladar return data for a plurality of prior ladar pulse shots by location in the coordinate space;

adapting a shot selection for the ladar system based on a look up and analysis of spatially indexed ladar return data for locations in the coordinate space that are within a defined vicinity around a targeted location in the coordinate space;

transmitting ladar pulse shots toward targeted locations in the coordinate space based on the adapted shot selection; and

adding synthetic ladar return data to a plurality of range points in a ladar frame based on the analysis, and wherein the adapted shot selection comprises an elimination of a plurality of ladar shots that were targeted toward range points corresponding to the added synthetic ladar return data.

59. An adaptive ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a memory that spatially indexes ladar return data for prior ladar pulse shots by location in the coordinate space; and

a processor configured to compute a static scene intensity map based on a look up and analysis of spatially indexed ladar return data stored in the memory for locations in the coordinate space that are within a defined vicinity around locations in the coordinate space classified as being static objects.

60. The system of claim 59 wherein the processor is further configured to compute the static scene intensity map based on the look up and analysis of spatially indexed ladar return data stored in the memory for locations in the coordinate space that are within a defined vicinity around optical flows in the coordinate space.

61. The system of claim 59 further comprising:

a camera that generates camera data of the coordinate space over time; and

wherein the processor is further configured to (1) predict a future location of a moving object in the scene based on the camera data, and (2) define a ladar shot for a ladar frame that targets the predicted future location of the moving object.

62. The system of claim 61 wherein the ladar transmitter is further configured to transmit the defined ladar shot toward the predicted future location.

63. The system of claim 59 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

64. The system of claim 63 wherein the processor is further configured to (1) traverse the tree structure based on the identified location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified location, and (2) extract ladar return data from the identified leaf nodes.

65. The system of claim 64 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) compute the static scene intensity map based on the interpolation.

66. The system of claim 63 wherein the processor is further configured to update the static scene intensity map for a plurality of new ladar pulse shots.

67. The system of claim 66 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

68. The system of claim 59 wherein the spatial index comprises a quad tree index.

69. The system of claim 59 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

70. The system of claim 59 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.