Adaptive control of ladar system camera using spatial index of prior ladar return data

• US 10,656,277 B1
• Filed: 03/18/2019
• Issued: 05/19/2020
• Est. Priority Date: 10/25/2018
• Status: Active Grant

• Current Assignee: AEye, Inc.
• Original Assignee: AEye, Inc.
• Inventors: Dussan, Luis Carlos, Steinhardt, Allan, Presutti, Federico, Benscoter, Joel David
• Primary Examiner(s): Bolda, Eric L
• Application Number:
  US16/356,101
• Time in Patent Office: 428 Days
• CPC Class Codes: G01S 17/006 Theoretical aspects
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/93 for anti-collision purposes
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4802 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G01S 7/4814 of transmitters alone
• CPC Class Codes: G01S 7/4816 of receivers alone
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: G01S 7/484 Transmitters
• CPC Class Codes: G01S 7/4861 Circuits for detection, sampling, integration or read-out
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates

Abstract

Disclosed herein are examples of ladar systems and methods where data about a plurality of ladar returns from prior ladar pulse shots gets stored in a spatial index that associates ladar return data with corresponding locations in a coordinate space to which the ladar return data pertain. This spatial index can then be accessed by a processor to retrieve ladar return data for locations in the coordinate space that are near a range point to be targeted by the ladar system with a new ladar pulse shot. This nearby prior ladar return data can then be analyzed by the ladar system to help define a control setting for use by a ladar system camera.

Claims

1. A ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a camera configured to generate images of a field of view that encompasses the coordinate space;

a memory configured to spatially index ladar return data for prior ladar pulse shots by location in the coordinate space, wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space; and

a processor configured to (1) traverse the tree structure based on an identified range point location in the coordinate space to identify leaf nodes in the tree structure that include ladar return data associated with locations within a defined vicinity around the identified range point location, (2) extract ladar return data from the identified leaf nodes, and (3) define a control setting for the camera based on an analysis of the extracted ladar return data; and

wherein the camera is further configured to generate new images of the field of view based on the defined control setting.

2. The system of claim 1 wherein the tree structure is organized in the memory as a quad tree index.

3. The system of claim 1 wherein the defined control setting comprises a light setting for the camera.

4. The system of claim 3 wherein the light setting comprises a shutter setting for the camera.

5. The system of claim 4 wherein the processor is further configured to adjust the shutter setting so that the camera shutter remains open longer in response to a determination that a dark object is present the field of view in a night environment.

6. The system of claim 1 wherein the processor is further configured to process the images to identify areas within the field of view for interrogation via ladar pulse shots.

7. The system of claim 6 wherein the ladar transmitter is configured to target the identified areas with new ladar pulse shots.

8. The system of claim 7 wherein the ladar transmitter includes a plurality of scanable mirrors;

wherein the processor is further configured to dynamically schedule a plurality of ladar pulse shots for the ladar transmitter to target a plurality of range points in the coordinate space including range points in the identified areas;

wherein the ladar transmitter is further configured to (1) controllably scan the scanable mirrors to target the ladar transmitter at the targeted range points in accordance with the scheduled ladar pulse shots, and (2) transmit a plurality of ladar pulse shots toward the targeted range points via the controllably scanned mirrors.

9. The system of claim 8 wherein the processor is further configured to perform the dynamic scheduling on a shot-by-shot basis.

10. The system of claim 1 wherein the processor is further configured to repeat the define operation on a shot-by-shot basis for a plurality of new ladar pulse shots.

11. The system of claim 10 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

12. The system of claim 1 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) define the control setting for the camera based on the interpolation.

13. The system of claim 1 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

14. The system of claim 1 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

15. The system of claim 1 wherein the ladar return data comprises (1) noise level data for the ladar returns from prior ladar pulse shots, and/or (2) pulse shape data for the ladar returns from prior ladar pulse shots.

16. The system of claim 1 wherein the coordinate space comprises an azimuth-elevation grid.

17. The system of claim 1 wherein the spatial index spatially indexes the ladar return data for a ladar frame.

19. The system of claim 1 wherein the processor comprises a field programmable gate array (FPGA).

18. A ladar system comprising:

a ladar transmitter configured to transmit a plurality of ladar pulse shots into a coordinate space toward a plurality of range points;

a camera configured to generate images of a field of view that encompasses the coordinate space;

a memory configured to spatially index ladar return data for prior ladar pulse shots by location in the coordinate space; and

a processor configured to define a control setting for the camera based on a look up and analysis of spatially indexed ladar return data stored in the memory for locations in the coordinate space that are within a defined vicinity around a targeted location in the coordinate space, wherein the defined vicinity is an adjustable value; and

wherein the camera is further configured to generate new images of the field of view based on the defined control setting.

39. The system of claim 18 wherein the spatially-indexed ladar return data is organized in the memory as a quad tree index.

40. The system of claim 18 wherein the defined control setting comprises a light setting for the camera.

41. The system of claim 40 wherein the light setting comprises a shutter setting for the camera.

42. The system of claim 41 wherein the processor is further configured to adjust the shutter setting so that the camera shutter remains open longer in response to a determination that a dark object is present the field of view in a night environment.

43. The system of claim 18 wherein the processor is further configured to process the images to identify areas within the field of view for interrogation via ladar pulse shots.

44. The system of claim 43 wherein the ladar transmitter is configured to target the identified areas with new ladar pulse shots.

45. The system of claim 44 wherein the ladar transmitter includes a plurality of scanable mirrors;

wherein the processor is further configured to dynamically schedule a plurality of ladar pulse shots for the ladar transmitter to target a plurality of range points in the coordinate space including range points in the identified areas;

wherein the ladar transmitter is further configured to (1) controllably scan the scanable mirrors to target the ladar transmitter at the targeted range points in accordance with the scheduled ladar pulse shots, and (2) transmit a plurality of ladar pulse shots toward the targeted range points via the controllably scanned mirrors.

46. The system of claim 45 wherein the processor is further configured to perform the dynamic scheduling on a shot-by-shot basis.

47. The system of claim 18 wherein the processor is further configured to repeat the define operation on a shot-by-shot basis for a plurality of new ladar pulse shots.

48. The system of claim 18 wherein the spatial index comprises a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space.

49. The system of claim 48 wherein the processor is further configured to (1) traverse the tree structure based on the identified range point location to identify leaf nodes in the tree structure that include ladar return data associated with locations within the defined vicinity of the identified range point location, and (2) extract ladar return data from the identified leaf nodes.

50. The system of claim 49 wherein the processor is further configured to (1) perform the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) define the camera setting based on the interpolation.

51. The system of claim 48 wherein the processor is further configured to update the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

52. The system of claim 18 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

53. The system of claim 18 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

54. The system of claim 18 wherein the ladar return data comprises (1) noise level data for the ladar returns from prior ladar pulse shots, and/or (2) pulse shape data for the ladar returns from prior ladar pulse shots.

55. The system of claim 18 wherein the coordinate space comprises an azimuth-elevation grid.

56. The system of claim 18 wherein the spatial index spatially indexes the ladar return data for a ladar frame.

57. The system of claim 18 wherein the processor comprises a field programmable gate array (FPGA).

20. A method for adaptively controlling a ladar system that interrogates a coordinate space with a plurality of ladar pulse shots, the method comprising:

spatially indexing ladar return data for a plurality of prior ladar pulse shots by location in the coordinate space to create a tree structure having a root node, a plurality of branches, and a plurality of leaf nodes, wherein the leaf nodes associate the ladar return data with corresponding locations in the coordinate space;

transmitting a plurality of ladar pulse shots into the coordinate space toward a plurality of range points;

a camera generating images of a field of view that encompasses the coordinate space;

traversing the tree structure based on an identified range point location in the coordinate space to identify leaf nodes in the tree structure that include ladar return data associated with locations within a defined vicinity around the identified range point location;

extracting ladar return data from the identified leaf nodes;

defining a control setting for the camera based on an analysis of the extracted ladar return data; and

the camera generating new images of the field of view based on the defined control setting.

21. The method of claim 20 wherein the spatially indexing step comprises arranging the tree structure in a memory as a quad tree index.

22. The method of claim 20 wherein the defined control setting comprises a light setting for the camera.

23. The method of claim 22 wherein the light setting comprises a shutter setting for the camera.

24. The method of claim 23 further comprising:

determining that a dark object is present the field of view in a night environment; and

wherein the defining step comprises defining the shutter setting in response to the determination that the dark object is present the field of view in the night environment so that a shutter of the camera remains open longer;

adjusting the camera shutter in accordance with the defined shutter setting.

25. The method of claim 20 further comprising:

processing the images to identify areas within the field of view for interrogation via ladar pulse shots.

26. The method of claim 25 further comprising

a ladar transmitter targeting the identified areas with new ladar pulse shots.

27. The method of claim 26 wherein the ladar transmitter transmits ladar pulse shots into the coordinate space via a plurality of scanning mirrors, the method further comprising:

dynamically scheduling a plurality of ladar pulse shots for the ladar transmitter to target a plurality of range points in the coordinate space including range points in the identified areas;

the ladar transmitter controllably scanning the mirrors to target the ladar transmitter at the targeted range points in accordance with the scheduled ladar pulse shots; and

the ladar transmitter transmitting a plurality of ladar pulse shots toward the targeted range points via the scanning mirrors.

28. The method of claim 27 wherein the dynamically scheduling step comprises performing the dynamic scheduling on a shot-by-shot basis.

29. The method of claim 20 further comprising:

repeating the defining step on a shot-by-shot basis for a plurality of new ladar pulse shots.

30. The method of claim 29 further comprising:

updating the tree structure leaf nodes with ladar return data from the new ladar pulse shots.

31. The method of claim 20 further comprising a processor (1) performing the analysis such that the analysis includes an interpolation of a plurality of data values within the extracted ladar return data, and (2) defining the control setting for the camera based on the interpolation.

32. The method of claim 20 wherein the ladar return data comprises intensity data for the ladar returns from prior ladar pulse shots.

33. The method of claim 20 wherein the ladar return data comprises range data for the ladar returns from prior ladar pulse shots.

34. The method of claim 20 wherein the ladar return data comprises (1) noise level data for the ladar returns from prior ladar pulse shots, and/or (2) pulse shape data for the ladar returns from prior ladar pulse shots.

35. The method of claim 20 wherein the coordinate space comprises an azimuth-elevation grid.

36. The method of claim 20 wherein the tree structure spatially indexes the ladar return data for a ladar frame.

37. The method of claim 20 wherein the traversing, extracting, and defining steps are performed by a field programmable gate array (FPGA).

38. A method for adaptively controlling a ladar system that interrogates a coordinate space with a plurality of ladar pulse shots, the method comprising:

spatially indexing ladar return data for a plurality of prior ladar pulse shots by location in the coordinate space;

transmitting a plurality of ladar pulse shots into the coordinate space toward a plurality of range points;

a camera generating images of a field of view that encompasses the coordinate space;

defining a control setting for the camera based on a look up and analysis of spatially indexed ladar return data stored for locations in the coordinate space that are within a defined vicinity around a targeted location in the coordinate space;

the camera generating new images of the field of view based on the defined control setting;

adjusting the defined vicinity; and

repeating the transmitting step, the defining step, and the generating new images step for a new ladar pulse shot using the adjusted defined vicinity.