On-board, computerized landing zone evaluation system for aircraft

• US 10,029,804 B1
• Filed: 05/12/2016
• Issued: 07/24/2018
• Est. Priority Date: 05/14/2015
• Status: Active Grant

• Current Assignee: Near Earth Autonomy, Inc.
• Original Assignee: Near Earth Autonomy, Inc.
• Inventors: Chamberlain, Lyle, Cover, Hugh, Grocholsky, Benjamin, Hamner, Bradley, Scherer, Sebastian, Singh, Sanjiv
• Primary Examiner(s): Smith, Jelani A
• Assistant Examiner(s): Martinez Borrero, Luis A
• Application Number:
  US15/152,944
• Time in Patent Office: 803 Days
• Field of Search: 701 2, 701 54
• CPC Class Codes: B64D 45/08 optical
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G05D 1/0676 specially adapted for landing
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/10036 Multispectral image; Hyperspectral image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0086 for monitoring terrain radar or analogous systems specially adapted for terrain avoidance G01S13/935
• CPC Class Codes: G08G 5/025 Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933

Abstract

A computerized, on-board flight system for an aircraft has a landing-zone evaluation system for evaluating a landing zone for the aircraft based on sensory data from numerous sensing systems, including lidar, radar, and/or various cameras, to identify potential, suitable landing sites in the landing zone. Based on data from these sensory systems, the landing-zone evaluation system geometrically and semantically evaluates the landing zone'"'"'s terrain, as well as identifies moving and stationary objects in the landing zone to identify the suitable landing sites for the aircraft.

Claims

1. A landing zone evaluation system for evaluating a landing zone for an aircraft, the landing system evaluation system comprising:

a plurality of sensor systems that comprise;

a lidar scanning system that continuously scans the landing zone as the aircraft travels toward the landing zone;

a camera system comprising one or more cameras pointed at the landing zone for continuously capturing image data of the landing zone as the aircraft travels toward the landing zone; and

an inertial navigation system (INS); and

an on-board computer system that is on-board the aircraft, wherein the on-board computer system is in communication with the plurality of sensor systems, and wherein the on-board computer system is programmed to;

continuously generate updated registered point clouds as the aircraft travels toward the landing zone based on data from the plurality of data systems, including data from the lidar scanning system and data from the INS;

continuously geometrically analyze points in the registered point clouds corresponding to the landing zone'"'"'s terrain as the aircraft travels toward the landing zone to identify potential landing sites in the landing zone, wherein the geometric analysis comprises;

determining points in the registered point clouds that correspond to a ground surface of the landing zone;

identifying whether or not there are one or more above-ground objects in the landing zone based on clusters of points above the ground surface in the registered point clouds; and

identifying low-deviation, generally level planes in the landing zone, continuously semantically analyze points in the registered point cloud corresponding to the landing zone'"'"'s terrain as the aircraft travels toward the landing zone, based on data from the plurality of sensor systems, including the image data from the camera system; and

identify one or more suitable landing sites in the landing zone based on the continuously updated geometric and semantic analysis, wherein the identification of the one or more suitable landing sites considers whether there is an above-ground object in the landing zone, the size of the aircraft, possible landing orientations for the aircraft, and a size of a low-deviation, generally level area in the landing zone.

2. The landing zone evaluation system of claim 1, wherein:

the aircraft comprises an autonomous aircraft;

the on-board computer system comprises a flight planning module that continuously determines an updated flight path for the autonomous aircraft based on the one or more suitable landing sites identified by the on-board computer system.

3. The landing zone evaluation system of claim 1, wherein the on-board computer system assigns points in the registered point clouds with geometric-analysis confidence scores that indicate the points'"'"' suitability for landing the aircraft based on the geometric analysis.

4. The landing zone evaluation system of claim 3, wherein the on-board computer system assigns points in the registered point clouds with semantic-analysis confidence scores that indicate the points'"'"' suitability for landing the aircraft based on the semantic analysis.

5. The landing zone evaluation system of claim 4, wherein the on-board computer system uses the geometric-analysis confidence scores and the semantic-analysis confidence scores to identify the one or more suitable landing sites in the landing zone.

6. The landing zone evaluation system of claim 1, wherein:

the aircraft comprises an autonomous aircraft;

the on-board computer system comprises a flight planning module that continuously determines an updated flight path for the autonomous aircraft; and

the flight planning module determines the updated flight path for the autonomous vehicle based on the one or more suitable landing sites identified by the on-board computer system.

7. The landing zone evaluation system of claim 6, wherein the on-board computer system is further programmed to:

continuously generate updated 3D maps of the vicinity of the aircraft as the aircraft travels toward the landing zone based on the registered point clouds, wherein the 3D maps comprise a plurality of cells extending in three dimensions, and each cell indicates a likelihood of an obstacle for the aircraft being in a location corresponding to the cell'"'"'s location in the 3D map;

determine, for cells in the 3D maps, a direction of and distance to a closet obstacle; and

the flight planning module uses the direction of and distance to the closest obstacle for the cells in the 3D maps in determining the flight path for the aircraft.

8. The landing zone evaluation system of claim 1, wherein the on-board computer system is further programmed to:

continuously generate updated 3D maps of the vicinity of the aircraft as the aircraft travels toward the landing zone based on the registered point clouds, wherein the 3D maps comprise a plurality of cells extending in three dimensions, and each cell indicates a likelihood of an obstacle for the aircraft being in a location corresponding to the cell'"'"'s location in the 3D map; and

determine, for cells in the 3D maps, a direction of and distance to a closet obstacle.

19. The landing zone evaluation system of claim 1, wherein:

identifying whether or not there is one or more objects in the landing zone further comprises determining whether there is a moving object in the landing zone by comparing registered point clouds for the landing zone from multiple time instances; and

the step of identification of the one or more suitable landing sites further considers whether there is a moving object in the landing zone.

9. A method for evaluating a landing zone for an aircraft as the aircraft travels toward the landing zone, the method comprising:

sensing data with a plurality of sensor systems from the aircraft as the aircraft travels toward the landing zone, wherein the plurality of sensor systems comprises;

a lidar scanning system that continuously scans the landing zone as the aircraft travels toward the landing zone;

a camera system comprising one or more cameras pointed at the landing zone for continuously capturing image data of the landing zone as the aircraft travels toward the landing zone; and

an inertial navigation system (INS); and

continuously generating, with an on-board computer system that is on-board the aircraft and that is in communication with the plurality of sensor systems, updated registered point clouds as the aircraft travels toward the landing zone, wherein the registered point clouds are generated based on data from the plurality of data systems, including data from the lidar scanning system and data from the INS;

continuously geometrically analyzing, by the on-board computer system, points in the registered point clouds corresponding to the landing zone'"'"'s terrain as the aircraft travels toward the landing zone to identify potential landing sites in the landing zone;

wherein the geometric analysis comprises;

determining points in the registered point clouds that correspond to a ground surface of the landing zone;

identifying whether or not there are one or more above-ground objects in the landing zone based on clusters of points above the ground surface in the registered point clouds, and identifying low-deviation, generally level planes in the landing zone;

continuously semantically analyzing, by the on-board computer system, points in the registered point cloud corresponding to the landing zone'"'"'s terrain as the aircraft travels toward the landing zone, based on data from the plurality of sensor systems, including the image data from the camera system; and

identifying, by the on-board computer system, one or more suitable landing sites in the landing zone based on the continuously updated geometric and semantic analysis, wherein the identification of the one or more suitable landing sites considers whether there is an above-ground object in the landing zone, the size of the aircraft, possible landing orientations for the aircraft, and a size of a low-deviation, generally level area in the landing zone.

10. The method of claim 9, wherein:

the aircraft comprises an autonomous aircraft with a flight planning module; and

the method further comprises determining, by the flight planning module, updated flight paths for the autonomous aircraft as the autonomous aircraft travels toward the landing zone based on the one or more suitable landing sites identified by the on-board computer system.

11. The method of claim 9, wherein geometrically analyzing the points in the registered point clouds comprises assigning geometric-analysis confidence scores to the points that indicate the points'"'"' suitability for landing the aircraft based on the geometric analysis.

12. The method of claim 11, wherein semantically analyzing the points in the registered point clouds comprises assigning semantic-analysis confidence scores to the point that indicate the points'"'"' suitability for landing the aircraft based on the semantic analysis.

13. The method of claim 12, wherein the on-board computer system uses the geometric-analysis confidence scores and the semantic-analysis confidence scores to identify the one or more suitable landing sites in the landing zone.

14. The method of claim 13, wherein:

the method further comprises identifying, by the on-board computer system, objects in the landing zone based on the continuously updated registered point clouds; and

the step of identifying the one or more suitable landing sites further uses the identification of objects in the landing zone to identify the one or more suitable landing sites.

15. The method of claim 14, wherein:

the aircraft comprises an autonomous aircraft that comprises a flight planning module; and

the method further comprises continuously determining, by the flight planning module, an updated flight path for the autonomous aircraft based in part on the one or more suitable landing sites identified by the on-board computer system.

16. The method of claim 15, further comprising, by the on-board computer system:

continuously generating updated 3D maps of the vicinity of the aircraft as it travels toward the landing zone based on the registered point clouds, wherein the 3D maps comprise a plurality of cells extending in three dimensions, and each cell indicates a likelihood of an obstacle for the aircraft being in a location corresponding to the cell'"'"'s location in the 3D map;

determining, for cells in the 3D maps, a direction of and distance to a closet obstacle, wherein the flight planning module uses the direction of and distance to the closest obstacle for the cells in the 3D maps in determining the flight path for the aircraft.

17. An aircraft comprising:

propulsion means for propelling the aircraft;

a plurality of sensor systems that comprise;

a lidar scanning system that continuously scans a landing zone for the aircraft as the aircraft travels toward the landing zone;

a camera system comprising one or more cameras pointed at the landing zone for the aircraft for continuously capturing image data of the landing zone as the aircraft travels toward the landing zone; and

an inertial navigation system (INS); and

an on-board computer system that is on-board the aircraft, wherein the on-board computer system is in communication with the plurality of sensor systems, and wherein the on-board computer system is programmed to;

continuously generate updated registered point clouds as the aircraft travels toward the landing zone, wherein the registered point clouds are generated based on data from the plurality of data systems, including data from the lidar scanning system and data from the INS;

continuously geometrically analyze points in the registered point clouds as the aircraft travels toward the landing zone to identify potential landing sites in the landing zone, wherein the geometric analysis comprises;

determining points in the registered point clouds that correspond to a ground surface of the landing zone;

identifying whether or not there are one or more above-ground objects in the landing zone based on clusters of points above the ground surface in the registered point clouds, and identifying low-deviation, generally level planes in the landing zone;

continuously semantically analyze points in the registered point cloud as the aircraft travels toward the landing zone, based on data from the plurality of sensor systems, including the image data from the camera system; and

identify one or more suitable landing sites in the landing zone based on the continuously updated geometric and semantic analysis, wherein the identification of the one or more suitable landing sites considers whether there is an above-ground object in the landing zone, the size of the aircraft, possible landing orientations for the aircraft, and a size of a low-deviation, generally level area in the landing zone.

18. The aircraft of claim 17, wherein:

the aircraft comprises an autonomous aircraft; and

on-board computer system is further programmed to;

continuously determine an updated flight path for the autonomous aircraft based in part on the one or more suitable landing sites; and

determine control signals for the propulsion means based on the updated flight path.