Semantics based safe landing area detection for an unmanned vehicle

• US 9,177,481 B2
• Filed: 12/13/2013
• Issued: 11/03/2015
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: Sikorsky Aircraft Corporation (Martin Marietta Corporation)
• Original Assignee: Sikorsky Aircraft Corporation (Martin Marietta Corporation)
• Inventors: Wang, Hongcheng, Xiong, Ziyou, Derenick, Jason C., Stathis, Christopher, Cherepinsky, Igor
• Primary Examiner(s): Mawari, Redhwan k
• Application Number:
  US14/105,759
• Publication Number:
  US 20150170526A1
• Time in Patent Office: 690 Days
• Field of Search: 701/27, 701/28, 701/16, 700/259
• US Class Current: 1/1
• CPC Class Codes: B64C 19/00 Aircraft control not otherwise provided for
• CPC Class Codes: G01C 23/005 Flight directors indicating arrangements specially adapted for rotary gyroscopes G01C19/32
• CPC Class Codes: G01S 13/935 for terrain-avoidance
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/51 Display arrangements
• CPC Class Codes: G05D 1/0676 specially adapted for landing
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G08G 5/02 Automatic approach or landing aids, i.e. systems in which flight data of incoming planes are processed to provide landing data landing aids fitted in or to aircraft B64D45/04; visual or acoustic landing aids B64F1/18

Abstract

A method for determining a suitable landing area for an aircraft includes receiving signals indicative of Light Detection And Ranging (LIDAR) information for a terrain via a LIDAR perception system; receiving signals indicative of image information for the terrain via a camera perception system; evaluating, with the processor, the LIDAR information and generating information indicative of a LIDAR landing zone candidate region; co-registering in a coordinate system, with the processor, the LIDAR landing zone candidate region and the image information; segmenting, with the processor, the co-registered image and the LIDAR landing zone candidate region to generate segmented regions; classifying, with the processor, the segmented regions into semantic classes; determining, with the processor, contextual information in the semantic classes; and ranking and prioritizing the contextual information.

Claims

1. A method for determining a suitable landing area for an aircraft, comprising:

receiving signals indicative of terrain information for a terrain via a three-dimension (3D) perception system;

receiving signals indicative of image information for the terrain via a camera perception system, the image information separate from the terrain information;

evaluating, with the processor, the terrain information and generating information indicative of a landing zone candidate region;

co-registering in a coordinate system, with the processor, the landing zone candidate region and the image information;

segmenting, with the processor, an image region from the image information corresponding to the landing zone candidate region to generate segmented regions;

classifying, with the processor, the segmented regions into semantic classes;

determining, with the processor, contextual information from a contextual model and using the contextual information to detect an error in at least one semantic classes; and

ranking and prioritizing the semantic classes.

2. The method of claim 1, wherein the receiving of the terrain information further comprises receiving the signals indicative of Light Detection And Ranging (LIDAR) terrain information for the terrain via a LIDAR perception system.

3. The method of claim 1, wherein the receiving of the terrain information further comprises receiving the terrain data as a 3D point cloud data.

4. The method of claim 3, wherein the evaluating of the terrain information further comprises evaluating the 3D point cloud data for at least a slope, a residual of fitting, and a variance of height from the terrain information.

5. The method of claim 4, wherein the evaluating of the 3D point cloud data further comprises identifying at least one of a slope, a residual of fitting, and a variance of height from the segmented regions.

6. The method of claim 1, wherein the co-registering further comprises receiving visible or multi-spectral image data from the image information.

7. The method of claim 1, further comprising adaptively segmenting the landing zone candidate region based on the aircraft status.

8. The method of claim 1, further comprising identifying at least one of a visual feature or a depth feature in the segmented regions.

9. The method of claim 8, wherein the identifying of the at least one of the visual or the depth feature further comprises identifying at least one of color, texture, and appearance from the image information or at least one of a slope, a residual of fitting, and a variance of height from the terrain information corresponding to each segmented region.

10. The method of claim 1, wherein the classifying of the segmented regions further comprises categorizing images in the segmented regions into the semantic classes.

11. The method of claim 1, further comprising determining a suitable landing zone from the prioritized ranked semantic classes.

12. The method of claim 1, wherein the ranking and prioritizing the semantic classes further comprises prioritizing the semantic classes based on one of priority or suitability.

13. A system for determining a suitable landing area for an aircraft, comprising:

a processor; and

memory having instructions stored thereon that, when executed by the processor, cause the system to;

receive signals indicative of terrain information for a terrain with a three-dimension (3D) perception system;

receive signals indicative of image information for the terrain with a camera perception system, the image information separate from the terrain information;

evaluate the terrain information and generate information indicative of a landing zone candidate region;

co-register in a coordinate system the landing zone candidate region and the image information;

segment image regions corresponding to the landing zone candidate regions to generate segmented regions;

classify the segmented regions into semantic classes;

determine contextual information from a contextual model and using the contextual information to detect an error in at least one semantic class; and

ranking and prioritizing the semantic class.

14. The system of claim 13, wherein the processor is configured to receive Light Detection And Ranging (LIDAR) terrain information for the terrain via a LIDAR perception system.

15. The system of claim 13, wherein the processor is configured to receive the terrain data as a 3D point cloud data.

16. The system of claim 13, wherein the processor is configured to evaluate the 3D point cloud data for at least one of a slope, a residual of fitting, and a variance of height from the terrain information.

17. The system of claim 13, wherein the processor is configured to receive visible or multi-spectral image data from the image information.

18. The system of claim 13, wherein the processor is configured to adaptively segmenting the landing zone candidate region based on the aircraft status.

19. The system of claim 13, wherein the processor is configured to identify at least one of a visual feature or a depth feature in the segmented regions.

20. The system of claim 13, wherein the processor is configured to identify at least one of a color, a texture, and an appearance from the segmented regions.

21. The system of claim 13, wherein the processor is configured to identify at least one of a slope, a residual of fitting, and a variance of height from the segmented regions.

22. The system of claim 13, wherein the processor is configured to determine a suitable landing zone from the prioritized ranked semantic classes.

23. The system of claim 13, wherein the processor is configured to prioritize the semantic classes based on one of priority or suitability.