Systems and methods for automated landing of a drone

• US 10,551,852 B2
• Filed: 07/20/2017
• Issued: 02/04/2020
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Percepto Robotics Ltd.
• Original Assignee: Percepto Robotics Ltd.
• Inventors: Blonder, Sagi, Benitah, Ariel, Gil-Ad, Eyal
• Primary Examiner(s): Sweeney, Brian P
• Application Number:
  US16/080,669
• Publication Number:
  US 20190227572A1
• Time in Patent Office: 929 Days
• Field of Search: 701 3
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 45/08 optical
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 70/00 Launching, take-off or landing arrangements
• CPC Class Codes: B64U 70/95 Means for guiding the landing UAV towards the platform, e.g. lighting means
• CPC Class Codes: G03B 15/006 Apparatus mounted on flying objects
• CPC Class Codes: G03B 35/02 by sequential recording
• CPC Class Codes: G05D 1/0676 specially adapted for landing
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0086 for monitoring terrain radar or analogous systems specially adapted for terrain avoidance G01S13/935
• CPC Class Codes: G08G 5/025 Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913

Abstract

There is provided a method of automatically landing a drone on a landing pad having thereon guiding-elements arranged in a pattern relative to a central region of the landing pad, comprising: receiving first image(s) captured by a camera of the drone, processing the first image(s) to compute a segmentation mask according to an estimate of a location of the landing pad, receiving second image(s) captured by the camera, processing the second image(s) according to the segmentation mask to compute a segmented region and extracting from the segmented region guiding-element(s), determining a vector for each of the extracted guiding-element(s), and aggregating the vectors to compute an estimated location of the central region of the landing pad, and navigating and landing the drone on the landing pad according to the estimated location of the central region of the landing pad.

Claims

1. A computer implemented method of automatically landing a drone on a landing pad having thereon a plurality of guiding-elements arranged in a pattern relative to a central region of the landing pad, comprising:

receiving at least one first image captured by a camera of the drone directed towards the landing pad;

processing the at least one first image to compute a segmentation mask according to an estimate of a location of the landing pad;

receiving at least one second image captured by the camera;

processing the at least one second image according to the segmentation mask to compute a segmented region and extracting from the segmented region at least one of the plurality of guiding-elements, determining a vector for each of the extracted at least one of the plurality of guiding-elements, and aggregating the vectors to compute an estimated location of the central region of the landing pad; and

navigating and landing the drone on the landing pad according to the estimated location of the central region of the landing pad.

2. The method of claim 1, wherein the drone is automatically vertically landed on the landing pad.

3. The method of claim 1, wherein the processing of the at least one first image comprises:

computing a segmented image according to a segmentation of the landing pad of the at least one first image;

identifying at least one closed region of the segmented image having an area according to an area requirement;

computing a first correlation of each identified at least one closed region with a predefined geometrical shape of the landing pad according to a first correlation requirement denoting a tolerance of similarity of the respective closed region to the predefined geometrical shape, to identify a candidate landing pad;

computing a segmentation mask of the landing pad according to a computed estimate of the location of the central region of the candidate landing pad and an estimate of the location of the edges of the candidate landing pad.

4. The method of claim 3, further comprising:

iteratively clustering intersecting identified closed regions to create an updated set of closed regions, terminating the iteratively clustering of the intersecting identified closed regions when at least two contours are identified having an area above a predefined threshold indicative of a rough pattern of the landing pad, and wherein the first correlation is computed for the updated set of closed regions computed when the iterative clustering terminates.

5. The method of claim 3, further comprising processing the segmented image for removal of effects of undesired features prior to the identification of the at least one closed region.

6. The method of claim 3, further comprising processing the segmented image for removal of guiding-elements prior to the identification of the at least one closed region.

7. The method of claim 3, wherein the predefined geometrical shape of the landing pad comprises a square, and wherein the first correlation requirement defines a tolerance of at least one of:

a ratio of the lengths of the edges of the identified at least one closed region, and an angle between two edges of the identified at least one closed region.

8. The method of claim 3, for each identified at least one closed region intersecting at least one border of the at least one image, computing an estimate of the non-visible portion of the at least one closed region located outside of the at least one image to create an estimate of the complete shape of the at least one closed region, wherein the first correlation is computed for the estimate of the complete shape of the at least one closed region.

9. The method of claim 8, wherein the estimate of the location of the central region of the landing pad is computed according to the estimate of the complete shape of the at least one closed region.

10. The method of claim 3, wherein when the candidate landing pad is spaced apart from boundaries of the at least one first image, further comprising computing a second correlation of an interior closed region of the candidate landing pad with a predefined geometrical shape of the interior of the landing pad according to a second correlation requirement defining tolerance of similarity of the interior closed region and the predefined geometrical shape of the interior of the landing pad, wherein the estimate of the location of the central region is according to the location of the interior closed region of the candidate landing pad.

11. The method of claim 10, further comprising identifying the landing pad according to a third correlation requirement denoting a predefined ratio between an area of the interior closed region of the candidate landing pad and an area of the corresponding larger closed region of the candidate landing pad.

12. The method of claim 1, wherein segmented image computed is computed as a binary segmentation according to a segmentation threshold, wherein a first pixel value of the segmented image denotes a portion of the landing pad and a second pixel value of the segmented image denotes background.

13. The method of claim 12, wherein the segmentation threshold is dynamically computed for each respective first image according to a computed histogram of the respective first image, wherein the segmentation threshold is within a tolerance requirement of a previously computed segmentation threshold for at least one previous first image.

14. The method of claim 1, wherein the extracting of the at least one of plurality of guiding-elements comprises computing a contour of each respective guiding-element by contour finding code and storing in a memory contours according to a contour requirement.

15. The method of claim 1, wherein a magnitude of the vector determined for each extracted guiding-element is computed according to a size of the respective extracted guiding-element, wherein a direction of the vector determined for each extracted guiding-element is computed according to an orientation of a distinct feature of the extracted guiding-element.

16. The method of claim 1, wherein each respective guiding-element is T-shaped, wherein the stem of the T-shaped guiding-element is oriented along the certain radial axis, wherein the stem of the T-shaped guiding-element has a length indicative of the distance from the respective T-shaped guiding-element to the central region, and the cross-bar of the T-shaped guiding-element is located relatively closer to the central region than the base of the T-shaped guiding-element.

17. The method of claim 1, further comprising:

processing the at least one second image to identify a central-mark defining a central region of the landing pad, and defining the estimated location of the central region according to the identified central-mark.

18. The method of claim 1, wherein the segmentation mask comprises a binary mask, the segmented region comprises a binary segmented region, and the at least one of the plurality of guiding-elements are extracted from the binary segmented region according to contour finding code.

19. The method of claim 1, wherein the segmented region is filtered to exclude features below a size requirement prior to the extracting.

20. The method of claim 1, wherein extracting the at least one of the plurality of guiding-elements comprises:

identifying at least one candidate guiding-element;

correlating each candidate guiding-element with a predefined guiding-element template according to a guiding-element correlation requirement.

21. The method of claim 20, wherein the vector associated with each respective candidate guiding-element is determined according to a vector associated with the predefined guiding-template correlated with the respective guiding-element.

22. The method of claim 1, further converting a first color space of each captured first and second image to a second color space according to the predefined color of the landing pad relative to expected colors of the environment in proximity to the landing pad, and processing the converted first and second images.

23. The method of claim 22, wherein the first color space is RGB (red, green, blue), the second color space is LUV, and the processing is performed according to the U channel of the converted first and second images.

24. The method of claim 1, further comprising:

processing the at least one second image and/or processing a received at least one third image captured by the camera, to extract a central-mark from the at least one second image and/or at least one third image, wherein the central-mark defines the central region of the landing pad,

wherein the navigating and landing the drone is performed according to the extracted central-mark.

25. The method of claim 24, wherein the extraction of the central-mark from the at least one second image and/or the at least one third image is performed by contour finding code according to edge detection.

26. The method of claim 1, wherein the at least one first image and the at least one second image are frames extracted from a video captured by a video camera.

27. A system for automatically landing a drone on a landing pad having thereon a plurality of guiding-elements arranged in a pattern relative to a central region of the landing pad, comprising:

a non-transitory memory having stored thereon a code for execution by at least one hardware processor of a computing device in communication with a drone navigation system of a drone, the code comprising;

code for receiving at least one first image captured by a camera of the drone directed towards the landing pad;

code for processing the at least one first image to compute a segmentation mask according to an estimate of a location of the landing pad;

code for receiving at least one second image captured by the camera;

code for processing the at least one second image according to the segmentation mask to compute a segmented region and extracting from the segmented region at least one of the plurality of guiding-elements, determining a vector for each of the extracted at least one of the plurality of guiding-elements, and aggregating the vectors to compute an estimated location of the central region of the landing pad; and

navigating and landing the drone on the landing pad according to the estimated location of the central region of the landing pad.