Unmanned aerial vehicle visual line of sight control

• US 9,588,516 B1
• Filed: 04/08/2016
• Issued: 03/07/2017
• Est. Priority Date: 02/08/2016
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Unmanned Innovation Inc (Delair SAS)
• Inventors: Michini, Bernard J., Steakley, Edward Dale, Gurel, Volkan
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Nolan, Peter D
• Application Number:
  US15/094,802
• Time in Patent Office: 333 Days
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/32 for cartography or topography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0033 by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle
• CPC Class Codes: G05D 1/0061 for transition from automatic pilot to manual pilot and vice versa
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/10 : Recognition assisted with metadata
• CPC Class Codes: G08G 5/006 : in accordance with predefined flight zones, e.g. to avoid prohibited zones
• CPC Class Codes: G08G 5/0069 : specially adapted for an unmanned aircraft
• CPC Class Codes: H04B 7/18506 : Communications with or from aircraft, i.e. aeronautical mobile service
• CPC Class Codes: H04W 4/021 : Services related to particular areas, e.g. point of interest [POI] services, venue services or geofences

Abstract

Methods, systems and apparatus, including computer programs encoded on computer storage media for unmanned aerial vehicle visual line of sight flight operations. A UAV computer system may be configured to ensure the UAV is operating in visual line of sight of one or more ground operators. The UAV may confirm that it has a visual line of sight with the one or more user devices, such as a ground control station, or the UAV may ensure that the UAV does not fly behind or below a structure such that the ground operator would not be able to visually spot the UAV. The UAV computer system may be configured in such a way that UAV operation will maintain the UAV in visual line of sight of a base location.

Claims

1. A method, comprising:

obtaining a base location of an unmanned aerial vehicle (UAV);

obtaining an in-flight location of the UAV;

determining, using a database of three-dimensional geographic features, a visual line of sight (VLOS) interruption between the base location and the in-flight location of the UAV; and

in response to the determining, instructing the UAV to perform a contingency action, wherein the method is performed by one or more computer processors.

2. The method of claim 1, wherein the base location is a launching location where the UAV took off, or a location of a user device controlling the UAV, and the in-flight location is a location of the UAV determined by a location subsystem of the UAV.

3. The method of claim 1, wherein the database of three-dimensional geographic features stores data on one or more natural or man-made structures.

4. The method of claim 1, wherein the database stores a list of geofences having at least three dimensions, and wherein determining that the VLOS interruption comprises determining that the UAV has entered a geofence that is behind a geographic feature that intersects a line between the base location and the in-flight location.

5. The method of claim 1, wherein the database stores a topographical map of the three-dimensional features, and wherein determining that the VLOS interruption comprises determining that, according to the map, at least one of the three-dimensional features intersects a line between the base location and the in-flight location.

6. The method of claim 5, wherein the topographical map indicates a respective height of each three-dimensional feature, and determining that at least one of the three-dimensional features intersects the line between the base location and the in-flight location comprises determining that an altitude of the UAV is below a height of the three-dimensional feature.

7. The method of claim 1, wherein determining that the VLOS between the base location and the in-flight location is interrupted comprises:

sending an optical signal from the UAV to one or more user devices; and

determining that the signal from the one or more user devices has been interrupted.

8. The method of claim 1, wherein the contingency action comprises navigating the UAV to a landing location or navigating the UAV to ascend to an altitude that is within the VLOS of the base location.

9. The method of claim 1, wherein the contingency action comprises navigating the UAV along a previous flight path until the VLOS between the base location and the in-flight location is no longer interrupted.

10. The method of claim 1, comprising, in response to determining that the VLOS between the base location and the in-flight location of the UAV has been interrupted, presenting an alert on a user device, the alert including at least one of a visual or an audio message.

11. A system, comprising:

one or more computer processors; and

a non-transitory storage device storing instructions operable to cause the one or more computer processors to perform operations comprising;

obtaining a base location of an unmanned aerial vehicle (UAV);

obtaining an in-flight location of the UAV;

determining, using a database of three-dimensional geographic features, that a visual line of sight (VLOS) between the base location and the in-flight location is or will be interrupted; and

in response to the determining, causing the UAV to perform a contingency action, the contingency action including providing a command to a controller to control at least one of a motor or actuator of the UAV to move a control surface of the UAV.

12. The system of claim 11, wherein the system is a cloud-based processing system, a system onboard the UAV or a system onboard a user device controlling the UAV.

13. The system of claim 12, wherein the base location is a launching location where the UAV took off or a location of a user device controlling the UAV, and the in-flight location is a location of the UAV determined by a location subsystem of the UAV.

14. The system of claim 12, wherein the database of three-dimensional geographic features stores data on one or more natural or man-made structures.

15. The system of claim 11, wherein the database stores a list of geofences having at least three dimensions, and wherein determining that the VLOS is interrupted comprises determining that the UAV has entered a geofence that corresponds to a geographic feature that intersects a line between the base location and the in-flight location.

16. The system of claim 11, wherein the database stores a topographical map of the three-dimensional features, and wherein determining that the VLOS is interrupted comprises determining that, according to the map, at least one of the three-dimensional features intersects a line between the base location and the in-flight location.

17. The system of claim 11, wherein determining that the VLOS between the base location and the in-flight location is interrupted comprises:

sending an optical signal from the UAV to one or more user devices; and

determining that the signal from the one or more user devices has been interrupted.

18. The system of claim 11, wherein the contingency action comprises navigating the UAV to a landing location or navigating the UAV to ascend to an altitude that is within the VLOS of the base location.

19. The system of claim 11, wherein the contingency action comprises navigating the UAV along a previous flight path until the VLOS between the base location and the in-flight location is no longer interrupted.

20. The system of claim 11, the operations comprising, in response to determining that the VLOS between the base location and the in-flight location of the UAV is or will be interrupted, presenting an alert on a user device, the alert including at least one of a visual or an audio message.

21. A non-transitory storage device storing instructions operable to cause one or more computer processors to perform operations comprising:

obtaining a base location of an unmanned aerial vehicle (UAV);

obtaining an in-flight location of the UAV;

predicting, using a database of three-dimensional geographic features, that a visual line of sight (VLOS) between the base location and the in-flight location will be interrupted; and

in response to the predicting, causing the UAV to perform a contingency action.

22. The non-transitory storage device of claim 21, wherein obtaining an in-flight location of the UAV comprises:

creating a point cloud of a structure; and

determining the in-flight location of the UAV using the point cloud.

23. The non-transitory storage device of claim 21, wherein predicting that the VLOS between the base location and the in-flight location will be interrupted comprises determining a signal strength of a wireless signal between the UAV and a controller is approaching a threshold value, and the contingency action comprises preventing the UAV from flying beyond a distance from the controller such that the signal strength is above the threshold value.

24. The non-transitory storage device of claim 21, the operations comprising submitting at least one of structural polygonal data indicating heights of structures or a topographical map from a controller to the UAV, wherein predicting that the VLOS between the base location and the in-flight location will be interrupted comprises determining that the UAV is approaching a space where a geographic feature intersects the VLOS according to the structural polygonal data or topographical map.

25. The non-transitory storage device of claim 21, the contingency action including providing a command to a controller to control at least one of a motor or actuator of the UAV to move a control surface of the UAV, wherein movement of the control surface causes the UAV to land at a landing location or to fly to a position that is within VLOS of a ground control station of the UAV.

26. The non-transitory storage device of claim 21, wherein predicting that the VLOS between the base location and the in-flight location will be interrupted comprises:

sending an optical signal from the UAV to one or more user devices; and

determining that the optical signal from the one or more user devices has been interrupted.

27. The non-transitory storage device of claim 21, wherein the contingency action comprises navigating the UAV to a landing location or navigating the UAV to ascend to an altitude that is within the VLOS of the base location.

28. The non-transitory storage device of claim 21, wherein the contingency action comprises navigating the UAV along a previous flight path until the VLOS of the base location and the in-flight location is no longer interrupted.

29. The non-transitory storage device of claim 21, the operations comprising, in response to predicting that the VLOS between the base location and the in-flight location of the UAV will be interrupted, presenting an alert on a user device, the alert including at least one of a visual or an audio message.