Backup navigation system for unmanned aerial vehicles

• US 10,387,727 B2
• Filed: 09/13/2017
• Issued: 08/20/2019
• Est. Priority Date: 09/13/2017
• Status: Active Grant

• Current Assignee: Wing Aviation LLC (Alphabet Inc.)
• Original Assignee: Wing Aviation LLC (Alphabet Inc.)
• Inventors: Abeywardena, Dinuka, Jourdan, Damien
• Primary Examiner(s): Grant, II, Jerome
• Application Number:
  US15/703,948
• Publication Number:
  US 20190080142A1
• Time in Patent Office: 706 Days
• Field of Search: 382181
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/25 Fixed-wing aircraft VTOL aircraft B64U10/20
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/10 Wings
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G01C 21/32 Structuring or formatting of map data
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G05D 1/106 Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
• CPC Class Codes: G05D 1/606 Compensating for or utilising external environmental conditions, e.g. wind or water currents station keeping G05D1/611
• CPC Class Codes: G06V 20/17 taken from planes or by drones

Abstract

Described is a method that involves operating an unmanned aerial vehicle (UAV) to begin a flight, where the UAV relies on a navigation system to navigate to a destination. During the flight, the method involves operating a camera to capture images of the UAV'"'"'s environment, and analyzing the images to detect features in the environment. The method also involves establishing a correlation between features detected in different images, and using location information from the navigation system to localize a feature detected in different images. Further, the method involves generating a flight log that includes the localized feature. Also, the method involves detecting a failure involving the navigation system, and responsively operating the camera to capture a post-failure image. The method also involves identifying one or more features in the post-failure image, and determining a location of the UAV based on a relationship between an identified feature and a localized feature.

Claims

1. A method comprising:

operating an unmanned aerial vehicle (UAV) to begin a flight along a first flight path to a destination, wherein the UAV uses a primary navigation system to navigate to the destination;

operating an image capture device coupled to the UAV and having a field-of-view (FOV) of an environment of the UAV to capture a first set of images as the UAV travels along the first flight path;

analyzing the images to localize visual features within the environment;

generating a local flight log for the first flight path, wherein the local flight log comprises location information and image data for the detected features;

detecting a failure event involving the primary navigation system;

responsive to detecting the failure event, capturing at least one post-failure image using the image capture device;

identifying, in the at least one post-failure image, one or more visual features within the environment; and

detecting a match between a first of the identified visual features and a first of the localized visual features, and responsively;

determining, based at least in part on (i) location information for the first localized visual feature indicated by the local flight log, and (ii) an orientation of the UAV corresponding to the capture of the post-failure image, a current location of the UAV.

2. The method of claim 1, wherein the image capture device is part of a backup navigation system of the UAV, and wherein the backup navigation system replaces navigational functionality of the primary navigation system upon detecting the failure event.

3. The method of claim 1, wherein the primary navigation system is reliant upon a global positioning system (GPS).

4. The method of claim 1, further comprising:

responsive to detecting the failure event, determining to fly the UAV to a safe zone; and

based on the current location of the UAV, determining at least a portion of a second flight path to the safe zone.

5. The method of claim 4, wherein the first flight path and the second flight path are the same flight path.

6. The method of claim 4, wherein the UAV is deployed from a home base to the destination, and wherein the safe zone is the home base of the UAV.

7. The method of claim 4, wherein the second flight path has a substantially similar altitude to the first flight path such that the image capture device during the second flight path has a substantially similar FOV of the environment to the FOV of the environment that the image capture device has during the first flight path.

8. The method of claim 1, wherein operating the image capture device to capture the first set of images along the flight path comprises:

using position data from the primary navigation system to associate each image with a respective geolocation.

9. The method of claim 1, wherein analyzing the images to localize visual features within the environment comprises:

analyzing the first set of images to identify visual features in the images;

comparing a first visual feature identified in a first image to visual features identified in other images of the first set of images;

detecting a match between the first visual feature and a corresponding visual feature of a second image; and

determining a location of the first visual feature.

10. The method of claim 9, wherein determining the location of the first visual feature comprises:

executing an algorithm using a first geolocation of the first image and a second geolocation of the second image to determine the location of the first visual feature.

11. The method of claim 10, wherein the algorithm is a triangulation algorithm.

12. The method of claim 1, wherein operating the UAV to begin a flight further comprises:

determining a time of the flight and environmental conditions along the first flight path;

selecting a map associated with at least one of the time of the flight and the environmental conditions, wherein the map is representative of at least an area along the first flight path; and

loading the map to a memory of the UAV before the UAV begins the flight.

13. The method of claim 1, wherein determining the current location of the UAV further comprises:

matching the first identified visual feature and a second identified visual feature to the first localized visual feature and a second localized visual feature;

based on coordinates of the first and second localized visual features, determining coordinates of the first and second visual features; and

using the coordinates of the first and second visual features as a basis for determining the current location of the UAV.

14. An unmanned aerial vehicle (UAV) comprising:

a primary navigation system that provides the UAV with navigational functionality;

a backup navigation system configured to replace the navigational functionality of the primary navigation system upon detecting a failure event involving the primary navigation system;

wherein the backup navigation system comprises an image capture device configured to capture images of an environment of the UAV and a controller,

wherein the controller is configured to;

operate the UAV to begin a flight along a first flight path to a destination;

operate the backup navigation system in a feature localization mode in which the controller;

(i) causes the image capture device to capture a first set of images along the first flight path, and (ii) analyzes the images to localize a set of visual features within the environment, wherein each localized visual feature is associated with a respective geolocation, and wherein the set of localized features is stored in a memory location;

detect a failure event involving the primary navigation system;

responsive to detecting the failure event, operate the backup navigation system in a UAV localization mode in which the controller;

(i) causes the image capture device to capture at least one post-failure image of the environment, (ii) identifies in the at least one post-failure image visual features within the environment, and (iii) compares the identified visual features to the set of localized visual features stored in the memory location; and

detect a match between a first of the identified visual features and a first of the localized visual features, and responsively determine, based at least in part on (i) location information for the first localized visual feature indicated by the local flight log, and (ii) an orientation of the UAV corresponding to the capture of the post-failure image, a current location of the UAV.

15. The UAV of claim 14, wherein the controller is further configured to:

responsive to detecting the failure event, determine to fly the UAV to a safe zone; and

based on the current location of the UAV, determine at least a portion of a second flight path to the safe zone.

16. The UAV of claim 14, wherein the primary navigation system comprises a global positioning system (GPS) device configured to receive a satellite signal, and wherein the failure event is at least one of:

(i) a malfunction of the GPS device, and (ii) a loss of the satellite signal.