Supervisory control of an unmanned aerial vehicle

• US 9,817,396 B1
• Filed: 12/31/2014
• Issued: 11/14/2017
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Wing Aviation LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Takayama, Leila, Alexander, Brandon, Graves, Roger William, Sadowski, Justin, Bachrach, Abraham
• Primary Examiner(s): Zanelli, Michael J
• Application Number:
  US14/587,091
• Time in Patent Office: 1,049 Days
• CPC Class Codes: B64C 27/00 Rotorcraft; Rotors peculiar thereto
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2101/67 the UAVs comprising tethers for lowering the goods
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0202 specially adapted to aircraft
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0013 with a ground station
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/025 Navigation or guidance aids radar or analogous systems specially adapted for landing purposes G01S13/913

Abstract

An unmanned aerial vehicle (UAV) is disclosed that may allow for supervisory control interaction by a remote operator to assist with navigation to a target location. The UAV may navigate to a target area and capture and send an image of the target area to the remote operator. The remote operator can then provide a user input that indicates a target location within the target area. Upon receiving an indication of the target area, the UAV can then autonomously navigate to the target location. In some examples, after reaching the target location, the UAV may initiate delivery of a payload at the target location using a retractable delivery system while the UAV hovers above.

Claims

1. An unmanned aerial vehicle (UAV) comprising:

an image capture system;

a communication system configured for wireless communications; and

a control system configured to;

use output from a navigation module to autonomously navigate the UAV in a forward-flight mode to a target area;

in response to determining that the UAV is located in the target area;

(i) operate the UAV in a hover mode, (ii) operate the image capture system to capture image data of at least a portion of the target area, and (iii) operate the communication system to send a first data communication based on the captured image data, wherein the first data communication is directed to a remote terminal;

receive, in response to the first data communication, a second data communication comprising an indication of a target location within the target area;

use a first image coordinate associated with the received indication of the target location, a second image coordinate associated with a point of view of the image capture system, and a current elevation of the UAV as a basis to determine a translational offset to the target location relative to a current location of the UAV;

determine flight-control commands to navigate the UAV from a current location within the target area to the target location within the target area, wherein the flight-control commands include one or more commands to cause the UAV to traverse the determined translational offset in hover mode; and

cause the UAV to navigate, in hover mode, to the target location in accordance with the determined flight-control commands.

2. The UAV of claim 1, wherein the first and second image coordinates are each associated with pixel coordinates within the image data of the target area.

3. The UAV of claim 1, wherein the control system determining the translational offset comprises the control system:

(i) determining an orientation of the translational offset based on a position of the first image coordinate with respect to the second image coordinate, and (ii) determining a magnitude of the translational offset based on a displacement between the first and second image coordinates and the current elevation.

4. The UAV of claim 1, wherein the image capture system is a downward-facing image capture system mounted on the UAV via a gimbal mount.

5. The UAV of claim 1, further comprising a retractable delivery system comprising:

(i) a delivery device configured to selectively secure a payload and release the payload, (ii) a tether coupled to the delivery device, and (iii) a retraction system configured to apply force on the tether while the delivery device is suspended from the UAV by the tether so as to alternately lower the delivery device from the UAV and retract the delivery device back to the UAV, and

wherein the control system is further configured to, in response to causing the UAV to fly to the target location, use the retractable delivery system to lower the payload to ground level and release the payload at ground level.

6. The UAV of claim 5, wherein the control system is further configured to, prior to using the retractable delivery system to lower and release the payload at ground level:

(i) send, to the remote terminal, a third data communication comprising an indication that the UAV is hovering at the target location, and (ii) receive, from the remote terminal, a fourth data communication comprising an authorization to initiate delivery.

7. The UAV of claim 5, wherein the control system is further configured to:

receive, from the remote terminal, a communication comprising authorization to retract the delivery device to the UAV; and

in response to receiving the authorization to retract, use the retractable delivery system to initiate retraction of the delivery device to the UAV.

8. The UAV of claim 5, wherein the control system is further configured to:

receive, from the remote terminal, a third data communication comprising an indication of a position adjustment for the UAV so as to account for a wind-induced movement of the delivery device while the delivery device is suspended from the UAV by the tether; and

cause the UAV to navigate, in hover mode, in accordance with the indicated position adjustment.

9. The UAV of claim 1, wherein the second data communication was generated by the remote terminal, wherein the indication of the target location was generated by the remote terminal based on the first data communication, and wherein the target location is indicated within the image data of the target area.

10. A method comprising:

causing an unmanned aerial vehicle (UAV) to navigate, in a forward-flight mode, to a target area;

in response to determining that the UAV is located in the target area;

(i) operating the UAV in a hover mode, (ii) operating an image capture system to capture image data of at least a portion of the target area, and (iii) sending a first data communication based on the captured image data, wherein the first data communication is directed to a remote terminal;

in response to the first data communication, receiving a second data communication comprising an indication of a target location within the target area;

using a first image coordinate associated with the received indication of the target location, a second image coordinate associated with a point of view of the image capture system, and a current elevation of the UAV as a basis for determining a translational offset to the target location relative to a current location of the UAV;

determining flight-control commands to navigate the UAV from a current location within the target area to the target location within the target area, wherein the flight-control commands include one or more commands to cause the UAV to traverse the determined translational offset in hover mode; and

causing the UAV to navigate, in hover mode, to the target location in accordance with the determined flight-control commands.

11. The method of claim 10, further comprising, in response to causing the UAV to navigate to the target location, using a retractable delivery system to lower a payload to ground level and release the payload at ground level, wherein the retractable delivery system comprises:

(i) a delivery device configured to selectively secure the payload and release the payload, (ii) a tether coupled to the delivery device, and (iii) a retraction system configured to apply force on the tether while the delivery device is suspended from the UAV by the tether so as to alternately lower the delivery device from the UAV and retract the delivery device back to the UAV.

12. The method of claim 11, further comprising, prior to using the retractable delivery system to lower and release the payload at ground level:

(i) sending, to the remote terminal, a third data communication comprising an indication that the UAV is hovering at the target location, and (ii) receiving, from the remote terminal, a fourth data communication comprising an authorization to initiate delivery.

13. The method of claim 11, further comprising:

receiving, from the remote terminal, a third data communication comprising authorization to retract the delivery device to the UAV; and

in response to receiving the authorization to retract, using the retractable delivery system to initiate retraction of the delivery device to the UAV.

14. The method of claim 11, further comprising:

receiving, from the remote terminal, a third data communication comprising an indication of a position adjustment for the UAV so as to account for a wind-induced movement of the delivery device while the delivery device is suspended from the UAV by the tether; and

causing the UAV to navigate, in hover mode, in accordance with the indicated position adjustment.

15. The method of claim 10, wherein the second data communication was generated by the remote terminal, wherein the indication of the target location was generated by the remote terminal based on the first data communication, and wherein the target location is indicated within the image data of the target area.

16. A method comprising:

displaying image data received from an image capture system situated on an unmanned aerial vehicle (UAV) while the UAV is located in a target area, wherein the displayed image data comprises at least a portion of the target area;

while displaying the image data, prompting a user for an input indicative of a target location within the target area;

displaying an indication of a delivery location preference associated with the target area;

subsequent to displaying the indication of the delivery location preference, receiving a user input indicative of the target location; and

in response to receiving the user input, sending an indication of the target location to the UAV such that the UAV responsively navigates, in hover mode, from a current location within the target area to the target location within the target area.

17. The method of claim 16, further comprising:

determining a translational offset to the target location relative to the current location, wherein the translational offset is based on a first image coordinate associated with the received user input indicative of the target location, a second image coordinate associated with a point of view of the image capture system, and a current elevation of the UAV, and wherein sending the indication of the target location to the UAV comprises sending an indication of the determined translational offset.