Control of an unmanned aerial vehicle through multi-touch interactive visualization

• US 9,891,621 B2
• Filed: 05/10/2017
• Issued: 02/13/2018
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Primary Examiner(s): FREJD, RUSSELL WARREN
• Application Number:
  US15/591,883
• Publication Number:
  US 20170329324A1
• Time in Patent Office: 279 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a magic wand user interaction paradigm is described for intuitive control of an FDA using a PMD. In other embodiments, methods for scripting a shot are described.

Claims

1. A method of controlling an unmanned aerial vehicle (UAV) in flight over a physical environment based on multi-touch gesture inputs applied to an interactive visualization of the physical environment, the method comprising:

causing display, via a touch screen display device, of a view of the physical environment, the view centered on a point of reference in the physical environment, the view of the physical environment including;

video captured by an image captured device coupled to the UAV; and

a rendering of a three-dimensional (3D) model indicative of a first pose of a virtual camera relative to the point of reference;

receiving, via the touch screen display device, a multi-touch gesture indicative of a request to change the pose of the virtual camera;

dynamically updating, as the multi-touch gesture is received, the rendering of the 3D model to be indicative of a second pose of the virtual camera relative to the point of reference; and

generating, based on the received multi-touch gesture, control commands configured to cause the UAV to autonomously maneuver such that video captured by the image captured device coupled to the UAV corresponds with the second pose of the virtual camera;

wherein the displayed view of the physical environment remains centered on the point of reference as the UAV autonomously maneuvers.

2. A method comprising:

causing display, via a touch screen display device, of a view of a physical environment from a first position relative to a point of reference, the view based at least in part on data gathered by sensors onboard an unmanned aerial vehicle (UAV) in flight over the physical environment at the first position;

receiving, via the touch screen display device, a touch gesture input indicative of a request to adjust the displayed view of the physical environment;

generating control commands configured to cause the UAV to autonomously fly to a second position relative to the point of reference based on the received touch gesture input; and

dynamically updating display of the view of the physical environment as the UAV autonomously flies from the first position to the second position.

3. The method of claim 2, wherein the displayed view is a video feed from an image capture device onboard the UAV.

4. The method of claim 3, further comprising:

generating control commands configured to adjust image capture by the image capture device based on the received touch gesture;

wherein adjusting image capture includes any of;

adjusting an orientation of the image capture device;

adjusting a zoom setting of the image capture device;

or

applying digital image processing to the video feed from the image capture device.

5. The method of claim 2, wherein the displayed view is a rendering of a three-dimensional (3D) model of the physical environment, the 3D model generated based at least in part on the data gathered by the sensors onboard the UAV.

6. The method of claim 5, wherein the 3D model is generated in real time or near real time as the UAV is in flight through the physical environment.

7. The method of claim 2, wherein the touch screen display device is in wireless communication with the UAV.

8. The method of claim 2, wherein the touch gesture is a multi-touch gesture.

9. The method of claim 2, wherein the touch gesture is a multi-touch pinch-to-zoom gesture, and wherein the autonomous flight of the UAV from the first position to the second position results in the displayed view of the physical environment increasing or decreasing in size while remaining centered on the point of reference.

10. The method of claim 2, wherein the touch gesture is a multi-touch scroll gesture, and wherein the autonomous flight of the UAV from the first position to the second position results in the displayed view of the physical environment moving in a direction of the multi-touch scroll gesture.

11. The method of claim 2, wherein the touch gesture is a multi-touch rotate gesture, and wherein the control commands configured to cause the UAV to autonomously fly to the second position relative to the point of reference include control commands configured to cause the UAV to fly along a circular path at a constant distance from the point of reference.

12. The method of claim 2, further comprising:

generating control commands configured to cause an image capture device onboard the UAV to adjust image capture so as to keep the point of reference centered in the view as the view changes from the first position to the second position.

13. The method of claim 2, wherein adjusting image capture includes adjusting an orientation of the image capture device and/or applying digital image processing to images captured by the image capture device.

14. The method of claim 2, wherein the image capture device includes a hybrid mechanical-digital gimbal system, and wherein adjusting image capture includes any of:

actuating a mechanical gimbal coupling the image capture device to the UAV to adjust an orientation of the image capture device;

or

applying digital image processing to a video feed from the image capture device to adjust an orientation of the view.

15. The method of claim 2, further comprising:

defining the point of reference based on the received touch gesture.

16. The method of claim 15, wherein defining the point of reference based on the received touch gesture includes:

detecting an interaction with a particular portion of the representation of the view of the physical environment displayed via the touch screen display device;

determining a physical object or surface in the physical environment corresponding to the particular portion of the representation; and

defining the point of reference relative to the physical object or surface.

17. The method of claim 2, wherein the point of reference is defined relative to a physical object or surface in the physical environment.

18. The method of claim 2, wherein the point of reference is defined based on a position of the touch screen display device in the physical environment, and wherein the position of the touch screen display device in the physical environment is determined based at least in part on sensors at the touch screen display device.

19. The method of claim 2, wherein the UAV is configured for autonomous flight using visual inertial odometry applied to images capture by an image capture device onboard the UAV.

20. The method of claim 2, wherein the displayed view of the physical environment includes a rendering of a three-dimensional (3D) model indicative of a first pose of a virtual camera relative to the point of reference, the method further comprising:

dynamically updating, as the touch gesture input is received, the rendering of the 3D model to be indicative of a second pose of the virtual camera relative to the point of reference;

wherein the control commands configured to cause the UAV to autonomously fly to the second position relative to the point of reference are such that the displayed view of the physical environment corresponds with the second pose of the virtual camera; and

wherein the displayed view of the physical environment remains centered on the point of reference as the UAV autonomously flies to the second position.

21. A system for controlling an image capture by an unmanned aerial vehicle (UAV) based on multitouch gestures, the system comprising:

a touch screen display;

one or more processors; and

one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

cause display, via the touch screen display, of a view of a physical environment from a first position relative to a point of reference, the view based at least in part on data gathered by sensors onboard the UAV in flight over the physical environment at the first position;

receive, via the touch screen display, a multi-touch touch gesture input indicative of a request to adjust the displayed view of the physical environment;

generate control commands configured to cause the UAV to autonomously fly to a second position relative to the point of reference based on the received multi-touch gesture input; and

dynamically update display of the view of the physical environment from the first position to the second position.

22. A non-transitory computer readable medium having instructions stored thereon which when executed by a computer system, cause the computer system to:

cause display, via a touch screen display device, of a view of a physical environment from a first position relative to a point of reference, the view based at least in part on data gathered by sensors onboard an unmanned aerial vehicle (UAV) in flight over the physical environment at the first position;

receive, via the touch screen display device, a multitouch touch gesture input indicative of a request to adjust the displayed view of the physical environment;

generate control commands configured to cause the UAV to autonomously fly to a second position relative to the point of reference based on the received multitouch touch gesture input; and

dynamically update display of the view of the physical environment from the first position to the second position.