MAGIC WAND INTERFACE AND OTHER USER INTERACTION PARADIGMS FOR A FLYING DIGITAL ASSISTANT

• US 20180246507A1
• Filed: 02/12/2018
• Published: 08/30/2018
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Granted Patent: US 10,816,967 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a magic wand user interaction paradigm is described for intuitive control of an FDA using a PMD. In other embodiments, methods for scripting a shot are described.

Claims

1. A method for controlling an unmanned aerial vehicle (UAV), the method comprising:

tracking, by a processing system, a pose of a mobile device in the physical environment;

defining, by the processing system, a virtual line extending from the mobile device based on the tracked pose of the mobile device; and

generating, by the processing system, control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes.

2. The method of claim 1, wherein tracking the pose of the mobile device includes:

receiving, by the processing system, from the mobile device, via a wireless communication link, data indicative of the pose of the mobile device.

3. The method of claim 2, wherein the data indicative of the pose of the mobile device is based on sensor data gathered by sensors at the mobile device.

4. The method of claim 1, wherein tracking the pose of the mobile device includes:

receiving, by the processing system, images of the mobile device captured by an image capture device onboard the UAV; and

processing, by the processing system, the images of the mobile device to estimate the pose of mobile device at a plurality of time instances.

5. The method of claim 4, wherein the pose of the mobile device at the plurality of time instances is estimated using visual inertial odometry.

6. The method of claim 1, further comprising:

receiving, by the processing system, sensor data from sensors onboard the UAV while the UAV is in motion;

processing, by the processing system, the sensor data to detect an obstacle; and

generating, by the processing system, control commands configured to cause the UAV to autonomously maneuver to avoid the obstacle even if such a maneuver causes the UAV to deviate from the virtual line based on the tracked pose of the mobile device.

7. The method of claim 6, further comprising:

receiving, by the processing system, updated sensor data from the sensors onboard the UAV processing, by the processing system, the updated sensor data to determine that the UAV has cleared the detected obstacle; and

generating, by the processing system, control commands configured to cause the UAV to autonomously maneuver back to a position along the virtual line based on the tracked pose of the mobile device.

8. The method of claim 1, further comprising:

receiving, by the processing system, from the mobile device, via a wireless communication link, data indicative of a first user input at the mobile device;

wherein the control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes are generated in response the first user input.

9. The method of claim 8, further comprising:

receiving, by the processing system, from the mobile device, via a wireless communication link, data indicative of a second user input at the mobile device; and

generating, by the processing system, in response to the second user input, control commands configured to cause the UAV to maneuver so as to remain along the virtual line while increasing or decreasing a separation distance between the UAV and the mobile device.

10. The method of claim 8, wherein the first user input is received at the mobile device via a touch screen display of the mobile device.

11. The method of claim 1, wherein generating control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes includes:

estimating, by the processing system, a position of the UAV relative to the virtual line based on the tracked pose of the mobile device; and

generating, by the processing system, a planned trajectory based on the estimated position of the UAV relative to the virtual line, the planned trajectory configured to cause the UAV to maneuver from the estimated position to a target position along the virtual line;

wherein the control commands are generated based on the generated planned trajectory.

12. The method of claim 11, wherein the position of the UAV is estimated using visual inertial odometry based on images received from image capture devices onboard the UAV.

13. The method of claim 11, wherein the processing system includes communicatively coupled processing components onboard the UAV and/or the mobile device.

14. The method of claim 1, further comprising:

tracking, by the processing system, a point of reference in the physical environment; and

generating, by the processing system, control commands configured to adjust image capture by the UAV so as to keep the tracked point of reference centered in a field of view of an image capture device onboard the UAV.

15. The method of claim 14, wherein the tracked point of reference corresponds to any of a tracked position of the mobile device and/or a tracked position of a user of the mobile device.

16. The method of claim 14, wherein the point of reference in the physical environment is defined based on a user input received via the mobile device.

17. The method of claim 14, wherein the control commands configured to adjust image capture by the UAV include any of:

control commands configured to adjust an orientation of the UAV relative to the point of reference;

control commands configured to cause a gimbal mechanism onboard the UAV to adjust an orientation of the image capture device relative to the UAV;

or

control commands configured to transform digital images captured by the image capture device.

18. An unmanned aerial vehicle (UAV) configured for autonomous flight through a physical environment, the UAV comprising:

a first image capture device configured to capture images of the physical environment; and

a propulsion system configured to maneuver the UAV through the physical environment; and

a processing system; and

an autonomous navigation system configured to;

receive images from the first image capture device;

track a pose of a mobile device located in the physical environment by processing the images received from the first image capture device;

define a virtual line extending from the mobile device based on the tracked pose of the mobile device; and

generate control commands configured to cause the UAV to autonomously maneuver using the propulsion system so as to remain along the virtual line as the pose of the mobile device changes.

19. The UAV of claim 18, further comprising:

a wireless communication interface configured to establish wireless communication links between the UAV and other devices.

20. The UAV of claim 19, wherein the autonomous navigation system is configured to further:

receive, from the mobile device, via the wireless communication interface, sensor data gathered by sensors at the mobile device; and

track the pose of the mobile device by processing the sensor data received from the mobile device.

21. The UAV of claim 19, wherein the autonomous navigation system is configured to further:

receive from the mobile device, via the wireless communication interface, data indicative of user input at the mobile device; and

in response to the user input, generate control commands configured to cause the UAV to maneuver so as to remain along the virtual line while increasing or decreasing a separation distance between the UAV and the mobile device.

22. The UAV of claim 18, further comprising:

one or more motion sensors including any one or more of an accelerometer, a gyroscope, or an inertial measurement unit (IMU).

23. The UAV of claim 22, wherein the autonomous navigation system is configured to track the pose of the mobile device and/or a pose of the UAV by processing the images received from the first image capture device with sensor data gathered by the one or more sensors using a process of visual inertial odometry.

24. The UAV of claim 18, wherein the autonomous navigation system is configured to further:

detect an obstacle in the physical environment by processing the images received from the fire image capture device; and

generate control commands configured to cause the UAV to autonomously maneuver to avoid the obstacle even if such a maneuver causes the UAV to deviate from the virtual line based on the tracked pose of the mobile device.

24-1. The UAV of claim 18, wherein the autonomous navigation system is configured to generate control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes by:

estimating a position of the UAV relative to the virtual line based on the tracked pose of the mobile device; and

generating a planned trajectory based on the estimated position of the UAV relative to the virtual line, the planned trajectory configured to cause the UAV to maneuver from the estimated position to a target position along the virtual line;

wherein the control commands are generated based on the generated planned trajectory.

25. The UAV of claim 18, wherein the navigation system comprises:

a motion planner configured to;

generate a planned trajectory through the physical environment by processing images received from the first image capture device; and

a flight controller configured to;

receive the planned trajectory from the motion planner;

generate the control commands based on the planned trajectory; and

output the control commands to the propulsion system to cause the UAV to maneuver along the planned trajectory.

26. The UAV of claim 18, further comprising:

a second image capture device configured to capture images of the physical environment, the second image capture device coupled to the UAV by a gimbal mechanism.

27. The UAV of claim 26, wherein the autonomous navigation system is configured to further:

track a point of reference in the physical environment; and

generate control commands configured to adjust image capture by the UAV so as to keep the tracked point of reference centered in a field of view of the second image capture device.

28. The UAV of claim 27, wherein the tracked point of reference corresponds to any of a tracked position of the mobile device and/or a tracked position of a user of the mobile device.

29. The UAV of claim 27, wherein the control commands configured to adjust image capture by the UAV include any of:

control commands configured to adjust an orientation of the UAV relative to the point of reference;

control commands configured to cause the gimbal mechanism to adjust an orientation of the second image capture device relative to the UAV;

or

control commands configured to transform images captured by the second image capture device.

30. An navigation system for an unmanned aerial vehicle (UAV), the navigation system comprising:

a processing unit; and

a memory unit coupled to the processing unit, the memory unit having instructions stored thereon, which when executed by the processing unit cause the navigation system to;

track a pose of a mobile device in the physical environment;

define a virtual line extending from the mobile device based on the tracked pose of the mobile device; and

generate control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes.

31. The navigation system of claim 30, wherein tracking the pose of the mobile device includes

receiving images of the mobile device captured by an image capture device onboard the UAV;

and

processing the images to estimate the pose of mobile device at a plurality of time instances.

32. The navigation system of claim 30, wherein the memory unit has further instructions stored thereon, which when executed by the processing unit, cause the navigation system to further:

detect an obstacle in a path of the UAV; and

generate control commands configured to cause the UAV to autonomously maneuver to avoid the obstacle even if such a maneuver causes the UAV to deviate from the virtual line based on the tracked pose of the mobile device.

33. The navigation system of claim 30, wherein the instructions for generating control commands configured to cause the UAV to autonomously maneuver so as to remain along the virtual line as the pose of the mobile device changes include instructions, which when executed by the processing unit, cause the navigation system to:

estimate a position of the UAV relative to the virtual line based on the tracked pose of the mobile device; and

generate a planned trajectory based on the estimated position of the UAV relative to the virtual line, the planned trajectory configured to cause the UAV to maneuver from the estimated position to a target position along the virtual line;

wherein the control commands are generated based on the generated planned trajectory.

34. The navigation system of claim 30, wherein the memory unit has further instructions stored thereon, which when executed by the processing unit, cause the navigation system to further:

track a point of reference in the physical environment; and

generate control commands configured to adjust image capture by the UAV so as to keep the tracked point of reference centered in a field of view of an image capture device onboard the UAV.