User interaction paradigms for a flying digital assistant

• US 10,466,695 B2
• Filed: 10/23/2017
• Issued: 11/05/2019
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US15/791,230
• Publication Number:
  US 20180095459A1
• Time in Patent Office: 743 Days
• Field of Search: 348114
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.

Claims

1. An unmanned aerial vehicle (UAV) configured to capture aerial video of multiple subjects in a physical environment, the UAV comprising:

an image capture device;

a wireless network interface configured for wireless communication with a plurality of mobile devices; and

a computer processor configured to;

receive, via the wireless network interface, a first image capture request from a first mobile device of the plurality of mobile devices;

determine, in response to the image capture request, a first position of the first mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device;

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the first position of the first mobile device;

receive, via the wireless network interface, a second image capture request from a second mobile device of the plurality of mobile devices, the second mobile device different than the first mobile device;

determine, in response to the image capture request, a second position of the second mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the position of the second mobile device.

2. The UAV of claim 1, wherein the computer processor is further configured to:

receive, via the wireless network interface, a third image capture request designating a third mobile device of the plurality of mobile devices, the third image capture request received from a fourth mobile device of the plurality of mobile devices;

determine, in response to the third image capture request, a third position of the third mobile device relative to the UAV based at least in part on images of the physical environment received via the image captured device; and

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the third position of the third mobile device.

3. The UAV of claim 1, wherein the computer processor is further configured to:

authenticate the first image capture request from the first mobile device before generating control commands configured to cause the UAV to maneuver.

4. The UAV of claim 3, wherein authenticating the first image capture request includes confirming a valid payment associated with the first image capture request.

5. The UAV of claim 4, wherein the valid payment is any of a one-time fee or a subscription.

6. The UAV of claim 1, wherein each of the plurality of mobile devices are associated with a different one of a plurality of users attending an event at a physical venue.

7. The UAV of claim 6, wherein the computer processor is further configured to:

generate control commands configured to cause the UAV to hover over the physical venue until an image captured request is received from one of the plurality of mobile devices.

8. The UAV of claim 1, wherein the computer processor is further configured to:

generate control commands configured to cause the UAV to maneuver such that the image capture device is no longer tracking the first position of the first mobile device after a period of time has elapsed from when the first image capture request was received from the first mobile device.

9. The UAV of claim 1, wherein the wireless network interface is further configured to transmit a live video stream, captured by the image capture device, to the first mobile device as the image capture device tracks the position of the first mobile device.

10. The UAV of claim 9, wherein the live video stream is displayed at a display of the first mobile device.

11. The UAV of claim 1, wherein the image capture device is coupled to the UAV by a gimbal device configured to adjust an orientation of image capture device relative to the UAV.

12. The UAV of claim 11, wherein the gimbal device is part of a hybrid mechanical-digital gimbal system.

13. The UAV of claim 12, wherein the hybrid mechanical-digital gimbal system includes:

a mechanical gimbal configured to rotate the image capture device about a first axis relative to the UAV; and

an image processing system configured to rotate images captured by the image capture device about a second axis relative to the UAV.

14. A method for capturing aerial video of multiple subjects in a physical environment by an unmanned aerial vehicle (UAV), the UAV including an image capture device, the method comprising:

receiving, by the UAV, a first wireless communication including a first image capture request from a first mobile device of a plurality of mobile devices;

determining, by the UAV, in response to the first image capture request, a first position of the first mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device;

autonomously maneuvering, by the UAV, such that the image capture device tracks the first position of the first mobile device;

receiving, by the UAV, a second wireless communication including a second image capture request from a second mobile device of the plurality of mobile devices, the second mobile device different than the first mobile device;

determining, by the UAV, in response to the second image capture request, a second position of the second mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

autonomously maneuvering, by the UAV, such that the image capture device tracks the second position of the second mobile device.

15. The method of claim 14, further comprising:

transmitting, by the UAV, a live video stream captured by the image capture device, to the first mobile device.

16. The method of claim 14, further comprising:

authenticating, by the UAV, the first image capture request before autonomously maneuvering such that the image capture device tracks the first position of the first mobile device.

17. The method of claim 14, further comprising:

receiving, by the UAV, a third wireless communication including a third image capture request designating a third mobile device of the plurality of mobile devices, the third image capture request received from a fourth mobile device of the plurality of mobile devices;

determining, by the UAV, in response to the third image capture request, a third position of the third mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

autonomously maneuvering, by the UAV, such that the image capture device tracks the third position of the third mobile device.

18. The method of claim 14, further comprising:

autonomously maneuvering, by the UAV, such that the image capture device is no longer tracking the first position of the first mobile device after a period of time has elapsed from when the first image capture request was received from the first mobile device.

19. The method of claim 14, wherein each of the plurality of mobile devices are associated with a different one of a plurality of users attending an event at a physical venue.

20. The method of claim 19, further comprising:

autonomously hovering, by the UAV, over the physical venue until an image captured request is received from one of the plurality of mobile devices.