USER INTERACTION PARADIGMS FOR A FLYING DIGITAL ASSISTANT

• US 20180095459A1
• Filed: 10/23/2017
• Published: 04/05/2018
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Granted Patent: US 10,466,695 B2
• Time in Patent Office: Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.

Claims

1. An unmanned aerial vehicle (UAV) configured to capture aerial video of multiple subjects in a physical environment, the UAV comprising:

an image capture device;

a wireless network interface configured for wireless communication with a plurality of mobile devices; and

a visual navigation system configured to;

receive, via the wireless network interface, an image capture request from a first mobile device of the plurality of mobile devices;

determine, in response to the image capture request, a position of the first mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the position of the first mobile device.

2. The UAV of claim 1, wherein the visual navigation system is further configured to:

receive, via the wireless network interface, an image capture request from a second mobile device of the plurality of mobile devices;

determine, in response to the image capture request, a position of the second mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the position of the second mobile device.

3. The UAV of claim 1, wherein the visual navigation system is further configured to:

receive, via the wireless network interface, an image capture request from a director device, the image capture request designating a second mobile device of the plurality of mobile devices;

determine, in response to the image capture request, a position of the second mobile device relative to the UAV based at least in part on images of the physical environment received via the image captured device; and

generate control commands configured to cause the UAV to maneuver such that the image capture device tracks the position of the second mobile device.

4. The UAV of claim 1, wherein the visual navigation system is further configured to:

authenticate the image capture request from the first mobile device before generating control commands configured to cause the UAV to maneuver.

5. The UAV of claim 4, wherein authenticating the image capture request includes confirming a valid payment associated with the image capture request.

6. The UAV of claim 5, wherein the valid payment is any of a one-time fee or a subscription.

7. The UAV of claim 1, wherein each of the plurality of mobile devices are associated with a user attending an event at a physical venue.

8. The UAV of claim 7, wherein the visual navigation system is further configured to:

generate control commands configured to cause the UAV to hover over the physical venue until an image captured request is received from one of the plurality of mobile devices.

9. The UAV of claim 1, wherein the visual navigation system is further configured to:

generate control commands configured to cause the UAV to maneuver such that the image capture device is no longer tracking the position of the first mobile device after a period of time has elapsed from when the image capture request was received from the first mobile device.

10. The UAV of claim 1, wherein the wireless network interface is further configured to transmit a live video stream, captured by the image capture device, to the first mobile device as the image capture device tracks the position of the first mobile device.

11. The UAV of claim 10, wherein the live video stream is displayed at a display of the first mobile device.

12. The UAV of claim 1, wherein the image capture device is coupled to the UAV by an image capture adjustment system configured to adjust an orientation of image capture device relative to the UAV.

13. The UAV of claim 12, wherein the image capture adjustment system includes a hybrid mechanical-digital gimbal system.

14. The UAV of claim 13, wherein the hybrid mechanical-digital gimbal system includes:

a mechanical gimbal configured to rotate the image capture device about a first axis relative to the UAV; and

an image processing system configured to rotate images captured by the image capture device about a second axis relative to the UAV.

15. A method for capturing aerial video of multiple subjects in a physical environment by an unmanned aerial vehicle (UAV), the UAV including an image capture device, the method comprising:

receiving, by the UAV, a first wireless communication including an image capture request from a first mobile device of a plurality of mobile devices;

determining, by the UAV, a position of the first mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

autonomously maneuvering, by the UAV, such that the image capture device tracks the position of the first mobile device.

16. The method of claim 15, further comprising:

receiving, by the UAV, a second wireless communication including an image capture request from a second mobile device of the plurality of mobile devices;

determining, by the UAV, a position of the second mobile device relative to the UAV based at least in part on images of the physical environment received via the image capture device; and

autonomously maneuvering, by the UAV, such that the image capture device tracks the position of the second mobile device.

17. The method of claim 15, further comprising:

transmitting, by the UAV, a live video stream captured by the image capture device, to the first mobile device.

18. The method of claim 15, further comprising:

authenticating, by the UAV, the image capture request before autonomously maneuvering such that the image capture device tracks the position of the first mobile device.

19. An unmanned aerial vehicle (UAV) comprising:

a propulsion system;

a power source coupled to the propulsion system;

an image capture device;

a data storage device configured to store images captured via the image capture device;

a processing unit; and

a memory unit having instructions stored thereon, which when executed by the processing unit, cause the UAV to;

generate control commands configured to cause the UAV to maneuver through a physical environment using the propulsion system;

capture, via the image capture device, video of the physical environment while the UAV maneuvers through the physical environment;

calculate a remaining flight time based on a monitored level of the power source;

calculate a remaining video recording time based on a monitored level of available storage space in the data storage device; and

generate control commands configured to cause the UAV to automatically land in response to detecting that either the calculated remaining flight time or the calculated remaining video recording time is below a specified threshold.

20. An autonomous unmanned aerial vehicle (UAV) comprising:

a propulsion system;

a power source coupled to the propulsion system; and

an autonomous navigation system configured to;

generate control commands configured to cause the UAV to autonomously maneuver through a physical environment using the propulsion system;

monitor a level of the power source; and

generate control commands configured to cause the UAV to automatically land in response to detecting that the level of the power source is below a first threshold.

21. The autonomous UAV of claim 20, further comprising:

an image capture device;

wherein the autonomous navigation system is further configured to generate control commands based at least in part on images of the physical environment received from the image capture device.

22. The autonomous UAV of claim 20, further comprising:

a wireless network interface configured for wireless communication with a mobile device;

wherein the autonomous navigation system is further configured to;

calculate a remaining flight time based on the monitored level of the power source and;

transmit an indication of the calculated remaining flight time to the mobile device via the wireless network interface.

23. The autonomous UAV of claim 20, further comprising:

an image capture device; and

a data storage device configured to store images captured via the image capture device;

wherein the autonomous navigation system is further configured to;

monitor a level of available storage space in the data storage device; and

generate control commands configured to cause the UAV to automatically land in response to detecting that the level of available storage space in the data storage device is below a second threshold.

24. The autonomous UAV of claim 20, further comprising:

a wireless network interface configured for wireless communication with a mobile device;

wherein the autonomous navigation system is further configured to;

calculate a remaining flight time based on the monitored level of the power source and;

calculate a remaining video recording time based on the monitored level of the available storage space in the data storage device; and

transmit an indication of the lesser of the calculated remaining flight time or calculated remaining video recording time to the mobile device via the wireless network interface.

25. A method comprising:

calculating a remaining flight time of an unmanned aerial vehicle (UAV) based on a monitored level of a power source coupled to a propulsion system of the aerial vehicle;

calculating a remaining video recording time based on a monitored level of available storage space in a data storage device coupled to an image capture device of the UAV; and

generating control commands configured to cause the UAV to automatically land in response to detecting that either the calculated remaining flight time or the calculated remaining video recording time is below a specified threshold.

26. The method of claim 25, wherein the control commands are generated by a visual navigation system onboard the UAV configured to autonomously maneuver the UAV based at least in part on images of the physical environment captured via the image capture device.

27. A method for capturing aerial video of a physical environment with synchronized distributed audio capture, the method comprising:

receiving, by a wireless network interface of a first mobile device, video of the physical environment captured by a camera of an unmanned aerial vehicle (UAV) in autonomous flight over the physical environment;

capturing, by a first microphone of the first mobile device, audio of the physical environment; and

synchronizing, by a processing unit of the first mobile device;

the video of the physical environment captured by the camera of the UAV; and

the audio of the physical environment captured by the first microphone of the first mobile device.

28. The method of claim 27, further comprising:

receiving, by the wireless network interface of a first mobile device, audio of the physical environment captured by a second microphone of a second mobile device; and

synchronizing, by the processing unit of the first mobile device;

the video of the physical environment captured by the camera of the UAV;

the audio of the physical environment captured by the first microphone of the first mobile device; and

the audio of the physical environment captured by the second microphone of the second mobile device.

29. The method of claim 27, further comprising:

receiving, by the wireless network interface of a first mobile device, audio of the physical environment captured by a plurality of microphones of a distributed network of mobile devices; and

synchronizing, by the processing unit of the first mobile device;

the video of the physical environment captured by the camera of the UAV;

the audio of the physical environment captured by the first microphone of the first mobile device; and

the audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices.

30. A method for capturing aerial video of a physical environment with synchronized distributed audio capture, the method comprising:

capturing, by a camera coupled to an unmanned aerial vehicle (UAV), video of the physical environment;

receiving, via a wireless network interface of the UAV, audio of the physical environment captured by a plurality of microphones of a distributed network of mobile devices; and

synchronizing, by a processing unit of the UAV;

the video of the physical environment captured by the camera of the UAV; and

the audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices.

31. The method of claim 30, further comprising:

capturing, by a microphone coupled to the UAV, audio of the physical environment; and

synchronizing, by a processing unit of the UAV;

the video of the physical environment captured by the camera of the UAV;

the audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices; and

the audio of the physical environment captured by the microphone coupled to the UAV.

32. An unmanned aerial vehicle (UAV) configured to capture aerial video of a physical environment with synchronized distributed audio capture, the UAV comprising:

a camera;

a wireless network interface configured for wireless communication with a plurality of mobile devices;

a processing unit; and

a memory having instructions stored thereon, which when executed by the processing unit, cause the UAV to;

capture, via the camera, video of the physical environment;

receive, via the wireless network interface, audio of the physical environment captured by a plurality of microphones of a distributed network of mobile devices; and

synchronize the video of the physical environment captured by the camera of the UAV and the audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices.

33. The UAV of claim 32, further comprising:

a microphone;

wherein the memory unit has further instructions stored thereon, which when executed by the processing unit, cause the UAV to further;

capture, via the microphone, audio for the physical environment; and

synchronize the video of the physical environment captured by the camera of the UAV, the audio of the physical environment captured by the microphone of the UAV, and the audio of the physical environment captured by the plurality of microphones of the distributed network of mobile devices.

34. A method comprising:

receiving sensor data gathered by sensors onboard an unmanned aerial vehicle (UAV) in flight through a physical environment;

generating a three dimensional (3D) model of the physical environment based at least in part on the sensor data received from the UAV;

determining a position and orientation of a display device relative to the physical environment;

placing a virtual camera in the 3D model of the physical environment at a position and orientation corresponding to the determined position and orientation of the display device;

generating a visual representation of at least a portion of the 3D model from a field of view of the virtual camera; and

causing display, via the display device, of the generated visual representation.

35. The method of claim 34, wherein the display device is any of a virtual reality display device or augmented reality display device.

36. The method of claim 34, wherein the sensors onboard the UAV include any one or more of:

an optical sensor, a motion sensor, or a proximity sensor.

37. The method of claim 34, wherein the 3D model of the physical environment is generated in real time or near real time as the UAV is in flight through the physical environment.

38. The method of claim 34, wherein the generated visual representation of the 3D model includes a visual representation of a physical object in the physical environment that is not in a line of sight with the display device.

39. The method of claim 34, further comprising:

tracking a motion of a physical object through the physical environment based on sensor data gathered by sensors onboard an unmanned aerial vehicle (UAV); and

updating the generated 3D model of the physical environment based on the tracked motion of the physical object through the physical environment.

40. The method of claim 34, wherein the generated visual representation includes a graphical user interaction element, the method further comprising:

detecting a user interaction with the graphical user interaction element; and

generating control commands configured to cause the UAV to maneuver based on the detected user interaction.

41. The method of claim 34, further comprising:

detecting a change in the position and orientation of the display device relative to the physical environment; and

generating control commands configured to cause the UAV to maneuver based on the detected change in position and orientation of the display device.

42. A method for implementing augmented reality (AR) based on sensor data collected by an unmanned aerial vehicle (UAV), the method comprising:

receiving sensor data gathered by sensors onboard the UAV in flight through a physical environment;

determining a position and orientation of an AR display device relative to the physical environment;

generating a graphical element based at least in part on the sensor data collected by the UAV and the determined position and orientation of the AR display device; and

causing display, by the AR display device, of the generated graphical element.

43. The method of claim 42, wherein the generated graphical element is displayed by the AR display device so as to partially overlay a view of the physical environment.

44. The method of claim 43, wherein the view of the physical environment is captured by a camera of the AR display device.

45. The method of claim 42, wherein the AR display device is a mobile phone.

46. The method of claim 42, wherein generating the graphical element includes:

generating a three dimensional (3D) model of the physical environment based at least in part on the sensor data received from the UAV;

placing a virtual camera in the 3D model of the physical environment at a position and orientation corresponding to the determined position and orientation of the AR display device;

wherein the generated graphical element includes a visual representation of at least a portion of the 3D model from a field of view of the virtual camera.

47. The method of claim 46, wherein the 3D model of the physical environment is generated in real time or near real time as the UAV is in flight through the physical environment.

48. The method of claim 46, wherein the generated visual representation of the 3D model includes a visual representation of a physical object in the physical environment that is not in a line of sight with the AR display device.