Virtual camera interface and other user interaction paradigms for a flying digital assistant

• US 9,798,322 B2
• Filed: 04/16/2015
• Issued: 10/24/2017
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Primary Examiner(s): Elahi, Shan
• Application Number:
  US14/689,008
• Publication Number:
  US 20160327950A1
• Time in Patent Office: 922 Days
• Field of Search: 348114
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.

Claims

1. A computer-implemented method for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the method comprising:

detecting, by a processor, a first position and orientation of a PMD relative to a first point of reference in a physical environment;

wherein, the first position and orientation of the PMD is detected based in part on sensor data gathered from sensors associated with the PMD;

transforming, by the processor, the detected first position and orientation of the PMD relative to first point of reference into a second position and orientation relative to a second point of reference in the physical environment;

wherein, the transforming includes applying a scale factor to the first position;

generating, by the processor, control commands configured to cause the FDA to autonomously fly to the second position and orientation relative to the second point of reference in the physical environment; and

causing display, by the processor, via a display of the PMD, of a representation of a field of view of the physical environment from the FDA as the FDA autonomously flies to the second position and orientation relative to the second point of reference;

wherein the representation of the field of view is generated based in part on sensor data gathered by sensors associated with the FDA in autonomous flight over the physical environment.

2. The method of claim 1, further comprising:

tracking a change in the first position and orientation of the PMD over time and updating the second position and orientation in real time or near real time, based on the tracked change in the first position and orientation.

3. The method of claim 1, further comprising:

generating control commands configured to adjust image capture by an image capture device associated with the FDA to correspond with the second orientation relative to the second point of reference.

4. The method of claim of claim 3, wherein adjusting image capture includes adjusting an orientation of the image capture device and/or applying digital image processing to images captured by the image capture device.

5. The method of claim 1, wherein the representation of the field of view from the second position and orientation is a live video feed from an image capture device associated with the FDA.

6. The method of claim 1, further comprising:

generating a virtual space including a three-dimensional model of the physical environment within a threshold proximity of the second point of reference;

wherein the three-dimensional model is generated based in part on sensor data gathered by the sensors associated with the FDA.

7. The method of claim 6, wherein the representation of the field of view from the second position and orientation includes a rendering of the virtual space from a field of view of a virtual camera associated with the virtual space, wherein the field of view of the virtual camera with relation to the virtual space corresponds with the field of view of the physical environment from the image capture device of the FDA at the second position and orientation.

8. The method of claim 1, further comprising:

adjusting the scale factor in response to an input received via the PMD.

9. The method of claim 1, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, or inertial measurement unit.

10. A system for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the system comprising:

one or more processors;

one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

detect a first position and orientation of a PMD relative to a first point of reference in the physical environment;

wherein, the detected first position and orientation of a PMD is based in part on sensor data gathered from sensors associated with the PMD;

transform the detected first position and orientation of the PMD relative to first point of reference into a second position and orientation relative to a second point of reference in the physical environment;

wherein, the transforming includes applying a scale factor to the first position;

generate control commands configured to cause the FDA to autonomously fly to the second position and orientation relative to the second point of reference in the physical environment; and

cause display, via a display of the PMD, of a field of view of the physical environment from the FDA as the FDA autonomously flies to the second position and orientation relative to the second point of reference;

wherein the representation of the field of view is generated based in part on sensor data gathered by sensors associated with the FDA in autonomous flight over the physical environment.

11. The system of claim 10, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

track a change in the first position and orientation of the PMD over time and update the second position and orientation in real time or near real time, based on the tracked change in the first position and orientation.

12. The system of claim 10, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate control commands configured to adjust image capture by an image capture device associated with the FDA to correspond with the second orientation relative to the second point of reference.

13. The system of claim of claim 12, wherein adjusting image capture includes adjusting an orientation of the image capture device and/or applying digital image processing to images captured by the image capture device.

14. The system of claim 10, wherein the representation of the field of view from the second position and orientation is a live video feed from an image capture device associated with the FDA.

15. The system of claim 10, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate a virtual space including a three-dimensional model of the physical environment within a threshold proximity of the second point of reference;

wherein the three-dimensional model is generated based in part on sensor data gathered by the sensors associated with the FDA.

16. The system of claim 15, wherein the representation of the field of view from the second position and orientation includes a rendering of the virtual space from a field of view of a virtual camera associated with the virtual space, wherein the field of view of the virtual camera with relation to the virtual space corresponds with the field of view of the physical environment from the second position and orientation.

17. The system of claim 10, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

adjust the scale factor in response to an input received via the PMD.

18. The system of claim 10, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, or inertial measurement unit.

19. The system of claim 10, wherein the display associated with the PMD is a virtual reality headset display.

20. A computer-implemented method for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the method comprising:

causing display, by a processor, via a touch display of the PMD, of a visual representation of a field of view of the physical environment from the FDA in autonomous flight over the physical environment;

wherein the visual representation of the field of view is generated based in part on sensor data gathered by sensors associated with the FDA;

receiving, by the processor, via the touch display of the PMD, a touch gesture, the touch gesture indicative of a selection, by a user of the PMD, of a point or area in the visual representation of the field of view of the physical environment;

identifying, by the processor, a point of reference in the physical environment corresponding to the selected point or area in the visual representation;

generating, by the processor, a flight path relative to the identified point of reference based on the touch gesture;

generating, by the processor, control commands configured to cause the FDA to autonomously fly along the generated flight path; and

dynamically updating, by the processor, display of the visual representation of the field of view of the physical environment from the FDA as the FDA autonomously flies along the generated flight path.

21. The method of claim 20, further comprising:

defining a reference plane relative to the point of reference;

wherein the reference plane corresponds to one or more identifiable physical surfaces in the physical environment; and

wherein the generated flight path is a two dimensional path at a constant or substantially constant height above the reference plane.

22. The method of claim 21, wherein the touch gesture is a single touch, wherein the generated flight path is a substantially straight line along the reference plane towards the point of reference, and wherein the control commands are further configured to cause the FDA to focus image capture on the point of reference.

23. The method of claim 22, wherein focusing image capture includes motion by the FDA along the two dimensional path to within a threshold proximity of the point of reference and adjustment by an image capture device associated with the FDA to center the point of reference in a field of view of the image capture device.

24. The method of claim 21, wherein the touch gesture is drawn line, and wherein the two dimensional path along the reference plane is a projection of the drawn line.

25. The method of claim 21, wherein the touch gesture is indicative of a selection of a preset paths, and wherein the two dimensional path along the reference plane is based on the selected preset paths.

26. The method of claim 20, wherein identification of the point of reference in the physical environment corresponding to the selected point or area is based in part on sensor data gathered from sensors associated with the FDA.

27. The method of claim 20, further comprising:

generating updated control commands in real time or near real time in response to a detected change in position and orientation of the FDA;

wherein the detected change in position and orientation of the FDA is based in part on sensor data gathered by sensors associated with the FDA.

28. A system for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the system comprising:

one or more processors; and

one or more memory units, the one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

cause display, via a touch display of the PMD, of a visual representation of a field of view of the physical environment from the FDA in autonomous flight over the physical environment;

wherein the visual representation of the field of view is generated based in part on sensor data gathered by sensors associated with the FDA;

receive, via the touch display of the PMD, a touch gesture, the touch gesture indicative of a selection, by a use of the FDA, of a point or area in the visual representation of the field of view of the physical environment;

identify a point of reference in the physical environment corresponding to the selected point or area in the visual representation;

generate a flight path relative to the identified point of reference based on the touch gesture;

generate control commands configured to cause the FDA to autonomously fly along the generated flight path; and

dynamically update display of the visual representation of the field of view of the physical environment from the FDA as the FDA autonomously flies along the generated flight path.

29. The system of claim 28, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

define a reference plane relative to the point of reference;

wherein the reference plane corresponds to one or more identifiable physical surfaces in the physical environment; and

wherein the generated flight path is a two dimensional path at a constant or substantially constant height above the reference plane.

30. A computer-implemented method for controlling image capture by a flying digital assistant (FDA) in autonomous flight over a physical environment using a portable multifunction device (PMD), the method comprising:

receiving, by a processor, images captured by an image capture device associated with the FDA in autonomous flight over the physical environment, the images characterized by a line of sight of the image capture device;

tracking, by the processor, an orientation of the PMD based on data received from sensors associated with the PMD;

adjusting, by the processor, the line of sight of the image capture device mounted to the FDA to correspond to the tracked orientation of the PMD; and

causing display, by the processor, via a display device of the PMD, of images captured by the image capture device as the line of sight of the image capture device is adjusted.

31. The method of claim 30, wherein the display device is integrated with the PMD.

32. The method of claim 30, wherein the field of view of the image capture device is adjusted using a hybrid mechanical digital gimbal mechanism.