VIRTUAL CAMERA INTERFACE AND OTHER USER INTERACTION PARADIGMS FOR A FLYING DIGITAL ASSISTANT

• US 20160327950A1
• Filed: 04/16/2015
• Published: 11/10/2016
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Granted Patent: US 9,798,322 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/4307 Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a user may control image capture from an FDA by adjusting the position and orientation of a PMD. In other embodiments, a user may input a touch gesture via a touch display of a PMD that corresponds with a flight path to be autonomously flown by the FDA.

Claims

1. A computer-implemented method for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the method comprising:

detecting a first position and orientation of a PMD relative to a first point of reference;

wherein, the detected first position and orientation of a PMD is based in part on sensor data gathered from sensors associated with the PMD;

transforming the detected first position and orientation of the PMD relative to first point of reference into a second position and orientation relative to a second point of reference;

wherein, the transforming includes applying a scale factor to the first position; and

generating a representation of a field of view of the physical environment from the second position and orientation relative to the second point of reference, the generated representation configured for presentation via a display device;

wherein the representation of the field of view is generated based in part on sensor data gathered by one or more sensors associated with the FDA in autonomous flight over the physical environment.

2. The method of claim 1, further comprising:

tracking a change in the first position and orientation over time and updating the second position and orientation in real time or near real time, based on the tracked change in the first position and orientation.

3. The method of claim 1, further comprising:

presenting the representation of the field of view of the physical environment via a display of associated with the PMD.

4. The method of claim 1, further comprising:

generating control commands configured to cause the FDA to fly to and maintain the second position relative to the second point of reference.

5. The method of claim 4, further comprising:

generating control commands configured to cause an FDA to adjust orientation and adjust image capture by an associated image capture device to correspond with the second orientation relative to the second point of reference.

6. The method of claim of claim 5, wherein adjusting image capture is accomplished by adjusting an orientation of the image capture device and/or applying digital image processing to images captured by the image capture device.

7. The method of claim 1, wherein the representation of the field of view from the second position and orientation is a live video feed from an image capture device associated with the FDA.

8. The method of claim 1, further comprising:

generating a virtual space including a three dimensional model of the physical environment within a threshold proximity of the second point of reference;

wherein the three-dimensional model is generated based in part on sensor data gathered by the one or more sensors associated with the FDA.

9. The method of claim 8, wherein the representation of the field of view from the second position and orientation includes one or more renderings of the virtual space from a field of view of a virtual camera associated with the virtual space, wherein the field of view of the virtual camera with relation to the virtual space corresponds with the field of view of the physical environment from the second position and orientation.

10. The method of claim 1, further comprising:

adjusting the scale factor in response to an input received via the PMD.

11. The method of claim 1, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

12. A system for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the system comprising:

one or more processors;

one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

detect a first position and orientation of a PMD relative to a first point of reference;

wherein, the detected first position and orientation of a PMD is based in part on sensor data gathered from sensors associated with the PMD;

transform the detected first position and orientation of the PMD relative to first point of reference into a second position and orientation relative to a second point of reference;

wherein, the transforming includes applying a scale factor to the first position; and

generate a representation of a field of view of the physical environment from the second position and orientation relative to the second point of reference, the generated representation configured for presentation via a display of the PMD;

wherein the representation of the field of view is generated based in part on sensor data gathered by one or more sensors associated with an FDA in autonomous flight over the physical environment.

13. The system of claim 12, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

track a change in the first position and orientation over time and updating the second position and orientation in real time or near real time, based on the tracked change in the first position and orientation.

14. The system of claim 12, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

present the representation of the field of view of the physical environment via a display of the PMD.

15. The system of claim 12, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate control commands configured to cause the FDA to fly to and maintain the second position relative to the second point of reference.

16. The system of claim 15, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate control commands configured to cause an FDA to adjust orientation and adjust image capture by an associated image capture device to correspond with the second orientation relative to the second point of reference.

17. The system of claim of claim 16, wherein adjusting image capture is accomplished by adjusting an orientation of the image capture device and/or applying digital image processing to images captured by the image capture device.

18. The system of claim 12, wherein the representation of the field of view from the second position and orientation is a live video feed from an image capture device associated with the FDA.

19. The system of claim 12, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate a virtual space including a three dimensional model of the physical environment within a threshold proximity of the second point of reference;

wherein the three-dimensional model is generated based in part on sensor data gathered by the one or more sensors associated with the FDA.

20. The system of claim 19, wherein the representation of the field of view from the second position and orientation includes one or more renderings of the virtual space from a field of view of a virtual camera associated with the virtual space, wherein the field of view of the virtual camera with relation to the virtual space corresponds with the field of view of the physical environment from the second position and orientation.

21. The system of claim 12, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

adjusting the scale factor in response to an input received via the PMD.

22. The system of claim 12, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

23. The system of claim 12, wherein the display associated with the PMD is a virtual reality headset display.

24. A computer-implemented method for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the method comprising:

presenting, via a touch display of the PMD, a visual representation of the aerial view of the physical environment;

wherein the visual representation of the aerial view is generated based in part on sensor data gathered by one or more sensors associated with the FDA in flight over the physical environment;

receiving, via the touch display of the PMD, a touch gesture, the touch gesture indicating a selection of a point or area in the visual representation of the aerial view of the physical environment;

identifying a point of reference in the physical environment corresponding to the selected point or area; and

generating control commands configured to cause the FDA to autonomously fly and/or adjust image capture relative to the point of reference.

25. The method of claim 24, further comprising:

defining a reference plane relative to the point of reference;

wherein the reference plane corresponds to one or more identifiable physical surfaces in the physical environment; and

wherein the generated control commands are configured to cause the FDA to fly a two dimensional path at a constant or substantially constant height above the reference plane.

26. The method of claim 25, wherein the touch gesture is a single touch, wherein the path is a substantially straight line towards the point of reference, and wherein the control commands are further configured to cause the FDA to focus image capture on the point of reference.

27. The method of claim 26, wherein focusing image capture includes motion by the FDA along the two dimensional path to within a threshold proximity of the point of reference and adjustment by an image capture device associated with the FDA to center the point of reference in a field of view of the image capture device.

28. The method of claim 25, wherein the touch gesture is drawn line, and wherein the two dimensional path along the reference plane is a projection of the drawn line.

29. The method of claim 25, wherein the touch gesture is a selection of one or more preset paths, and wherein the two dimensional path along the reference plane is based on the selected one or more preset paths.

30. The method of claim 24, wherein identification of the point of reference in the physical environment corresponding to the selected point or area is based in part on sensor data gathered from sensors associated with the FDA.

31. The method of claim 24, further comprising:

generating updated control commands in real time or near real time in response to a detected change in position and orientation of the FDA;

wherein the detected change in position and orientation of the FDA is based in part on sensor data gathered by sensors associated with the FDA.

32. A system for providing an aerial view of a physical environment using a flying digital assistant (FDA) and a portable multifunction device (PMD), the system comprising:

one or more processors; and

one or more memory units, the one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

present, via a touch display of the PMD, a visual representation of the aerial view of the physical environment;

wherein the visual representation of the aerial view is generated based in part on sensor data gathered by one or more sensors associated with the FDA in flight over the physical environment;

receive, via the touch display of the PMD, a touch gesture, the touch gesture indicating a selection of a point or area in the visual representation of the aerial view of the physical environment;

identify a point of reference in the physical environment corresponding to the selected point or area; and

generate control commands configured to cause the FDA to autonomously fly and/or adjust image capture relative to the point of reference.

33. The system of claim 32, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

define a reference plane relative to the point of reference;

wherein the reference plane corresponds to one or more identifiable physical surfaces in the physical environment; and

wherein the generated control commands are configured to cause the FDA to fly a two dimensional path at a constant or substantially constant height above the reference plane.

34. A computer-implemented method for controlling image capture by a flying digital assistant (FDA) in autonomous flight over a physical environment using a portable multifunction device (PMD), the method comprising:

capturing images by one or more image capture devices associated with the FDA, the image capture characterized by a line of sight;

tracking an orientation of the PMD;

adjusting the line of sight of image capture at the FDA to match to the tracked orientation of the PMD;

displaying the captured images on a display device;

35. The method of claim 34, wherein the tracked orientation of the PMD is based in part on sensor data gathered by sensors associated with the PMD.

36. The method of claim 34, wherein the display device is integrated with the PMD.

37. The method of claim 34, wherein the one or more image capture devices are adjustable using a hybrid mechanical digital gimbal mechanism.