Magic wand interface and other user interaction paradigms for a flying digital assistant

• US 9,678,506 B2
• Filed: 04/16/2015
• Issued: 06/13/2017
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Bachrach, Abraham, Bry, Adam, Donahoe, Matthew
• Primary Examiner(s): FREJD, RUSSELL WARREN
• Application Number:
  US14/688,998
• Publication Number:
  US 20150370250A1
• Time in Patent Office: 789 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 20/87 Mounting of imaging devices, e.g. mounting of gimbals
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 23/00 Combined instruments indicating more than one navigational value, e.g. for aircraft; Combined measuring devices for measuring two or more variables of movement, e.g. distance, speed or acceleration
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target

Abstract

Methods and systems are described for new paradigms for user interaction with an unmanned aerial vehicle (referred to as a flying digital assistant or FDA) using a portable multifunction device (PMD) such as smart phone. In some embodiments, a magic wand user interaction paradigm is described for intuitive control of an FDA using a PMD. In other embodiments, methods for scripting a shot are described.

Claims

1. A method for controlling a flying digital assistant (FDA), the method comprising:

determining, by one or more processors, a current position of the FDA relative to a point of reference;

receiving a first input, the first input based on a detected motion of a portable multifunction device (PMD);

calculating, by one or more processors, a first target position of the FDA relative to the point of reference based on the detected motion of the PMD;

tracking the motion the PMD over time;

updating, by one or more processors, the first target position of the FDA based on the tracked motion of the PMD over time;

recognizing the first input as one or more of a plurality of preset gestures based on the tracked motion of the PMD over time;

calculating, by one or more processors, a predicted path of motion for the FDA from the current position to the first target position based on the first input; and

generating, by one or more processors, control commands configured to cause the FDA to maneuver from the current position, along the predicted path, to the first target position relative to the point of reference.

2. The method of claim 1, further comprising:

determining, by one or more processors, a current orientation of the FDA relative to the point of reference;

calculating, by one or more processors, a target orientation of the FDA relative to the point of reference based on the detected motion of the PMD;

updating, by one or more processors, the target orientation of the FDA based on the tracked motion of the PMD over time; and

generating, by one or more processors, control commands configured to cause the FDA to maneuver from the current orientation to the target orientation.

3. The method of claim 1, further comprising:

defining a three dimensional coordinate system relative to the point of reference;

wherein, any given point in the coordinate system is defined by a first, second, and third coordinate;

wherein the generated control commands are configured such that the position of the FDA at any point between the current position and the first target position is constrained such that the first coordinate remains constant or substantially constant.

4. The method of claim 3, wherein the three dimensional coordinate system is defined as a cylindrical polar coordinate system;

wherein the first coordinate is a radial distance r from a reference axis intersecting the point of reference;

wherein the second coordinate is an azimuth angle θ

about the reference axis; and

wherein the third coordinate is a distance z along the reference axis.

5. The method of claim 3, wherein the three dimensional coordinate system is defined as a spherical polar coordinate system;

wherein the first coordinate is a radial distance r from the point of reference;

wherein the second coordinate is an azimuth angle θ

about a reference axis intersecting the point of reference; and

wherein the third coordinate is an inclination angle φ

off a reference plane normal to the reference axis.

6. The method of claim 3, wherein the three dimensional coordinate system is defined as a Cartesian coordinate system;

wherein, the first coordinate is a distance z off a two-dimensional plane associated with a physical surface.

7. The method of claim 3, further comprising:

receiving a second input, the second input received via the PMD;

determining, by one or more processors, an updated current position of the FDA relative to the point of reference;

calculating, by one or more processors, a second target position of the FDA relative to the point of reference based on the second input;

generating, by one or more processors, control commands configured to cause the FDA to maneuver from the updated current position to the second target position;

wherein, the generated control commands are configured such that the position of the FDA at any point between the updated current position and the second target position is constrained such that the second and third coordinates remain constant or substantially constant.

8. The method of claim 7, wherein the second input is received via a touch screen display of the PMD.

9. The method of claim 7, wherein the second input is based in part on a detected motion of the PMD.

10. The method of claim 9, wherein the detected motion of the PMD is recognizable as a throwing gesture, and wherein the generated commands are further configured to cause the FDA to follow a trajectory while maneuvering from the updated current position to the second target position, the trajectory corresponding to a characteristic of the throwing gesture.

11. The method of claim 1, wherein the FDA includes an image capture device;

wherein the image capture device is configured to track a second point of reference relative to the position of the FDA.

12. The method of claim 1, wherein the point of reference is constant relative to a position of the PMD at all times.

13. The method of claim 1, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

14. The method of claim 1, wherein the determined current position of the FDA relative to the point of reference is based in part on sensor data gathered by sensors at the FDA, and wherein the detected motion of the PMD is based in part on sensor data gathered by sensors at the PMD.

15. The method of claim 1, further comprising:

detecting an obstacle while the FDA is in motion; and

generating, by one or more processors, control commands configured to cause the FDA to maneuver to avoid the obstacle.

16. The method of claim 1, wherein the PMD comprises a position and/or orientation sensor, and a communications interface.

17. A system for controlling a flying digital assistant (FDA), the system comprising:

one or more processors; and

one or more memory units, the one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

determine a current position of the FDA relative to a point of reference;

receive a first input, the first input based on a detected motion of a portable multifunction device (PMD);

calculate, a first target position of the FDA relative to the point of reference based on the detected motion of the PMD;

track the motion of the PMD over time;

update the first target position of the FDA based on the tracked motion of the PMD over time;

recognize the first input as one or more of a plurality of preset gestures based on the tracked motion of the PMD over time;

calculate a predicted path of motion for the FDA from the current position to the first target position based on the first input; and

generate control commands configured to cause the FDA to maneuver from the current position, along the predicted path, to the first target position, relative to the point of reference.

18. The system of claim 17, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

determine a current orientation of the FDA relative to the point of reference;

calculate a target orientation of the FDA relative to the point of reference based on the detected motion of the PMD;

update the target orientation of the FDA based on the tracked motion of the PMD over time; and

generate control commands configured to cause the FDA to maneuver from the current orientation to the target orientation.

19. The system of claim 17, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

define a three dimensional coordinate system relative to the point of reference;

wherein, any given point in the coordinate system is defined by a first, second, and third coordinate;

wherein the generated control commands are configured such that the position of the FDA at any point between the current position and the first target position is constrained such that the first coordinate remains constant or substantially constant.

20. The system of claim 19, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

receive a second input, the second input received via the PMD;

determine an updated current position of the FDA relative to the point of reference;

calculate a second target position of the FDA relative to the point of reference based on the second input; and

generate control commands configured to cause the FDA to maneuver from the updated current position to the second target position;

wherein, the generated control commands are configured such that the position of the FDA at any point between the updated current position and the second target position is constrained such that the second and third coordinates remain constant or substantially constant.

21. The system of claim 17, wherein the FDA includes an image capture device configured to track a second point of reference relative to the position and orientation of the FDA.

22. The system of claim 17, wherein the point of reference is constant relative to the position of the PMD at all times.

23. The system of claim 17, wherein the positions and orientations of the FDA or PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

24. The system of claim 17, wherein the determined current position of the FDA relative to the point of reference is based in part on sensor data gathered by sensors at the FDA, and wherein the detected motion of the PMD is based in part on sensor data gathered by sensors at the PMD.

25. The system of claim 17, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

detect an obstacle while the FDA is in motion; and

generate control commands configured to cause the FDA to maneuver to avoid the obstacle.

26. The system of claim 17, wherein the PMD comprises a position and/or orientation sensor, and a communications interface.

27. A method for scripting image capture by a flying digital assistant (FDA), the FDA including an image capture device, the method comprising:

tracking a position and orientation of a portable multifunction device (PMD), relative to a first point of reference, over a period of time;

generating a first flight path, relative to the first point of reference, based on the tracked position and orientation of the PMD over the period of time;

translating the first flight path into a second flight path, relative to a second point of reference, wherein the second flight path is a scaled version of the first flight path;

determining a position and orientation of the FDA relative to the second point of reference; and

generating a set of control commands configured to cause the FDA to move along the second flight path and adjust image capture by the image capture device, based on the second flight path and the determined position and orientation of the FDA.

28. The method of claim 27, wherein the second point of reference is associated with a physical environment, and wherein the first point of reference is associated with a scale model of the physical environment.

29. The method of claim 27, wherein generating the set of control commands occurs in real time or near real time with the tracking of the position and orientation of the PMD.

30. The method of claim 27, further comprising:

automatically adjusting the second flight path, while the FDA is moving along the second flight path, based on sensor data gathered by sensors associated with the FDA.

31. The method of claim 27, further comprising:

detecting an obstacle while the FDA is moving along the second flight path; and

automatically adjusting the second flight path to avoid a collision between the obstacle and the FDA.

32. The method of claim 27, wherein the PMD includes a camera and a display device, and wherein the orientation of the PMD at any point during the period of time includes a field of view captured by the camera, the field of view being viewable via the display device.

33. The method of claim 27, further comprising:

generating a three dimensional model of the physical environment associated with the second point of reference, the generated three dimensional model based on sensor data gathered by sensors associated with the FDA.

34. The method of claim 27, wherein the positions and orientations of the FDA and PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

35. A system for scripting image capture by a flying digital assistant (FDA), the FDA including an image capture device, the system comprising:

one or more processors; and

one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

track a position and orientation of a portable multifunction device (PMD), relative to a first point of reference, over a period of time;

generate a first flight path, relative to the first point of reference, based on the tracked position and orientation of the PMD over the period of time;

translate the first flight path into a second flight path, relative to a second point of reference, wherein the second flight path is a scaled version of the first flight path;

determine a position and orientation of the FDA relative to the second point of reference; and

generate set of control commands configured to cause the FDA to move along the second flight path and adjust image capture by the image capture device, based on the second flight path and the determined position and orientation of the FDA.

36. The system of claim 35, wherein the second point of reference is associated with a physical environment, and wherein the first point of reference is associated with a scale model of the physical environment.

37. The system of claim 35, wherein generating the set of control commands occurs in real time or near real time with the tracking of the position and orientation of the PMD.

38. The system of claim 35, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

automatically adjust the second flight path, while the FDA is moving along the second flight path, based on sensor data gathered by sensors associated with the FDA.

39. The system of claim 35, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

detect an obstacle while the FDA is moving along the second flight path; and

automatically adjusting the second flight path to avoid a collision between the obstacle and the FDA.

40. The system of claim 35, wherein the PMD includes a camera and a display device, and wherein the orientation of the PMD at any point during the period of time includes a field of view captured by the camera, the field of view being viewable via the display device.

41. The system of claim 35, wherein the one or more memory units have further instructions stored thereon which when executed by the one or more processors, cause the system to further:

generate a three dimensional model of the physical environment associated with the second point of reference, the generated three dimensional model based on sensor data gathered by sensors associated with the FDA.

42. The system of claim 35, wherein the positions and orientations of the FDA and PMD are determined using a process of visual inertial odometry based on sensor data gathered by one or more of an optical sensor, accelerometer, gyroscope, and inertial measurement unit.

43. A method for controlling a flying digital assistant (FDA), the method comprising:

determining, by one or more processors, a current position of the FDA relative to a point of reference;

defining a three dimensional coordinate system relative to the point of reference;

wherein, any given point in the coordinate system is defined by a first, second, and third coordinate;

receiving a first input, the first input based on a detected motion of the PMD;

calculating, by one or more processors, a first target position of the FDA relative to the point of reference based on the detected motion of the PMD;

tracking the motion the PMD over time;

updating, by one or more processors, the first target position of the FDA based on the tracked motion of the PMD over time;

generating, by one or more processors, control commands configured to cause the FDA to maneuver from the current position to the first target position;

wherein the generated control commands are configured such that the position of the FDA at any point between the current position and the first target position is constrained such that the first coordinate remains constant or substantially constant;

receiving a second input, the second input received via the PMD;

determining, by one or more processors, an updated current position of the FDA relative to the point of reference;

calculating, by one or more processors, a second target position of the FDA relative to the point of reference based on the second input;

generating, by one or more processors, control commands configured to cause the FDA to maneuver from the updated current position to the second target position;

wherein, the generated control commands are configured such that the position of the FDA at any point between the updated current position and the second target position is constrained such that the second and third coordinates remain constant or substantially constant.

44. The method of claim 43, wherein the second input is received via a touch screen display of the PMD.

45. The method of claim 43, wherein the second input is based in part on a detected motion of the PMD.

46. The method of claim 45, wherein the detected motion of the PMD is recognizable as a throwing gesture, and wherein the generated commands are further configured to cause the FDA to follow a trajectory while maneuvering from the updated current position to the second target position, the trajectory corresponding to a characteristic of the throwing gesture.

47. A system for controlling a flying digital assistant (FDA), the system comprising:

one or more processors; and

one or more memory units, the one or more memory units having instructions stored thereon, which when executed by the one or more processors, cause the system to;

determine a current position of the FDA relative to a point of reference;

define a three dimensional coordinate system relative to the point of reference;

wherein, any given point in the coordinate system is defined by a first, second, and third coordinate;

receive a first input, the first input based on a detected motion of a portable multifunction device (PMD);

calculate, a first target position of the FDA relative to the point of reference based on the detected motion of the PMD;

track the motion of the PMD over time;

update the first target position of the FDA based on the tracked motion of the PMD over time;

generate control commands configured to cause the FDA to maneuver from the current position to the first target position

wherein the generated control commands are configured such that the position of the FDA at any point between the current position and the first target position is constrained such that the first coordinate remains constant or substantially constant;

receive a second input, the second input received via the PMD;

determine an updated current position of the FDA relative to the point of reference;

calculate a second target position of the FDA relative to the point of reference based on the second input; and

generate control commands configured to cause the FDA to maneuver from the updated current position to the second target position;

wherein, the generated control commands are configured such that the position of the FDA at any point between the updated current position and the second target position is constrained such that the second and third coordinates remain constant or substantially constant.