Systems, devices, and methods for on-board sensing and control of micro aerial vehicles

• US 10,037,028 B2
• Filed: 07/25/2016
• Issued: 07/31/2018
• Est. Priority Date: 07/24/2015
• Status: Active Grant

• Current Assignee: Trustees Of The University Of Pennsylvania (University of Pennsylvania)
• Original Assignee: Trustees Of The University Of Pennsylvania (University of Pennsylvania)
• Inventors: Loianno, Giuseppe, Mulgaonkar, Yash Shailesh, Kumar, R. Vijay
• Primary Examiner(s): EL CHANTI, HUSSEIN A
• Application Number:
  US15/218,938
• Publication Number:
  US 20170023937A1
• Time in Patent Office: 736 Days
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 10/80 UAVs characterised by their small size, e.g. micro air vehicles [MAV]
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G08G 5/0026 located on the ground
• CPC Class Codes: G08G 5/003 Flight plan management
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

Systems, devices, and methods for on-board sensing and control of robotic vehicles (e.g., MAVs) using commercial off-the-shelf hand-held electronic devices as a sensing and control system are provided. In one aspect, a system for controlling a micro aerial vehicle may include one or more sensors, a state estimation module in communication with the one or more sensors, the state estimation module being configured to generate an estimated pose of the micro aerial vehicle based on inputs from the one or more sensors, and a position controller in communication with the state estimation module and configured to communicate attitude commands to an attitude controller of the micro aerial vehicle. Each of the one or more sensors, the state estimation module, and the position controller may be contained in a commercial off-the-shelf hand-held electronic device that is configured to be coupled to the micro aerial vehicle.

Claims

1. A system for controlling a micro aerial vehicle comprising:

one or more sensors, wherein the one or more sensors comprise one or more cameras and one or more of a gyroscope and/or an accelerometer;

a state estimation module in communication with the one or more sensors, the state estimation module comprising;

a visual inertial odometry system in communication with the one or more cameras and the one or more of a gyroscope and/or an accelerometer; and

an estimator in communication with the visual inertial odometry system and the one or more sensors, the estimator being configured to generate a state estimate of the micro aerial vehicle based on inputs from the one or more of a gyroscope and/or an accelerometer and to generate an estimated pose by updating the state estimate based on inputs from the visual inertial odometry system; and

a position controller in communication with the state estimation module and configured to communicate attitude commands to an attitude controller of the micro aerial vehicle;

wherein the state estimation module is configured to transmit the estimated pose to the position controller at a rate of about 200 Hz or greater; and

wherein each of the one or more sensors, the state estimation module, and the position controller are contained in a hand-held electronic device that is mounted to the micro aerial vehicle.

2. The system of claim 1, wherein the estimator comprises an Unscented Kalman Filter in communication with the one or more sensors.

3. The system of claim 1, wherein the state estimation module is configured to send visualization data of the estimated pose to a remote user interface.

4. The system of claim 1, wherein the position controller comprises a nonlinear controller.

5. The system of claim 1, wherein the position controller is configured to receive trajectory planning information from a remote user interface.

6. The system of claim 1, wherein the attitude controller is contained in the hand-held electronic device.

7. A method for controlling a micro aerial vehicle using a hand-held electronic device, the method comprising:

mounting the hand-held electronic device to the micro aerial vehicle;

receiving inputs from one or more sensors in the hand-held electronic device, wherein the one or more sensors comprise one or more cameras and one or more of a gyroscope and/or an accelerometer;

generating an estimated pose of the micro aerial vehicle based on the inputs from the one or more sensors, wherein generating the estimated pose comprises;

generating a state estimate of the micro aerial vehicle based on inputs from the one or more of a gyroscope and/or an accelerometer; and

using visual inertial odometry to localize dynamics of a body of the micro aerial vehicle with respect to an inertial frame and update the state estimate based on inputs from the one or more cameras;

generating attitude commands based on the estimated pose of the micro aerial vehicle; and

communicating the attitude commands to a control component of the micro aerial vehicle;

wherein generating an estimated pose of the micro aerial vehicle comprises updating the state estimate at a rate of about 200 Hz or greater;

wherein receiving inputs, generating an estimated pose, and generating attitude commands are all performed on the hand-held electronic device.

8. The method of claim 7, wherein generating an estimated pose comprises applying an Unscented Kalman Filter to the inputs from one or more sensors to estimate a full state of the micro aerial vehicle.

9. The method of claim 7, wherein generating attitude commands further comprises receiving trajectory planning information from a remote user interface.

10. The method of claim 7, wherein generating attitude commands further comprises coordinating a trajectory plan with at least one additional micro aerial vehicle.

11. The method of claim 10, wherein coordinating a trajectory plan comprises:

at a remote user interface, determining a solution for a system of equations that represent straight-line trajectories of each of the micro aerial vehicle and the at least one additional micro aerial vehicle;

sending coefficients of a respective one of the system of equations to the micro aerial vehicle; and

generating the attitude commands based on the coefficients.

12. The method of claim 7, wherein the control component is contained in the hand-held electronic device.

13. The method of claim 7, further comprising sending visualization data of the estimated pose to a remote user interface.

14. The method of claim 13, wherein the visualization data of the estimated pose comprises point clouds.