Unmanned aerial vehicle sensor calibration during flight

• US 10,032,275 B1
• Filed: 06/21/2016
• Issued: 07/24/2018
• Est. Priority Date: 06/21/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Watson, Joshua John, Novak, Benjamin Griffin, O'Brien, Barry James, Wilcox, Scott Michael, Caro, Benjamin Israel, Boyd, Scott Patrick
• Primary Examiner(s): Paige, Tyler
• Application Number:
  US15/188,919
• Time in Patent Office: 763 Days
• Field of Search: 701 3
• CPC Class Codes: B64C 39/022 Tethered aircraft
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/60 for transporting passengers; for transporting goods other than weapons
• CPC Class Codes: B64U 2101/64 for parcel delivery or retrieval
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: B64U 50/11 using internal combustion piston engines
• CPC Class Codes: B64U 50/12 using turbine engines, e.g. turbojets or turbofans
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: B64U 80/25 for recharging batteries; for refuelling
• CPC Class Codes: G01C 25/00 Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass testing, calibrating or compensating compasses G01C17/38
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones

Abstract

This disclosure describes systems, methods, and apparatus for automating the verification of aerial vehicle sensors as part of a pre-flight, flight departure, in-transit flight, and/or delivery destination calibration verification process. At different stages, aerial vehicle sensors may obtain sensor measurements about objects within an environment, the obtained measurements may be processed to determine information about the object, as presented in the measurements, and the processed information may be compared with the actual information about the object to determine a variation or difference between the information. If the variation is within a tolerance range, the sensor may be auto adjusted and operation of the aerial vehicle may continue. If the variation exceeds a correction range, flight of the aerial vehicle may be aborted and the aerial vehicle routed for a full sensor calibration.

Claims

1. An unmanned aerial vehicle (UAV) in-flight sensor calibration method, comprising:

aerially navigating a UAV to position a marker within a field of view of a camera coupled to the UAV, wherein the marker is at a known position within an environment in which the UAV is operating and along a planned flight path of the UAV;

obtaining, with the camera, an image of the marker while the marker is in the field of view of the camera;

processing the image to determine at least one of a size of the marker as represented in the image, a shape of the marker as represented in the image, a color of the marker as represented in the image, a position of the marker as represented in the image, an orientation of the marker as represented in the image, or a pattern of the marker as represented in the image;

determining a variation between at least one of the size of the marker as represented in the image, the shape of the marker as represented in the image, the color of the marker as represented in the image, the position of the marker as represented in the image, the orientation of the marker as represented in the image, or the pattern of the marker as represented in the image and an actual size of the marker, an actual shape of the marker, an actual color of the marker, an actual position of the marker, an actual orientation of the marker, or an actual pattern of the marker;

determining that the variation is within a tolerance range; and

altering a calibration of at least one sensor of the UAV based at least in part on the variation; and

continuing a flight of the UAV toward a destination.

2. The UAV in-flight sensor calibration method of claim 1, wherein the marker is at least one of a physical structure within the environment, a celestial body, a transportation unit, a heat signature, a body of water, a graphical representation, or a projected pattern.

3. The UAV in-flight sensor calibration method of claim 1, wherein the at least one sensor is at least one of a camera, a distance determining element, an electronic speed control, a navigation controller, an altimeter, an inertial measurement unit, a barometer, a pitot sensor, a heading indicator, or a magnetic sensor.

4. The UAV in-flight sensor calibration method of claim 1, wherein determining the variation is performed by an aerial vehicle management system that is remote from the UAV.

5. An aerial vehicle, comprising:

a frame;

a sensor coupled to the frame;

an aerial vehicle control system in communication with the sensor and configured to at least;

cause the aerial vehicle to aerially navigate within an environment;

cause the sensor to obtain data representative of an object at a known location within the environment;

process the data to generate processed information that includes information indicative of at least one aspect of the object as represented in the data;

compare the processed information with actual information corresponding to the object to determine a variation between the processed information and the actual information;

determine if the variation is within a tolerance range; and

in response to a determination that the variation is within the tolerance range, continue the aerial navigation of the aerial vehicle within the environment.

6. The aerial vehicle of claim 5, wherein the comparison of the processed information with the actual information includes comparing a position of the object as represented in the processed information with an actual position of the object to determine a variation of the position of the object as represented in the data and the actual position of the object.

7. The aerial vehicle of claim 6, wherein the aerial vehicle control system is further configured to at least:

adjust a calibration of a navigation component of the aerial vehicle control system based at least in part on the variation.

8. The aerial vehicle of claim 6, wherein the position of the object as represented in the data is determined based at least in part on a processing of the data and orientation information of the aerial vehicle as maintained by a navigation component of the aerial vehicle control system.

9. The aerial vehicle of claim 5, wherein the aerial vehicle control system is further configured to at least:

adjust a calibration of the sensor of the aerial vehicle based at least in part on the variation.

10. The aerial vehicle of claim 5, wherein the comparison of the processed information with the actual information includes comparing a shape of the object as represented in the processed information with an actual shape of the object to determine a variation of the shape of the object as represented in the data and the actual shape of the object.

11. The aerial vehicle of claim 10, wherein the aerial vehicle control system is further configured to at least:

adjust a calibration of the sensor of the aerial vehicle based at least in part on the variation.

12. The aerial vehicle of claim 5, wherein the comparison of the processed information with the actual information includes comparing an orientation of the object as represented in the processed information with an actual orientation of the object to determine a variation of the orientation of the object as represented in the data and the actual orientation of the object.

13. The aerial vehicle of claim 12, wherein the aerial vehicle control system is further configured to at least:

adjust a calibration of a heading indicator of the aerial vehicle based at least in part on the variation.

14. The aerial vehicle of claim 5, wherein the object is at least one of a physical structure within the environment, a celestial body, a transportation unit, a heat signature, a body of water, a graphical representation, or a projected pattern.

15. The aerial vehicle of claim 5, wherein the aerial vehicle control system is further configured to at least:

adjust a calibration of at least one of a camera, an electronic speed control, a distance determining element, a navigation controller, an altimeter, an inertial measurement unit, a barometer, a pitot sensor, a heading indicator, or a magnetic sensor.

16. The aerial vehicle of claim 5, wherein a flight path navigated by the aerial vehicle within the environment is between a materials handling facility and a customer specified delivery destination for an item carried by the aerial vehicle.

17. An in-flight calibration method, comprising:

obtaining, during aerial navigation of an unmanned aerial vehicle (“

UAV”

) and with a sensor coupled to the UAV, data representative of a marker within an environment;

processing the data to generate processed information that includes at least one aspect of the marker as represented in the data;

determining a variation between the processed information and actual information that corresponds to at least one actual aspect of the marker;

determining that the variation between the processed information and the actual information does not exceed a threshold; and

continuing the aerial navigation of the UAV.

18. The in-flight calibration method of claim 17, further comprising:

altering a calibration of at least one sensor of the UAV based at least in part on the variation.

19. The in-flight calibration method of claim 18, wherein the variation is a difference between at least a portion of the processed information and the actual information.

20. The in-flight calibration method of claim 17, wherein the marker is a heat signature generated as a result of at least one of a heat generated by a motor of the UAV or a heat generated by a landing gear of the UAV.