Unmanned aerial image capture platform

• US 10,520,943 B2
• Filed: 08/12/2016
• Issued: 12/31/2019
• Est. Priority Date: 08/12/2016
• Status: Active Grant

• Current Assignee: Skydio, Inc.
• Original Assignee: Skydio, Inc.
• Inventors: Martirosyan, Hayk, Bry, Adam, Donahoe, Matthew, Bacharach, Abraham, Sadowski, Justin Michael
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Gadomski, Stefan
• Application Number:
  US15/235,513
• Publication Number:
  US 20180046187A1
• Time in Patent Office: 1,236 Days
• Field of Search: None
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/106 Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/292 Multi-camera tracking

Abstract

Methods and systems are disclosed for an unmanned aerial vehicle (UAV) configured to autonomously navigate a physical environment while capturing images of the physical environment. In some embodiments, the motion of the UAV and a subject in the physical environment may be estimated based in part on images of the physical environment captured by the UAV. In response to estimating the motions, image capture by the UAV may be dynamically adjusted to satisfy a specified criterion related to a quality of the image capture.

Claims

1. A method comprising:

estimating a motion of an unmanned aerial vehicle (UAV) in autonomous flight through a physical environment based in part on images captured by an image capture device associated with the UAV;

estimating a motion of a subject in the physical environment based in part on the images captured by the image capture device associated with the UAV;

generating and continually updating a three-dimensional (3D) map of the physical environment while the UAV is in autonomous flight, the 3D map generated based in part on the images captured by the image capture device associated with the UAV;

defining, within the 3D map, a virtual line between a virtual representation of an estimated position of the subject and a virtual representation of an estimated position of the UAV; and

in response to estimating the motions of the UAV and the subject, generating control commands to dynamically adjust image capture of the subject by the image capture device associated with the UAV to satisfy a specified criterion related to a quality of the image capture;

wherein to satisfy the specified criterion the control commands to dynamically adjust image capture are configured such that, at a given time, the virtual line does not intersect a virtual representation of another physical object.

2. The method of claim 1, wherein the control commands to dynamically adjust image capture include control commands to dynamically adjust the motion of the UAV.

3. The method of claim 2, wherein the control commands to dynamically adjust the motion of the UAV include control commands configured to cause the UAV to avoid a collision with another physical object in the physical environment.

4. The method of claim 2, wherein the control commands to dynamically adjust the motion of the UAV include control commands configured to cause the UAV to fly along a continually updated flight path through the physical environment.

5. The method of claim 2, wherein the control commands to dynamically adjust the motion of the UAV include control commands to configured to cause the UAV to fly a pre-scripted flight path.

6. The method of claim 2, wherein the control commands to dynamically adjust the motion of the UAV include control commands configured to cause the UAV to fly a path of maximum energy efficiency.

7. The method of claim 1, wherein to satisfy the specified criterion, the control commands to dynamically adjust image capture are further configured such that, at a given time, image capture of the subject by the UAV is not obstructed by another physical object in the physical environment.

8. The method of claim 1, further comprising:

estimating a position of a light source in the physical environment based in part on the images captured by the image capture device associated with the UAV;

wherein to satisfy the specified criterion the control commands to dynamically adjust image capture are further configured such that, at a given time, the virtual line does not intersect a virtual representation of another physical object.

9. The method of claim 1,

wherein the light source is the Sun.

10. The method of claim 1, wherein estimating the motion of the UAV and the subject is further based in part on the generated 3D map.

11. The method of claim 1, further comprising:

identifying the subject in the physical environment by comparing captured images of the subject using trained two-dimensional (2D) and/or three-dimensional (3D) appearance models;

determining based on the identification of the subject and the estimated motion of the subject that the image capture of the subject is intended for a particular type of scene;

wherein, to satisfy the specified criterion, the control commands to dynamically adjust image capture are further configured such that the UAV follows a pre-scripted flight path associated with the particular type of scene.

12. The method of claim 1, further comprising:

predicting a future motion of the subject based on any of;

the estimated motion of the subject, a previously estimated motion of the subject, the captured images, or a 3D map of the physical environment;

wherein the control commands to dynamically adjust image capture are further based on the predicted future motion of the subject.

13. The method of claim 1, wherein the specified criterion is based on quantified metrics associated with subjective image capture quality standards.

14. The method of claim 13, wherein the subjective image capture quality standards are based on models generated by applying machine learning to historical data.

15. The method of claim 1, wherein the control commands to dynamically adjust image capture include control commands to adjust any of focus, aperture, shutter speed, light sensitivity (ISO), frame rate, color balance, image framing, or image stabilization.

16. The method of claim 1, wherein the image capture device includes:

an array of image capture devices; and

a gimbaled image capture device;

wherein the motions of the subject and UAV are estimated based on images captured by the array of image capture devices; and

wherein the control commands are configured to dynamically adjust image capture by the gimbaled image capture device.

17. The method of claim 1, wherein the motion of the UAV is estimated using a process of visual inertial odometry.

18. The method of claim 16, wherein the gimbaled image capture device includes a camera mounted to a motorized gimbal.

19. The method of claim 18, wherein the motorized gimbal is a hybrid mechanical digital gimbal, the hybrid mechanical digital gimbal configured to:

adjust an orientation of the camera about a first axis; and

process images captured by the camera to simulate adjustments about a second axis.

20. The method of claim 1, wherein the images captured by the image capture device include video.

21. A system comprising:

a processing unit; and

a memory unit, the memory unit having instructions stored thereon, which when executed by the processing unit cause the system to;

estimate a motion of an unmanned aerial vehicle (UAV) in autonomous flight through a physical environment based in part on images captured by an image capture device associated with the UAV;

estimate a motion of a subject in the physical environment based in part on the images captured by the image capture device associated with the UAV;

generate and continually update a three-dimensional (3D) map of the physical environment while the UAV is in autonomous flight, the 3D map generated based in part on the images captured by the image capture device associated with the UAV;

define, within the 3D map, a virtual line between a virtual representation of an estimated position of the subject and a virtual representation of an estimated position of the UAV; and

in response to estimating the motions of the UAV and the subject, generate control commands to dynamically adjust image capture of the subject by the image capture device associated with the UAV to satisfy a specified criterion related to a quality of the image capture;

wherein to satisfy the specified criterion the control commands to dynamically adjust image capture are configured such that, at a given time, the virtual line does not intersect a virtual representation of another physical object.

22. An unmanned aerial vehicle (UAV) comprising:

an array of image capture devices configured to capture images of a physical environment surrounding the UAV, while in flight;

a gimbaled image capture device configured to capture images of a subject in the physical environment;

a processing unit communicatively coupled to the array of image capture devices and the gimbaled image capture device; and

a memory unit having instructions stored thereon, which when executed by the processing unit, cause the processing unit to;

estimate a motion of the UAV while in autonomous flight through the physical environment based in part on the-images captured by the array of image capture devices;

estimate a motion of the subject in the physical environment based in part on the images captured by the array of image capture devices;

generate and continually update a three-dimensional (3D) map of the physical environment while the UAV is in autonomous flight, the 3D map generated based in part on the images captured by the array of image capture devices;

define, within the 3D map, a virtual line between a virtual representation of an estimated position of the subject and a virtual representation of an estimated position of the UAV; and

in response to estimating the motions of the UAV and the subject, generate control commands to dynamically adjust image capture of the subject by the gimbaled image capture device to satisfy a specified criterion related to a quality of the image capture;

wherein to satisfy the specified criterion the control commands to dynamically adjust image capture are configured such that, at a given time, the virtual line does not intersect a virtual representation of another physical object.