Action camera system for unmanned aerial vehicle

• US 9,769,387 B1
• Filed: 11/05/2014
• Issued: 09/19/2017
• Est. Priority Date: 11/05/2013
• Status: Active Grant

• Current Assignee: Draganfly Innovations Inc. (Frontier Energy Corp.)
• Original Assignee: Trace Live Network Inc.
• Inventors: Beard, Paul, McDermott, Craig, Chell, Cameron, Racicot, Erika, Clarke, Jamie, McDonald, Bruce, Readwin, Paul
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US14/533,995
• Time in Patent Office: 1,049 Days
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 30/2247 Characters composed of bars, e.g. CMC-7
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An action camera system for an unmanned aerial vehicle (UAV) selects a target based on a reference image captured by an onboard camera. Image processing determines a desired orientation of the target to the UAV, by which the UAV can track the target and provide streaming video images from the desired orientation. Image processing establishes a visual lock on the target and controls the UAV to maintain the desired orientation while capturing streaming images. Additional position data provided by sensors aboard a smartphone carried by the target enhances the tracking ability of the action camera system and enables predictive analysis of the target'"'"'s future position. The action camera system may additionally provide preselected modes of operation that control UAV movement and image capture depending on the user'"'"'s desired objectives.

Claims

1. A method of tracking at least one target selected from one or more potential target objects via an unmanned aerial vehicle (UAV), comprising:

selecting the at least one target;

capturing at least one first image via an onboard camera, the at least one first image defining a first orientation of the at least one target to the UAV and including at least one first image element corresponding to at least one of the target and a pattern uniquely associated with the at least one target;

defining, based on at least the at least one first image element, one or more first tracking parameters associated with the at least one first orientation;

capturing at least one second image via the onboard camera, the at least one second image defining a second orientation of the at least one target to the UAV and including at least one second image element;

defining, based on at least the at least one second image element, one or more second tracking parameters associated with the at least one second orientation;

receiving aboard the UAV at least one first element of position data determined by a sensor from a communications device associated with the at least one target; and

maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on at least one of a second tracking parameter and the at least one first element of position data.

2. The method of claim 1, wherein the at least one first image element includes at least

a first bounding box corresponding to the at least one target, the first bounding box associated with the first orientation and having a first pixel size and a first position relative to the at least one first image;

and

a second bounding box corresponding to the at least one pattern uniquely associated with the at least one target, the second bounding box associated with the first orientation and having a second pixel size and a second position relative to the at least one first image.

3. The method of claim 2, wherein the defining, based on at least the at least one first image element, one or more first tracking parameters associated with the at least one first orientation includes:

defining the one or more first tracking parameters based on at least one of the first pixel size, the first position, the second pixel size, and the second position.

4. The method of claim 1, wherein:

the one or more first tracking parameters include at least one of a first distance between the UAV and the at least one target, a first angle of elevation between the UAV and the at least one target, and a first bearing between the UAV and the at least one target; and

the one or more second tracking parameters include at least one of a second distance between the UAV and the at least one target, a second angle of elevation between the UAV and the at least one target, and a second bearing between the UAV and the at least one target.

5. The method of claim 1, wherein the defining, based on at least the at least one second image element, one or more second tracking parameters associated with the at least one second orientation includes

attempting to identify within the at least one second image at least one third image element corresponding to the at least one target, the at least one third image element including a third bounding box associated with the second orientation and having a third pixel size and a third position relative to the at least one second image;

attempting to identify within the at least one second image at least one fourth image element corresponding to the pattern uniquely associated with the at least one target, the at least one fourth image element including a fourth bounding box associated with the second orientation and having a fourth pixel size and a fourth position relative to the at least one second image;

defining the at least one or more second tracking parameters based on at least one of the third pixel size, the third position, the fourth pixel size, and the fourth position; and

if at least one of the third image element and the fourth image element cannot be identified, defining the one or more second tracking parameters based on at least one of the at least one element of position data and at least one previously determined second tracking parameter.

6. The method of claim 1, wherein the receiving aboard the UAV at least one first element of position data determined by a sensor from the at least one communications device includes

receiving aboard the UAV at least one first element of position data determined at least one of an accelerometer, an inclinometer, a gyrometer, a magnetometer, a GNSS receiver, and an application installed on the at least one communications device from the at least one communications device.

7. The method of claim 1, further comprising:

determining at least one of a current position, an acceleration, a velocity, a heading, a bearing, and a future position based on the at least one element of position data.

8. The method of claim 7, wherein the maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on at least one of a second tracking parameter and the at least one first element of position data includes

maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on at least one of a second tracking parameter and the at least one of a current position, an acceleration, a velocity, a heading, a bearing, and a future position.

9. The method of claim 1, wherein the maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on at least one of a second tracking parameter and the at least one first element of position data includes:

comparing the second orientation to the first orientation;

maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on the comparison of the second orientation to the first orientation.

10. The method of claim 1, further comprising:

selecting at least one operating mode from a plurality of operating modes, the at least one operating mode including at least one of an image resolution, a frame speed, an aspect ratio, an orientation of the at least one target to the UAV, and a sequence of movements of the UAV relative to the at least one target.

11. The method of claim 10, wherein the maintaining the first orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on at least one of a second tracking parameter and the at least one first element of position data includes:

maintaining an orientation of at least one target to the UAV by adjusting at least one of an angle of the onboard camera or a rotor speed of the UAV based on the at least one operating mode.

12. An action camera system for tracking at least one target with an unmanned aerial vehicle (UAV), comprising:

at least one camera fixed to the UAV and configured to capture (a) at least one first image defining a first orientation of the at least one target to the UAV and (b) at least one second image defining a second orientation of the at least one target to the UAV;

at least one imaging processor connected to the at least one camera and configured to;

(a) identify within the at least one first image at least one first image element corresponding to the at least one target, the at least one first image element including a first bounding box associated with the first orientation and having a first pixel size and a first position relative to the at least one first image;

(b) identify within the at least one first image at least one second image element corresponding to a pattern uniquely associated with the at least one target, the at least one second image element including a second bounding box associated with the first orientation and having a second pixel size and a second position relative to the at least one first image;

(c) determine one or more first tracking parameters associated with the first orientation, based on at least the first image element and the second image element;

(d) receive at least one element of position data determined by a sensor via a wireless link connected to a communications device associated with the at least one target;

(e) identify within the at least one second image at least one third image element corresponding to the at least one target, the at least one third image element including a third bounding box associated with the second orientation and having a third pixel size and a third position relative to the at least one second image;

(f) identify within the at least one second image at least one fourth image element corresponding to the pattern uniquely associated with the at least one target, the at least one fourth image element including a fourth bounding box associated with the second orientation and having a fourth pixel size and a fourth position relative to the at least one second image;

(g) determine at least one or more second tracking parameters associated with the second orientation based on at least one of the at least one third image element, the at least one fourth image element, the at least one element of position data, and at least one previously determined second tracking parameter; and

(h) maintain the first orientation of the at least one target to the UAV by at least one of adjusting an angle of the onboard camera and directing the attitude control system to adjust a rotor speed of the UAV based on at least one of the one or more second tracking parameters and the at least one first element of position data;

at least one attitude control system fixed to the UAV and connected to the at least one imaging processor, the at least one attitude control system configured to adjust at least one rotor speed of the UAV; and

at least one attitude sensor connected to the attitude control system and configured to determine at least one of an angle of rotation of the UAV, an orientation of the UAV, a position of the UAV, a velocity of the UAV, and an acceleration of the UAV.

13. The action camera system of claim 12, wherein the at least one imaging processor is configured to determine at least one of a distance between the at least one target and the UAV, an angle of elevation between the at least one target and the UAV, and a bearing of the at least one target relative to the UAV.

14. The action camera system of claim 12, wherein the at least one imaging processor is configured to receive at least one element of position data determined by at least one of an accelerometer, an inclinometer, a gyrometer, a magnetometer, a GNSS receiver, and an application installed on the communications device via the wireless link connected to the communications device associated with the at least one target.

15. The action camera system of claim 12, wherein the at least one imaging processor is configured to determine at least one of a current position, a velocity, an acceleration, a heading, a bearing, and a future position based on the at least one element of position data.

16. The action camera system of claim 15, wherein the at least one imaging processor is configured to maintain the first orientation of the at least one target to the UAV by at least one of adjusting an angle of the onboard camera and directing the attitude control system to adjust a rotor speed of the UAV based on at least one of the current position, the velocity, the acceleration, the heading, the bearing, and the future position.

17. The action camera system of claim 12, wherein the at least one imaging processor is further configured to:

compare the second orientation to the first orientation; and

maintain the first orientation of at least one target to the UAV by at least one of adjusting an angle of the onboard camera and directing the attitude control system to adjust a rotor speed of the UAV based on the comparison of the second orientation to the first orientation.

18. The action camera system of claim 12, wherein the at least one imaging processor is further configured to:

select at least one operating mode from a plurality of operating modes, the at least one operating mode including at least one of an image resolution, a frame speed, an aspect ratio, an orientation of the at least one target to the UAV, and a sequence of movements of the UAV relative to the at least one target; and

maintain the first orientation of at least one target to the UAV by at least one of adjusting an angle of the onboard camera and directing the attitude control system to adjust a rotor speed of the UAV based on the at least one operating mode.

19. The action camera system of claim 12, wherein the action camera system is embodied in an unmanned aerial vehicle comprising:

at least one rotor; and

at least one motor connected to the at least one rotor and to a power source, the at least one motor configured to rotatably drive the at least one rotor;

wherein the action camera system and the at least one rotor are fixed to at least one of a frame and a platform.

20. An apparatus for focusing an image of a target on a sensor, comprising:

(a) a frame for attaching said apparatus to at least one of a stationary or movable platform;

(b) a sensor for detecting electromagnetic radiation in at least two-dimensions;

(c) an optical string for focusing an electromagnetic image on said sensor;

(d) storage for storing said focused electromagnetic images from said sensor;

(e) a computer for tracking an object moving in relation to said frame;

(f) a first detector for receiving target location information from at least two of sonar, reflected light, and radio frequency;

(g) a second detector for receiving location information from a target sensor array having a wireless transponder, said target sensor array including at least one of a GNSS system, a cellular link, a magnetometer, and an accelerometer;

(h) a user interface for selectively programming said computer with information including at least target location and target activity;

(i) a target movement predictor for assisting in tracking said target based at least in part on target activity; and

(j) output means for controlling said frame in relation to said at least one of a stationary platform and in directing the movement of a movable platform.