Pixel based image tracking system for unmanned aerial vehicle (UAV) action camera system

• US 9,785,147 B1
• Filed: 09/02/2016
• Issued: 10/10/2017
• Est. Priority Date: 08/13/2014
• Status: Active Grant

• Current Assignee: Draganfly Innovations Inc. (Frontier Energy Corp.)
• Original Assignee: Trace Live Network Inc.
• Inventors: McDermott, Craig, McDonald, Bruce
• Primary Examiner(s): Camby, Richard
• Application Number:
  US15/256,193
• Time in Patent Office: 403 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 10/14 with four distinct rotor axes, e.g. quadcopters
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/101 specially adapted for aircraft
• CPC Class Codes: G05D 1/12 Target-seeking control
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

An unmanned aerial vehicle (UAV) may select a subject to track or follow and capture images including the subject. A visual recognition system of the UAV may generate a profile of reference images portraying the subject breaking the reference images down into pixel sets corresponding to the subject and his/her components and distinguishing characteristics (body parts, clothing, facial features, accessories). The visual recognition system may break the incoming stream of incoming images into pixel sets, analyzing the pixel sets to distinguish the subject from his/her surroundings (i.e., in a crowd) and determine movement of the subject and the current orientation of the UAV to the subject. The UAV may then change its heading, velocity, or position based on any difference between the current orientation and the desired or predetermined orientation between the UAV and the subject.

Claims

1. An unmanned aerial vehicle (UAV), comprising:

at least one rotor fixed to an airframe, the at least one rotor coupled to one or more motors configured to drive the at least one rotor at at least one rotor speed;

an attitude control system fixed to the airframe and configured to;

determine at least one first orientation of the UAV;

adjust the first orientation by adjusting the at least one rotor speed via the one or more motors;

at least one camera attached to the airframe, the camera configured to;

capture at least one reference image, the reference image corresponding to at least one of a subject and a reference orientation of the UAV to the subject; and

capture at least one current image;

a visual recognition system operatively coupled to the camera and configured to;

generate at least one reference pixel set based on the reference image, the reference pixel set including at least one reference pixel subset (RPS) corresponding to at least one aspect of the subject;

generate at least one current pixel set based on the current image, the current pixel set including at least one of

     1) a subject pixel subset (SPS) corresponding to the subject;

     2) a non-subject pixel subset (NSPS), and

     3) a potential subject pixel subset (PSPS);

classify the at least one PSPS as a SPS or a NSPS by comparing the PSPS to one or more of the at least one RPS and the at least one SPS; and

determine a current orientation of the UAV to the subject based on the at least one SPS;

at least one non-transitory memory coupled to the visual recognition system and configured to store one or more of the reference pixel set and the current pixel set; and

a subject tracking system operatively coupled to the visual recognition system and the memory and configured to;

define a target orientation of the UAV to the subject;

maintain the target orientation by directing the attitude control system to adjust the at least one rotor speed;

determine at least one position change of the UAV by comparing one or more of the target orientation and the current orientation; and

direct the attitude control system to adjust the at least one rotor speed based on the at least one position change.

2. The UAV of claim 1, wherein the visual recognition system is configured to classify the at least one PSPS as a SPS or a NSPS based on one or more of:

a first size of the PSPS relative to the associated current pixel set;

a first position of the PSPS relative to the associated current pixel set;

a first shape of the PSPS;

a first edge of the PSPS; and

comparing at least one first pixel of the PSPS with at least one second pixel of one or more of the SPS and the RPS.

3. The UAV of claim 1, wherein the at least one SPS includes at least one component pixel subset (CPS) corresponding to the at least one aspect.

4. The UAV of claim 3, wherein the at least one aspect includes at least one of a body part of the subject, an article of clothing of the subject, an accessory of the subject, and a distinguishing feature of the subject.

5. The UAV of claim 3, wherein the at least one aspect includes at least one of a color, a texture, a reflectivity, and a pattern.

6. The UAV of claim 3, wherein the visual recognition system is configured to determine a current orientation of the UAV to the subject based on one or more of:

a second size of the CPS relative to the associated SPS;

a second position of the CPS relative to the associated SPS;

a second shape of the CPS;

a second edge of the CPS; and

comparing at least one of a first CPS of a first SPS and a second CPS of a second SPS, the first CPS and the second CPS corresponding to at least one first aspect of the at least one aspect.

7. The UAV of claim 1, wherein the visual recognition system is configured to determine a current orientation of the UAV to the subject based on one or more of:

a third size of the SPS relative to the associated current pixel set;

a third position of the SPS relative to the associated current pixel set;

a third shape of the SPS;

a third edge of the SPS; and

comparing at least one of a third SPS of a first current pixel set and a fourth SPS of a second current pixel set.

8. The UAV of claim 1, wherein the subject tracking system is configured to maintain the target orientation by framing the subject centrally in the current image.

9. The UAV of claim 1, wherein the subject tracking system is configured to determine the at least one position change by comparing at least one of a first current orientation corresponding to a first current image and a second current orientation corresponding to a second current image.

10. The UAV of claim 1, wherein the at least one position change is a position change, and the subject tracking system is configured to determine the at least one position change by determining at least one of a second position change of the subject, a heading change of the subject, and a velocity change of the subject.

11. The UAV of claim 1, wherein the at least one position change is associated with at least one of a heading of the UAV, a velocity of the UAV, and a rotational orientation of the UAV.

12. A method for tracking a subject via an unmanned aerial vehicle (UAV), the method comprising:

capturing at least one reference image of a subject via a camera attached to a UAV, the reference image corresponding to a reference orientation of the UAV to the subject;

generating at least one reference pixel set based on the reference image, the reference pixel set including at least one reference pixel subset (RPS) corresponding to an aspect of the subject;

determining at least one target orientation of the UAV to the subject;

maintaining the target orientation by adjusting at least one of a position of the UAV, a heading of the UAV, and a rotational orientation of the UAV;

capturing at least one current image via the camera;

generating at least one current pixel set based on the current image, the current pixel set including at least one of;

a subject pixel subset (SPS) corresponding to at least one of the subject and a current orientation of the UAV to the subject;

a non-subject pixel subset (NSPS) not corresponding to the subject; and

a potential subject pixel subset (PSPS);

classifying the at least one PSPS as a SPS or a NSPS by comparing the PSPS to one or more of the at least one RPS and the at least one SPS;

determining the current orientation based on the at least one SPS;

determining at least one position change of the UAV by comparing one or more of the target orientation and the current orientation; and

adjusting at least one of the position, the heading, and the rotational orientation based on the at least one position change.

13. The method of claim 12, wherein classifying the at least one PSPS as a SPS or a NSPS by comparing the PSPS to one or more of the at least one RPS and the at least one SPS includes:

comparing at least one first pixel of the PSPS with at least one second pixel of one or more of the SPS and the RPS.

14. The method of claim 12, wherein generating at least one current pixel set based on the current image includes:

defining at least one component pixel set (CPS) of the SPS, the CPS corresponding to the at least one aspect.

15. The method of claim 14, wherein determining the current orientation based on the at least one SPS includes:

determining the current orientation based on at least one of a size of the CPS relative to the associated SPS, a position of the CPS relative to the associated SPS, a shape of the CPS, and an edge of the CPS.

16. The method of claim 14, wherein determining the current orientation based on the at least one SPS includes:

comparing at least one of a first CPS of a first SPS and a second CPS of a second SPS, the first CPS and the second CPS corresponding to at least one first aspect of the at least one aspect.

17. The method of claim 12, wherein determining the current orientation based on the at least one SPS includes:

determining the current orientation based on at least one of a size of the SPS relative to the associated current pixel set, a position of the SPS relative to the associated current pixel set, a shape of the SPS, and an edge of the SPS.

18. The method of claim 12, wherein determining the current orientation based on the at least one SPS includes:

comparing at least one of a third SPS of a first current pixel set and a fourth SPS of a second current pixel set.

19. The method of claim 12, wherein determining the current orientation based on the at least one SPS includes:

comparing at least one of a first current orientation corresponding to a first current image and a second current orientation corresponding to a second current image.

20. The method of claim 12, wherein determining at least one position change of the UAV by comparing one or more of the target orientation and the current orientation includes:

determining at least one of a position change of the subject, heading change of the subject, and a velocity change of the subject.