Cascade recognition for personal tracking via unmanned aerial vehicle (UAV)

• US 9,710,709 B1
• Filed: 03/09/2015
• Issued: 07/18/2017
• Est. Priority Date: 03/07/2014
• Status: Active Grant

• Current Assignee: Draganfly Innovations Inc. (Frontier Energy Corp.)
• Original Assignee: Trace Live Network Inc.
• Inventors: Clarke, Jamie, McDermott, Craig, Racicot, Erika, Readwin, Paul, Beard, Paul, Chell, Cameron
• Primary Examiner(s): Jones, Heather
• Application Number:
  US14/642,370
• Time in Patent Office: 862 Days
• Field of Search: None
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/10 Rotorcrafts
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/00 UAVs specially adapted for particular uses or applications
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps
• CPC Class Codes: H04N 5/44 Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Systems and methods for tracking a subject using an unmanned aerial vehicle (UAV) are disclosed. An unmanned aerial vehicle (UAV) includes an onboard camera to capture/stream multiple images. An onboard visual recognition module isolates image elements by analyzing the captured images, and determines whether the image elements correspond or do not correspond to the subject. Current image elements corresponding to a subject are stored in a positive database, and non-corresponding current image elements are stored in a negative database. An onboard subject tracking module defines the chosen subject, determines attributes of the subject based on comparisons of image elements, and follows the subject by directing the attitude control system to adjust the velocity and orientation of the UAV based on the determined attributes.

Claims

1. A system for acquiring and following a subject with at least one unmanned aerial vehicle (UAV) the UAV having at least one rotor configured to rotate at a rotor speed, the system comprising:

at least one camera fixed to the UAV, the at least one camera configured to capture at least one image;

at least one attitude control system configured to adjust a rotor speed of the at least one rotor;

at least one data storage unit fixed to the UAV;

a plurality of databases stored in the data storage unit, including at least a first database of at least one first image element corresponding to the subject and a second database of at least one second image element not corresponding to the subject;

at least one visual recognition module coupled to the at least one camera and to the plurality of databases, the at least one visual recognition module configured to;

extract at least one current image element from the at least one image;

determine whether the at least one current image element is a first current image element or a second current image element;

store the at least one first current image element in the first database; and

store the at least one second current image element in the second database; and

at least one subject tracking module coupled to the at least one visual recognition module and to the at least one attitude control system, the at least one subject tracking module configured to;

define the subject by analyzing at least one reference image captured by the at least one camera;

determine at least one attribute of the subject by performing at least one comparison of at least one current image element with at least one of a first image element and a second image element; and

direct the attitude control system of the UAV to adjust at least one rotor speed of the UAV based on the at least one attribute; and

at least one transmitter configured to transmit the at least one image via wireless link.

2. The system of claim 1, wherein the at least one subject tracking module is configured to

designate at least one image as a reference image;

designate at least one first current image element of the reference image as a first reference image element;

designate at least one second current image element of the reference image as a second reference image element;

determine at least one reference relationship between at least two of a first reference image element and a second reference image element;

populate at least one of the first database and the third database with the at least one first reference image element;

populate at least one of the second database and the third database with the at least one second reference image element; and

populate the third database with the at least one reference relationship.

3. The system of claim 2, wherein the at least one subject tracking module is configured to

determine at least one attribute of the subject by performing at least one comparison of at least one current image element with at least one of a first image element, a second image element, and a reference image element.

4. The system of claim 2, wherein the at least one reference relationship corresponds to at least one of a location, a dimension, an orientation, a time, an atmospheric condition, and a gesture of the subject.

5. The system of claim 4, wherein the at least one subject tracking module is configured to populate the third database with at least one procedure associated with a gesture of the subject.

6. The system of claim 5, wherein the at least one subject tracking module is configured to direct the at least one attitude control system of the UAV to adjust at least one rotor speed of the at least one UAV based on the at least one procedure.

7. The system of claim 1, wherein the at least one first image element corresponds to at least one of a hue of the subject, a color scheme of the subject, a texture of the subject, a shape of the subject, a face of the subject, a facial feature of the subject, and a body part of the subject.

8. The system of claim 1, wherein the at least one second image element corresponds to at least one of a non-subject person, a manmade structure, and a natural feature.

9. The system of claim 1, wherein the at least one visual recognition module is configured to determine whether the at least one current image element corresponds to at least one of a foreground element, a background element, a hue, a color scheme, a texture, a face, a facial feature, a Haar-like feature, a body part, a shape, a manmade structure, a natural feature, and a landmark.

10. The system of claim 1, wherein the at least one visual recognition module is configured to determine whether the at least one current image element corresponds to the subject via at least one of a facial feature recognition algorithm, a facial recognition algorithm, a color pattern detection algorithm, and an infrared detection algorithm.

11. The system of claim 1, further comprising:

two or more UAVs, each UAV having

an airframe;

at least one rotor fixed to the airframe, each rotor configured to rotate at a rotor speed;

at least one camera fixed to the UAV, the at least one camera configured to capture at least one image;

an attitude control system configured to adjust a rotor speed of the at least one rotor; and

at least one transmitter configured to transmit the at least one image via wireless link.

12. The system of claim 11, wherein the data storage unit, the visual recognition module, and the subject tracking module are embodied in a ground-based station coupled to the at least one UAV via wireless link.

13. The system of claim 11, wherein each UAV of the two or more UAVs comprises:

a data storage unit of the at least one data storage unit;

a visual recognition module of the at least one visual recognition module; and

a subject tracking module of the at least one subject tracking module.

14. The system of claim 1, further comprising:

at least one display device wirelessly coupled to the at least one transmitter, the at least one display device configured to receive the at least one image and display the at least one image to a user.

15. The system of claim 14, wherein the at least one display device includes at least one of a pair of goggles configured to display the at least one image to the wearer, a smartphone, and a portable computing device.

16. An unmanned aerial vehicle (UAV), comprising:

an airframe;

at least one rotor fixed to the airframe, each rotor configured to rotate at a rotor speed;

at least one camera fixed to the UAV, the at least one camera configured to capture at least one image;

an attitude control system configured to adjust a rotor speed of the at least one rotor;

a data storage unit fixed to the UAV;

a plurality of databases stored in the data storage unit, including at least a first database of at least one first image element corresponding to the subject and a second database of at least one second image element not corresponding to the subject;

a visual recognition module coupled to the at least one camera and to the plurality of databases, the visual recognition module configured to;

extract at least one current image element from the at least one image;

determine whether the at least one current image element is a first current image element or a second current image element;

store the at least one first current image element in the first database; and

store the at least one second current image element in the second database; and

a subject tracking module coupled to the visual recognition module and to the attitude control system, the subject tracking module configured to;

define the subject by analyzing at least one reference image captured by the at least one camera;

determine at least one attribute of the subject by performing at least one comparison of at least two of a first image element, a second image element, and a current image element; and

direct the attitude control system of the UAV to adjust at least one rotor speed of the UAV based on the at least one attribute; and

at least one transmitter configured to transmit the at least one image via wireless link.

17. A method of acquiring and following a subject via at least one unmanned aerial vehicle, (UAV), comprising:

defining a subject by capturing at least one reference image via at least one camera of the at least one UAV;

capturing at least one first image via the at least one camera;

extracting at least one current image element from the at least one first image;

determining whether the at least one current image element is a first image element corresponding to the subject or a second image element not corresponding to the subject;

storing the at least one first image element in a first database;

storing the at least one second image element in the second database; and

a second database;

determining at least one attribute of the subject by performing a comparison of the at least one current image element to at least one of a first image element, a second image element, and a reference image element; and

adjusting at least one rotor speed of the at least one UAV based on at least the at least one attribute of the subject.

18. The method of claim 17, wherein defining a subject by capturing at least one reference image via at least one camera of the at least one UAV includes:

capturing at least one image including the subject;

designating the at least one image as a reference image;

extracting at least one reference image element from the at least one reference image;

determining whether the at least one reference image element is a first reference image element corresponding to the subject or a second reference image element not corresponding to the subject;

determining at least one reference relationship between at least two of a first reference image element and a second reference image element;

populating at least one of the first database and a third database with the at least one first reference image element;

populating at least one of the second database and the third database with the at least one second reference image element; and

populating the third database with the at least one reference relationship.

19. The method of claim 18, wherein the at least one reference relationship corresponds to at least one of a location, a dimension, an orientation, a time, and an atmospheric condition.

20. The method of claim 18, wherein populating the third database with the at least one reference relationship includes:

populating the third database with at least one procedure associated with a gesture of the subject.

21. The method of claim 20, wherein adjusting at least one rotor speed of the at least one UAV based on at least the at least one attribute of the subject includes:

adjusting at least one rotor speed of the at least one UAV based on the at least one procedure.

22. The method of claim 17, wherein determining whether the at least one current image element is a first image element corresponding to the subject or a second image element not corresponding to the subject includes:

comparing the at least one current image element to at least one of a first reference image element and a second reference image element.

23. The method of claim 17, wherein determining whether the at least one current image element is a first image element corresponding to the subject or a second image element not corresponding to the subject includes

determining whether the at least one current image element corresponds to at least one of a foreground element, a background element, a hue, a color scheme, a texture, a face, a facial feature, a Haar-like feature, a body part, a shape, a manmade structure, a natural feature, and a landmark.

24. The method of claim 17, wherein determining whether the at least one current image element is a first image element corresponding to the subject or a second image element not corresponding to the subject includes

determining whether the at least one current image element is a first image element corresponding to the subject or a second image element not corresponding to the subject via at least one of a facial feature recognition algorithm, a facial recognition algorithm, a color pattern detection algorithm, and an infrared detection algorithm.

25. The method of claim 17, wherein determining at least one attribute of the subject by performing a comparison of the at least one current image element to at least one of a first image element, a second image element, and a reference image element includes:

determining at least one of a location of the subject, a position of the subject, an orientation of the subject, a gesture of the subject, and a heading of the subject.

26. The method of claim 17, further comprising:

transmitting the at least one image to at least one non-subject user via wireless link.