Visually intelligent camera device with peripheral control outputs

• US 10,375,359 B1
• Filed: 11/12/2015
• Issued: 08/06/2019
• Est. Priority Date: 11/12/2014
• Status: Active Grant

• Current Assignee: Draganfly Innovations Inc. (Frontier Energy Corp.)
• Original Assignee: Trace Live Network Inc.
• Inventors: Beard, Paul, McDermott, Craig
• Primary Examiner(s): Findley, Christopher G
• Application Number:
  US14/939,369
• Time in Patent Office: 1,363 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/102 specially adapted for vertical take-off of aircraft
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A system and related method for tracking a moving subject via a ground-based or airborne peripheral device includes plugging an intelligent camera device to the unmanned vehicle, establishing a peripheral link by which the intelligent camera device can assume control of the unmanned vehicle'"'"'s control systems. Based on analysis of images captured by the intelligent camera device, in addition to position data associated with the unmanned vehicle or with the subject, the intelligent camera device may autonomously maneuver the unmanned vehicle to track or follow the subject while maintaining the moving subject in a consistent framing orientation.

Claims

1. An intelligent camera system, comprising:

a housing connectable to a peripheral vehicle by a peripheral port;

at least one visual image sensor situated at least partially within the housing and configured to capture at least one visual image;

at least one wireless transceiver configured to;

establish a wireless link to at least one mobile communications device carried by a subject;

and

receive, via the wireless link, position data corresponding to the subject from the mobile communications device;

at least one processor coupled to the at least one visual image sensor and to the wireless transceiver, the processor configured to;

select the at least one subject based on the at least one visual image;

establish a first data link to the peripheral vehicle via the at least one peripheral port;

identify at least one control system of the peripheral vehicle via the first data link;

generate at least one peripheral instruction based on one or more of the at least one visual image and the position data;

and

adjust at least one of a location of the peripheral vehicle, a heading of the peripheral vehicle, a velocity of the peripheral vehicle, and an orientation of the peripheral vehicle by forwarding the peripheral instruction to the at least one control system via the first data link;

and

at least one memory operatively coupled to the at least one processor and configured to store the at least one visual image.

2. The intelligent camera system of claim 1, wherein:

the at least one processor is configured to select the at least one subject based on the at least one visual image and the position data.

3. The intelligent camera system of claim 2, wherein the at least one processor is further configured to:

(1) receive local position data from a position sensor of the peripheral vehicle, via the first data link;

and

(2) determine a relative position of the subject based on at least one of the at least one first visual image, the local position data, and the received position data.

4. The intelligent camera system of claim 3, wherein the at least one position sensor includes at least one of a GNSS receiver, an accelerometer, a compass, and a magnetometer.

5. The intelligent camera system of claim 1, wherein the at least one processor is configured to determine at least one of a velocity of the subject and a heading of the subject based on one or more of the at least one visual image and the position data.

6. The intelligent camera system of claim 5, wherein the at least one processor is configured to determine at least one future position of the subject based on one or more of the at least one visual image and the position data.

7. The intelligent camera system of claim 5, wherein the at least one processor is configured to determine at least one of a velocity of the subject and a heading of the subject by comparing the at least one visual image to at least one reference image associated with the subject.

8. The intelligent camera system of claim 1, wherein:

the peripheral vehicle includes an unmanned aerial vehicle (UAV) having at least one rotor configured to rotate at a rotor speed, the at least one rotor articulated by at least one motor coupled to the at least one control system; and

the peripheral instruction includes adjusting at least one of a position of the peripheral vehicle, an altitude of the peripheral vehicle, a rotational orientation of the peripheral vehicle, a heading of the peripheral vehicle, and a velocity of the peripheral vehicle by adjusting at least one rotor speed.

9. The intelligent camera system of claim 1, wherein the peripheral vehicle includes an unmanned ground vehicle.

10. A method for tracking a subject via a peripheral device, the method comprising:

establishing a first data link by removably attaching an apparatus to the peripheral device, the apparatus including at least one visual image sensor and at least one processor coupled to the at least one visual image sensor;

identifying at least one control system of the peripheral device via the at least one processor;

configuring the at least one control system to receive at least one peripheral instruction from the at least one processor;

capturing at least one first visual image via the at least one visual image sensor;

selecting a subject to track based on the at least one first visual image;

establishing a wireless link to at least one mobile communication device carried by the subject via a wireless transceiver of the apparatus;

receiving position data corresponding to the subject from a position sensor of the mobile communication device via the wireless link;

determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject based on the at least one first visual image and the received position data;

generating at least one peripheral instruction via the at least one processor based on at least one of the visual image and the received position data;

and

adjusting at least one of a location of the peripheral device, a heading of the peripheral device, a velocity of the peripheral device, and an orientation of the peripheral device by forwarding the at least one peripheral instruction to the at least one control system.

11. The method of claim 10, wherein establishing a first data link by removably attaching an apparatus to the peripheral device, the apparatus including at least one image sensor and at least one processor coupled to the at least one visual image sensor, includes:

establishing a first data link by removably attaching an apparatus to at least one of an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV), the apparatus including at least one visual image sensor and at least one processor coupled to the at least one visual image sensor.

12. The method of claim 11, wherein:

establishing a first data link by removably attaching an apparatus to at least one of an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV), the apparatus including at least one visual image sensor and at least one processor coupled to the at least one visual image sensor, includes

establishing a first data link by removably attaching an apparatus to at least one unmanned aerial vehicle (UAV), the at least one UAV including at least one rotor configured to rotate at a rotor speed and at least one motor coupled to, and configured to actuate, the at least one rotor, the apparatus including at least one image sensor and at least one processor coupled to the at least one image sensor;

and

adjusting at least one of a location of the peripheral device, a heading of the peripheral device, a velocity of the peripheral device, and an orientation of the peripheral device by forwarding the at least one peripheral instruction to the at least one control system includes adjusting at least one rotor speed by forwarding the at least one peripheral instruction to the at least one control system.

13. The method of claim 10, wherein determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject based on the at least one first visual image and the received position data includes:

receiving, via the at least one processor, local position data from a position sensor of the peripheral device via the first data link.

14. The method of claim 13, wherein determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject based on the at least one first visual image and the received position data includes:

determining, via the at least one processor, a relative position of the subject based on at least one of the at least one first visual image, the local position data, and the received position data.

15. The method of claim 10, wherein determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject based on the at least one first visual image and the received position data includes:

determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject by comparing the at least one first visual image to at least one reference image associated with the subject.

16. The method of claim 10, wherein:

determining, via the at least one processor, at least one of a heading of the subject and a velocity of the subject based on the at least one first visual image and the received position data includes determining, via the at least one processor, at least one future position of the subject based on the at least one first visual image and the received position data; and

generating the at least one peripheral instruction via the at least one processor based on at least one of the visual image and the received position data includes generating the at least one peripheral instruction via the at least one processor based on the at least one future position of the subject.