Systems and methods for target tracking

• US 9,164,506 B1
• Filed: 08/28/2014
• Issued: 10/20/2015
• Est. Priority Date: 07/30/2014
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Zang, Bo
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): Nolan, Peter D
• Application Number:
  US14/471,954
• Time in Patent Office: 418 Days
• Field of Search: 701/2, 701/11, 348/113, 348/114, 348/144, 348/169, 348/170, 348/171, 348/172, 348/211.99
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/12 Target-seeking control
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft

Abstract

The present invention provides systems, methods, and devices related to target tracking by UAVs. The UAV may be configured to receive target information from a control terminal related to a target to be tracked by an imaging device coupled to the UAV. The target information may be used by the UAV to automatically track the target so as to maintain predetermined position and/or size of the target within one or more images captured by the imaging device. The control terminal may be configured to display images from the imaging device as well as allowing user input related to the target information.

Claims

1. A system for controlling an unmanned aerial vehicle (UAV), comprising:

one or more receivers, individually or collectively, configured to receive from a remote user (1) one or more navigation commands to move the UAV along a flight path, and (2) target information of a target to be tracked by an imaging device on the UAV; and

one or more processors, individually or collectively, configured to track the target according to the target information by automatically adjusting at least one of the UAV or the imaging device while the UAV moves along the flight path according to the one or more navigation commands from the remote user, wherein the one or more processors, individually or collectively, make a determination to adjust

     1) the UAV,

     2) the imaging device, or

     3) both the UAV and the imaging device, wherein said determination is dependent upon a) number of rotational axes of the imaging device and orientation of said rotational axes relative to the UAV;

b) a navigation path of the UAV;

or c) a maximum angular speed allowable for the UAV or the imaging device.

2. The system of claim 1, wherein the target information comprises initial target information comprising an initial position or an initial size of the target within an image captured by the imaging device.

3. The system of claim 1, wherein the target information comprises target type information comprising color, texture, or pattern information.

4. The system of claim 3, wherein tracking the target according to the target information further comprises identifying, based on the target type information, the target to track from within one or more images captured by the imaging device using an image recognition algorithm.

5. The system of claim 1, wherein the target information comprises expected target information comprising an expected position or an expected size of the target within an image captured by the imaging device.

6. The system of claim 1, wherein the UAV comprises a plurality of rotors configured to generate lift for the UAV.

7. The system of claim 1, wherein the imaging device is coupled to the UAV via a carrier configured to permit the imaging device to rotate around at least two axes relative to the UAV.

8. The system of claim 7, wherein the target information comprises expected target information and tracking the target according to the target information comprises:

determining current target information of the target based on one or more images captured by the imaging device;

detecting a deviation of the current target information from the expected target information; and

calculating an adjustment to the UAV, the carrier, or the imaging device so as to substantially correct the deviation.

9. The system of claim 8, wherein the deviation is related to a change in position of the target and the calculated adjustment is related to an angular velocity for the imaging device relative to a yaw axis or a pitch axis of the imaging device.

10. The system of claim 9, wherein the calculated adjustment is used to generate control signals for the carrier so as to cause the imaging device to move relative to the UAV.

11. The system of claim 8, wherein the one or more processors are configured to provide a warning signal if the adjustment falls outside a predetermined range corresponding to a predetermined range of control lever amount of a control system comprising a navigation control system for the UAV or a control system for the carrier.

12. The system of claim 1, wherein said determination is dependent upon a maximum angular speed allowable for the UAV or the imaging device.

13. A method for controlling an unmanned aerial vehicle (UAV), comprising:

receiving, from a remote user, user-specified target information of a target to be tracked by an imaging device on the UAV, the user-specified target information including a predetermined position or predetermined size of the target within an image captured by the imaging device, the imaging device coupled to the UAV via a carrier configured to permit the imaging device to move relative to the UAV;

detecting, by one or more processors onboard the UAV, a deviation from the predetermined position or the predetermined size of the target based on one or more images captured by the imaging device; and

automatically adjusting the UAV, the carrier, or the imaging device so as to substantially correct the detected deviation from the predetermined position or the predetermined size of the target, wherein the one or more processors, individually or collectively, make a determination to adjust

     1) the UAV,

     2) the imaging device, or

     3) both the UAV and the imaging device, wherein said determination is dependent upon a) a navigation path of the UAV;

or b) a maximum linear speed allowable for the UAV.

14. The method of claim 13, wherein said determination is dependent upon a maximum linear speed allowable for the UAV.

15. The method of claim 13, wherein the predetermined position of the target comprises an initial position or an expected position of the target and the predetermined size of the target comprises an initial size or an expected size of the target.

16. The method of claim 13, wherein detecting the deviation from the predetermined position or the predetermined size of the target comprises comparing a position or size of the target within the one or more images captured by the imaging device with the predetermined position or predetermined size, respectively.

17. The method of claim 13, wherein the adjustment involves adjusting a linear velocity of the UAV that is dependent upon the detected deviation from the predetermined size of the target within the image.

18. The method of claim 17, wherein the deviation is related to a change in size of the target and the adjustment is related to a linear velocity for the UAV or one or more parameters of the imaging device and wherein the one or more operational parameters of the imaging device comprise focal length, zoom level, imaging mode, image resolution, focus, depth of field, exposure, lens speed, or field of view.

19. The method of claim 13, wherein the UAV comprises a plurality of rotors configured to generate lift for the UAV.

20. The method of claim 13, further comprising providing a warning signal if the adjustment falls outside a predetermined range corresponding to a predetermined range of control lever amount of a control system comprising a navigation control system for the UAV or a control system for the carrier.

21. The method of claim 13, further comprising transmitting, in substantially real-time, images captured by the imaging device to a remote user device accessible to the remote user, the remote user device comprises a display for displaying the images captured by the imaging device and an input device for providing the target information comprising a touchscreen, joystick, keyboard, mouse, stylus, or a wearable device.

22. An apparatus for controlling an unmanned aerial vehicle (UAV), comprising:

a display configured to display one or more images captured by an imaging device coupled to the UAV;

an input device configured to receive a user selection of a target from within at least one of the one or more images being displayed on the display;

one or more processors, individually or collectively, configured to generate target information of the target based at least in part on the user selection of the target; and

a transmitter configured to provide the target information to the UAV so as to allow the UAV to autonomously track the target according to the target information, wherein the one or more processors, individually or collectively, make a determination to adjust

     1) the UAV,

     2) the imaging device, or

     3) both the UAV and the imaging device, wherein said determination is dependent upon a) number of rotational axes of the imaging device and orientation of said rotational axes relative to the UAV;

b) a navigation path of the UAV;

or c) a maximum angular speed allowable for the UAV or the imaging device.

23. The apparatus of claim 22, wherein the target information comprises initial target information, target type information, and/or expected target information.

24. The apparatus of claim 22, wherein the user selection of the target is achieved by a user selecting an area of the at least one of the one or more images being displayed on the display, the selected area corresponding to the target.

25. The apparatus of claim 22, wherein the one or more processors, individually or collectively, are further configured to display, on the display, the selected target with a selection indicator in response to the user selection of the target, the selection indicator indicating that the target has been selected by the user.

26. A system for controlling an unmanned aerial vehicle (UAV), comprising:

a receiver configured to receive, in substantially real-time, one or more images captured by an imaging device coupled to the UAV;

an input device configured to receive user-specified target information of a target including a predetermined position or a predetermined size of a target within an image from the one or more images;

a transmitter for providing the user-specified target information to the UAV, and the user'"'"'s input on allocation of control over a tracking process between the user and an onboard processing system of the UAV; and

a display configured to display the one or more images and a tracking indicator associated with the target within the one or more images, the tracking indicator indicating that the target is being tracked by the UAV in substantially real-time according to the user-specified target information.

27. The system of claim of 26, wherein the system is configured to dynamically adjust the allocation of the control over the tracking process between the user and the UAV based upon a surrounding of the UAV, speed, altitude, computer resource available on board or off board the UAV.

28. The system of claim 26, wherein the imaging device is coupled to the UAV via a carrier configured to permit the imaging device to rotate relative to the UAV along at least one axis and wherein the input device is further configured to receive one or more commands adapted to control a speed, position, orientation or attitude of the UAV and/or the carrier.

29. The system of claim 28, wherein the target is tracked by the imaging device according to the target information via automatic adjustment to at least one of the UAV, the carrier, or the imaging device.

30. The system of claim 26, wherein the predetermined position comprises an initial or expected position of the target and the predetermined size comprises an initial or expected size of the target.