System and method for supporting smooth target following

• US 10,976,753 B2
• Filed: 11/12/2018
• Issued: 04/13/2021
• Est. Priority Date: 09/15/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Zhao, Cong, Pang, Minjian, Li, Rui, Zhou, You, Liu, Zhe, Zhou, Guyue
• Primary Examiner(s): Zhou, Zhihan
• Application Number:
  US16/188,144
• Publication Number:
  US 20190082088A1
• Time in Patent Office: 883 Days
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2101/31 for surveillance
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/104 using satellite radio beacon positioning systems, e.g. GPS
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/106 Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
• CPC Class Codes: G05D 1/12 Target-seeking control
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/685 : performed by mechanical compensation
• CPC Class Codes: H04N 23/695 : Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects

Abstract

A method for supporting target tracking includes obtaining a feature model for a target that represents imagery characteristics of the target, extracting one or more features from one or more images captured by an imaging device carried by a movable object, and applying the feature model on the one or more features to determine similarity between the one or more features and the feature model.

Claims

1. A method for supporting target tracking, comprising:

obtaining a feature model for a target, wherein the feature model represents imagery characteristics of the target;

extracting one or more features from one or more images captured by an imaging device carried by a movable object; and

applying the feature model on the one or more features to determine similarity between the one or more features and the feature model,

wherein the movable object moves with respect to the target such that the one or more images have a different perspective projection from the feature model and, before applying the feature model on the one or more features, the method further comprises;

compensating at least one of the feature model and the one or more features for the different perspective projection.

2. The method of claim 1, further comprising:

acquiring the target based on an initialization image captured by the imaging device.

3. The method of claim 2, further comprising:

receiving a selection of a point in the initialization image;

providing a plurality of object proposal candidates according to the selected point, each of the object proposal candidates being represented by a bounding box; and

receiving a selection of one of the object proposal candidates as an object proposal for the target.

4. The method of claim 3, further comprising:

determining the feature model based on the object proposal.

5. The method of claim 1, wherein:

the feature model is represented in spatial domain and/or frequency domain.

6. The method of claim 1, wherein:

the one or more features are extracted from a searching window in the one or more images.

7. The method of claim 1, further comprising:

calculating correlation between the feature model and each of the one or more features; and

/or

matching the one or more features with the feature model.

8. The method of claim 1, further comprising:

obtaining a projective transformation of the target in the one or more images based on state information of the imaging device.

9. The method of claim 8, wherein:

the state information of the imaging device is determined based on position and attitude information of the movable object and attitude information of the imaging device.

10. The method of claim 9, wherein:

the position and attitude information of the movable object is received from a control module of the movable object, and the attitude information of the imaging device is received from a payload stabilization control module, wherein the payload stabilization control module controls a stabilization system, which stabilizes the imaging device on the movable object.

11. The method of claim 8, wherein compensating at least one of the feature model and the one or more features for the different perspective projection further comprises:

applying a perspective transformation on the feature model to compensate for the projective transformation of the target.

12. The method of claim 8, wherein compensating at least one of the feature model and the one or more features for the different perspective projection further comprises:

applying a reverse perspective transformation on the one or more images to compensate for the projective transformation of the target.

13. The method of claim 1, further comprising:

updating the feature model based on matching of the one or more features with the feature model, wherein the updated feature model is adapted to be applied to a subsequent image.

14. A method for supporting target tracking, comprising:

obtaining a feature model for a target, wherein the feature model represents imagery characteristics of the target;

extracting one or more features from one or more images captured by an imaging device carried by a movable object;

applying the feature model on the one or more features to determine similarity between the one or more features and the feature model;

obtaining a projective transformation of the target in the one or more images based on state information of the imaging device; and

applying a reverse perspective transformation on the one or more images to compensate for the projective transformation of the target,

wherein;

the reverse perspective transformation is determined based on a pitch angle of the image device.

15. A system for supporting target tracking, comprising:

one or more microprocessors; and

a controller running on the one or more microprocessors, wherein the controller operates to;

obtain a feature model for a target, wherein the feature model represents imagery characteristics of the target;

extract one or more features from one or more images captured by an imaging device carried by a movable object; and

apply the feature model on the one or more features to determine similarity between the one or more features and the feature model,

wherein the movable object moves with respect to the target such that the one or more images have a different perspective projection from the feature model and, before applying the feature model on the one or more features, the controller further operates to;

compensate at least one of the feature model and the one or more features for the different perspective projection.

16. The system of claim 15, wherein:

the controller further operates to acquire the target based on an initialization image captured by the imaging device.

17. The system of claim 16, wherein the controller further operates to

receive a selection of a point in the initialization image;

provide a plurality of object proposal candidates according to the selected point, each of the object proposal candidates being represented by a bounding box; and

receive a selection of one of the object proposal candidates as an object proposal for the target.

18. The system of claim 17, wherein:

the controller further operates to determine the feature model based on the object proposal.

19. The system of claim 15, wherein:

the one or more features are extracted from a searching window in the one or more images.