Apparatus and method for object tracking

• US 9,613,273 B2
• Filed: 05/19/2015
• Issued: 04/04/2017
• Est. Priority Date: 05/19/2015
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation), University of Technology at Sydney
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation), University of Technology at Sydney
• Inventors: Chen, Zhe, Wang, Chaohui, Hong, Zhibin, Mei, Xue, Prokhorov, Danil, Tao, Dacheng
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US14/716,661
• Publication Number:
  US 20160342837A1
• Time in Patent Office: 686 Days
• Field of Search: 382103, 382107
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

An object tracking device includes a short-term processing portion and a long short-term processing portion that are implemented by circuitry and work in a collaborative manner to track an object. The short-term processing portion includes a filter that tracks the object based on short-term memory and spatiotemporal consistency. The long short-term processing portion performs key-point matching-tracking and estimation based on a key-point database in order to track the object. A controller determines an output of the object tracking device based on the processing conducted by the short-term and long short-term processing portions of the tracking device, respectively.

Claims

1. An object tracking device comprising:

a first tracker implemented by circuitry, the circuitry configured to;

determine and extract features, of a candidate image patch from an input image frame,

evaluate the candidate image patch and cyclically shifted image patches of the candidate image patch, to determine a location of a tracked object;

resize and extract features from a predetermined number of image patches surrounding the determined location to obtain a first instance of the tracked object;

a second tracker implemented by circuitry, the circuitry configured to;

detect a plurality of key-points from the input image frame,

classify, based on a key-point database, each detected key-point as one of a matched target key-point, a matched background key-point, and an unmatched key-point,

compute, based on consecutive input image frames, an active set of key-points,

estimate, based on the computed active set of key-points and the matched target key-points, a set of inlier key-points included in a target bounding box that corresponds to a second instance of the tracked object; and

a controller implemented by circuitry, the circuitry being configured to generate a tracked object based on a comparison of the first instance of the tracked object and the second instance of the tracked object.

2. The object tracking device of claim 1, wherein the determined location of the tracked object has a highest filtering score, and the first instance of the tracked object is included in an image patch having a highest filter response.

3. The object tracking device of claim 2, wherein the circuitry is further configured to:

update the filter coefficients and an object template in an interpolating fashion at a predetermined update rate.

4. The object tracking device of claim 1, wherein the extracted features are based on a histogram of oriented gradient descriptors and color attributes, and the detected key-points are characterized by scale-invariant-feature-transform descriptors.

5. The object tracking device of claim 1, wherein the circuitry of the controller is configured to classify the detected key-point as a matched target key-point based on a matching confidence of the detected key-point and a first neighbor of the detected keypoint that belongs to the key-point database being above a first threshold.

6. The object tracking device of claim 5, wherein the circuitry of the controller is further configured to classify the detected key-point as a matched target key-point based on a ratio of distance parameter being below a second threshold, the ratio of distance parameter being a ratio of the Euclidean distance between the detected key-point and the first nearest neighbor to the Euclidean distance between the detected key-point and the second nearest neighbor.

7. The object tracking device of claim 5, wherein the matching confidence is a cosine similarity between the detected key-point and the first nearest neighbor of the detected key-point.

8. The object tracking device of claim 1, wherein the active set of key-points includes key-points whose displacement between the consecutive input image frames is less than a predetermined threshold.

9. The object tracking device of claim 1, wherein the circuitry of the controller is further configured to

compute an intersection-over-union (IOU) of the first instance of the tracked object and the second instance of the tracked object;

and

determine whether the first instance of the tracked object and the second instance of the tracked object are consistent based on the computed IOU and a predetermined consistency threshold.

10. The object tracking device of claim 1, wherein the circuitry of the controller is further configured to:

update the active set of key-points for each input image frame by determining a redundant set of key-points within the active set of key-points, the redundant set of key-points being computed based on quantization ID'"'"'s of the key-points; and

update the key-point database based on a number of inlier key-points being greater than a predetermined threshold and a number of matched background key-points lying within the target bounding box being zero.

11. The object tracking device of claim 10, wherein a timespan corresponding to an amount of time the key-points are maintained in the key-point database is exponentially distributed based on a relative strength parameter of the key-point database, a period of a forgetting parameter, and a scale of a timespan parameter.

21. The object tracking device of claim 1, wherein each image patch of the predetermined number of image patches surrounding the determined location has a unique size.

12. A method of object tracking comprising:

determining, and extracting features of a candidate image patch, from an input image frame;

evaluating by circuitry, the candidate image patch and cyclically shifted image patches of the candidate image patch, to determine a location of a tracked object;

resizing, and extracting features, from a predetermined number of image patches surrounding the determined location to obtain a first instance of the tracked object;

detecting a plurality of key-points from the input image frame;

classifying, based on a key-point database, each detected key-point as one of a matched target key-point, a matched background key-point, and an unmatched key-point,

computing by circuitry, based on consecutive input image frames, an active set of key-points;

estimating, based on the computed active set of key-points and the matched target key-points, a set of inlier key-points included in a target bounding box that corresponds to a second instance of the tracked object; and

generating by circuitry, a tracked object based on a comparison of the first instance of the tracked object and the second instance of the tracked object.

13. The method of object tracking of claim 12, wherein the determined location of the tracked object has a highest filtering score, and the first instance of the tracked object is included in an image patch having a highest filter response.

14. The method of object tracking of claim 12, further comprising:

updating the filter coefficients and an object template in an interpolating fashion at a predetermined update rate.

15. The method of object tracking of claim 12, wherein the circuitry is configured to classify the detected key-point as a matched target key-point based on a matching confidence of the detected key-point and a first neighbor of the detected keypoint that belongs to the key-point database being above a first threshold.

16. The method of object tracking of claim 15, wherein the circuitry is further configured to classify the detected key-point as a matched target key-point based on a ratio of distance parameter being below a second threshold, the ratio of distance parameter being a ratio of the Euclidean distance between the detected key-point and the first nearest neighbor to the Euclidean distance between the detected key-point and the second nearest neighbor.

17. The method of object tracking of claim 12, further comprising:

computing by circuitry, an intersection-over-union (IOU) of the first instance of the tracked object and the second instance of the tracked object; and

determining whether the first instance of the tracked object and the second instance of the tracked object are consistent based on the computed IOU and a predetermined consistency threshold.

18. A non-transitory computer readable medium having stored thereon a program that when executed by a computer causes the computer to execute a method to track an object, the method comprising:

determining, and extracting features of a candidate image patch, from an input image frame;

evaluating the candidate image patch and cyclically shifted image patches of the candidate image patch, to determine a location of a tracked object;

resizing and extracting features, from a predetermined number of image patches surrounding the determined location to obtain a first instance of the tracked object;

detecting a plurality of key-points from the input image frame;

classifying, based on a key-point database, each detected key-point as one of a matched target key-point, a matched background key-point, and an unmatched key-point;

computing based on consecutive input image frames, an active set of key-points;

estimating, based on the computed active set of key-points and the matched target key-points, a set of inlier key-points included in a target bounding box that corresponds to a second instance of the tracked object; and

generating a tracked object based on a comparison of the first instance of the tracked object and the second instance of the tracked object.

19. The non-transitory computer readable medium of claim 18, wherein the determined location of the tracked object has a highest filtering score, and the first instance of the tracked object is included in a first image patch having a highest filter response.

20. The non-transitory computer readable medium of claim 18, the method further comprising:

computing an intersection-over-union (IOU) of the first instance of the tracked object and the second instance of the tracked object; and

determining whether the first instance of the tracked object and the second instance of the tracked object are consistent based on the computed IOU and a predetermined consistency threshold.