Object tracking by hierarchical association of detection responses

• US 8,705,793 B2
• Filed: 10/08/2010
• Issued: 04/22/2014
• Est. Priority Date: 10/08/2009
• Status: Active Grant

• Current Assignee: University of Southern California
• Original Assignee: University of Southern California
• Inventors: Nevatia, Ramakant, Huang, Chang, Wu, Bo
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US12/901,316
• Publication Number:
  US 20110085702A1
• Time in Patent Office: 1,292 Days
• Field of Search: 382/100, 382/103, 382/226, 382/227
• US Class Current: 382/103
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Systems, methods, and computer readable storage media are described that can provide a multi-level hierarchical framework to progressively associate detection responses, in which different methods and models are adopted to improve tracking robustness. A modified transition matrix for the Hungarian algorithm can be used to solve the association problem that considers not only initialization, termination and transition of tracklets but also false alarm hypotheses. A Bayesian inference approach can be used to automatically estimate a scene structure model as the high-level knowledge for the long-range trajectory association.

Claims

1. An image processing system configured to receive a plurality of input images and track objects within image frames, the system comprising:

a hardware processing system configured to;

perform a first level of association between detection responses in image frames, wherein the detection responses include a vector having position, size, and frame index components, including forming one or more tracklets between pairs of detection responses having suitable link probabilities;

receive the one or more tracklets formed as a result of the first level of association, and perform a second level of association, including producing a tracklet association between tracklets;

perform a third level of association, including estimating a scene structure model from the second level of association; and

provide as an output a trajectory set of the tracklets.

2. The system of claim 1, wherein for the first level of association, the hardware processing system is configured to calculate the link probabilities as products of affinities.

3. The system of claim 2, wherein the affinities comprise a position affinity.

4. The system of claim 2, wherein the affinities comprise a size affinity.

5. The system of claim 2, wherein the affinities comprise an appearance affinity.

6. The system of claim 1, wherein for the first level of association, the hardware processing system is configured to link two responses in response to an affinity between the two responses being greater than a first threshold.

7. The system of claim 1, wherein for the first level of association, the hardware processing system is configured to link two responses in response to an affinity between the two responses being greater than an affinity of conflicting pairs.

8. The system of claim 1, wherein the second level of association includes an iterative process of multiple rounds, and wherein for each round the hardware processing system is configured to receive as an input a tracklet generated in the previous round and in response produce a further association.

9. The system of claim 8, wherein the objective of the first round of association of the second level is formulated as a MAP problem.

10. The system of claim 8, wherein for the second level of association, the hardware processing system is further configured to transform a maximum a posteriori (MAP) problem into an assignment problem in a way that distinguishes between true trajectories and false alarm trajectories using a Hungarian algorithm.

11. An article of manufacture comprising:

a non-transitory machine readable storage medium; and

executable program instructions embodied in the machine readable storage medium that when executed by a processor of a programmable computing device configures the programmable computing device to control an image processing system receiving a plurality of images, to perform functions for object tracking within image frames, including instructions to;

perform a first level of association between detection responses in image frames, including forming one or more tracklets between pairs of detection responses having suitable link probabilities, wherein the detection responses include a vector having position, size, and frame index components;

receive the one or more tracklets formed as a result of the first level of association, and perform a second level of association, including producing a tracklet association between tracklets;

perform a third level of association, including estimating a scene structure model from the second level of association; and

provide as an output a trajectory set of the tracklets.

12. The article of manufacture of claim 11, wherein the executable program instructions further comprise instruction to:

for the second level of association, transform a maximum a posteriori (MAP) problem into an assignment problem in a way that distinguishes between true trajectories and false alarm trajectories using a Hungarian algorithm.

13. A method of object tracking within image frames, the method comprising:

using a processing system, performing a first level of association between detection responses in image frames, wherein the detection responses include a vector having position, size, and frame index components, the first level of association including forming one or more tracklets between pairs of detection responses having suitable link probabilities, wherein the first level of association produces a tracklet set, and the second level of association comprises, for each tracklet in the in the tracklet set, obtaining a motion model and an appearance model;

using the processing system, receiving the one or more tracklets formed as a result of the first level of association, performing a second level of association, including producing a tracklet association between tracklets, wherein the second level of association comprises obtaining an optimal tracklet association set and a corresponding tracklet set;

using the processing system, performing a third level of association, wherein performing a third level of association includes estimating a scene structure model from the second level of association; and

provide as an output a trajectory set of the tracklets.

14. The method of claim 13, wherein performing a first level of association comprises calculating the link probabilities as products of position, size, and appearance affinities.

15. The method of claim 13, wherein for the first level of association, the two responses are linked in response to the affinity being greater than a first threshold.

16. The method of claim 13, wherein the second level of association includes an iterative process of multiple rounds, wherein each round receives as an input a tracklet generated in the previous round and in response produces further association.

17. The method of claim 13, wherein the objective of the first round of association of the second level is formulated as a maximum a posteriori (MAP) problem.

18. The method of claim 17, further comprising transforming the MAP problem into an assignment problem in a way that distinguishes between true trajectories and false alarm trajectories using a Hungarian algorithm.

19. The method of claim 13, wherein for the first level of association, the one or more tracklets have the same initialization and termination probabilities as to entries and exits in respective frames.

20. The method of claim 13, wherein for the third level of association, an entry map and an exit map are produced from the second level of association and used to specify the initialization and termination of each tracklet in an image scene, and a scene occluder map is produced from the second level of association and used to revise the link probabilities.