Multi-object tracking with a knowledge-based, autonomous adaptation of the tracking modeling level

• US 8,670,604 B2
• Filed: 11/22/2010
• Issued: 03/11/2014
• Est. Priority Date: 12/01/2009
• Status: Active Grant

• Current Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Original Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Inventors: Eggert, Julian, Zhang, Chen
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US12/951,127
• Publication Number:
  US 20110129119A1
• Time in Patent Office: 1,205 Days
• Field of Search: None
• US Class Current: 382/131
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/251 involving models

Abstract

The invention proposes a method for object and object configuration tracking based on sensory input data, the method comprising the steps of:

• • (1.1) Basic recruiting: Detecting interesting parts in sensory input data which are not yet covered by already tracked objects and incrementally initializing basic tracking models for these parts to continuously estimate their states,
  • (1.2) Tracking model complexity adjustment: Testing, during runtime more complex and more simple prediction and/or measurement models on the tracked objects, and
  • (1.3) Basic release: Releasing trackers from parts of the sensory data where the tracker prediction and measurement processes do not get sufficient sensory support for some time.

Claims

1. A method for tracking objects based on sensory input data (3) supplied from a stereo video camera (30, 31), the method comprising the following steps:

processing, via a processor comprising a long-term memory and a short-term memory database, the sensory input data (3) supplied using one or more tracking models (9), each tracking model comprising a tracker prediction and a measurement process,

deciding (16) whether the sensory input data (3) contain parts not yet covered by the tracking model (9), and in a positive case, initializing new tracking models,

releasing (29) a tracked object if the tracker prediction and the measurement process do not get sufficient sensory support for some time, and

adjusting an abstraction level of the tracking models used by evaluating performance of tracking models during run-time and using the tracking models showing an optimum performance according to a performance criterion, wherein the abstraction level of the tracking models is adjusted by scanning a tracking model graph from the long-term memory to select alternative tracking model candidates related to current ones in terms of graph connectivity, evaluating performance of the alternative tracking model candidates, and deciding whether to use one of the alternative tracking model candidates as a tracking model based on a comparison of results of the evaluating for each tracking model.

2. The method of claim 1, wherein the abstraction level is adjusted by combining independently tracked objects of the tracked objects into an object configuration that is then tracked as a single compound, wherein the object configuration comprises an object with multiple movements grouped together.

3. The method of claim 1, comprising the steps of:

detecting parts in sensory input data (3) which are not yet covered by already tracked objects and incrementally initializing basic tracking models (17) for these parts to continuously estimate states of not-yet tracked objects,

testing, during runtime, more complex and more simple prediction and measurement models on the tracked objects, and

releasing trackers from parts of the sensory input data (3) where the tracker prediction and measurement processes do not get sufficient sensory support for some time.

4. The method according to claim 3, wherein the testing comprises:

choosing new prediction and measurement model candidates from a suitable, situation-dependent set of models, and determine if an alternative prediction and measurement model will be used for further tracking of an object or object configuration.

5. The method of claim 4, where the choosing of the situation-dependent set of prediction and/or confirmation models is supported by a graph-like structure comprising the steps of:

storing a basic graph structure of tracking models that describes the tracking models and neighborhood relationships between the tracking models, wherein the graph structure has been created by design or acquired by a learning system,

during object tracking, selecting prediction and measurement models from the tracking models available in the graph structure, each tracked object retaining a memory of which model(s) from the graph structure it is using, and

during tracking model complexity adjustment, using the neighborhood relationships of the tracking models and a position in the graph structure of a tracked object to systematically reduce a set of model candidates that are inspected.

6. The method of claim 5, where the graph structure from the storing gets adjusted, improved and extended by evaluating statistics of the tracked objects and supervised data from a tutoring instance.

7. The method of claim 5, where the graph structure of prediction and measurement models is hierarchical and has a form of directed acyclical graphs.

8. The method of claim 5, where parts of the graph structure represent compositionality information, wherein several subordinate trackers can be merged by virtue of a superordinate tracker that constrains or influences the subordinate trackers.

9. The method of claim 5, where parts of the graph structure represent information on how to combine different subordinate trackers from different sensory modalities.

10. The method of claim 9, where a constraint or influence of a superordinate tracker on subordinate trackers involves a state transformation step to adjust to a state description of the subordinate trackers.

11. The method according to claim 1, where the tracking models are implemented by probabilistic methods.

12. The method according to claim 1, wherein the sensory input data are data from a stereo video camera.

13. A computer software program product stored on a non-transitory computer readable medium, implementing a method according to claim 1 when run on a computing device.

14. A computer-based tracking system, provided with streaming sensors and having a computing unit designed for performing a method according to claim 1.

15. The method of claim 9, wherein the different sensory modalities comprise combined objects from different cameras by exploiting their sensory causes as a single physical object, represented by a superordinate tracker.

16. The method of claim 10, wherein the superordinate tracker operates in 3D and constrains the subordinate tracker working in 2D, thereby making a 3D to 2D coordinate transformation step necessary when passing information from the superordinate tracker operating in 3D to the subordinate tracker working in 2D.

17. The method of claim 11, wherein the probabilistic methods comprise one or more of Dynamic Bayesian Networks, particle filters, and multiple switching dynamic models for mixing and evaluation of prediction models.

18. The method of claim 14, wherein the streaming sensors comprise video cameras.