MULTI-OBJECT TRACKING WITH A KNOWLEDGE-BASED, AUTONOMOUS ADAPTATION OF THE TRACKING MODELING LEVEL

• US 20110129119A1
• Filed: 11/22/2010
• Published: 06/02/2011
• Est. Priority Date: 12/01/2009
• Status: Active Grant

• Current Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Original Assignee: Honda Research Institute Europe GmbH (Honda Motor Company)
• Inventors: ZHANG, Chen, EGGERT, Julian
• Granted Patent: US 8,670,604 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/251 involving models

Abstract

The invention proposes a method for object and object configuration tracking based on sensory input data, the method comprising the steps of:

(1.1) Basic recruiting: Detecting interesting parts in sensory input data which are not yet covered by already tracked objects and incrementally initializing basic tracking models for these parts to continuously estimate their states,

(1.2) Tracking model complexity adjustment: Testing, during runtime more complex and more simple prediction and/or measurement models on the tracked objects, and

(1.3) Basic release: Releasing trackers from parts of the sensory data where the tracker prediction and measurement processes do not get sufficient sensory support for some time.

Claims

1. A method for tracking objects based on sensory data (3) supplied from streaming sensors such as e.g. video camera(s) (30, 31), the method comprising the following steps:

processing the sensory input data (3) using one or more tracker models (9), each tracker model comprising a tracker prediction and a measurement process,

deciding (16) whether the sensory data (3) contain interesting parts not yet covered by a tracker model (9), and in the positive case, initializing new tracker objects,

releasing (29) a tracked object if the tracker prediction and measurement processes do not get sufficient sensory support for some time, and

adjusting the abstraction level of the tracking models used by testing more complex and simpler tracking models during run-time.

2. The method according to claim 1, wherein the abstraction level of the tracking models is adjusted by:

scanning a tracking model graph from a long-term memory (11) to select alternative tracking model candidates related to the current ones in terms of graph connectivity,

performance evaluation of the alternative tracking model candidates, and

deciding if one of the alternative models will be used to continue tracking a given object.

3. The method of claim 1, wherein the abstraction level is adjusted by combining independently tracked objects into an object configuration that is then tracked as a single compound.

12. The method according to claim 1, where the tracking models are implemented by means of probabilistic methods such as e.g. Dynamic Bayesian Networks, particle filters, and multiple switching dynamic models for the mixing and evaluation of prediction models.

13. The method according to claim 1, wherein the sensory input data are data from a stereo video camera.

14. A computer software program product, implementing a method according to claim 1 when run on a computing device.

15. A computer-based tracking system, provided with streaming sensors such as e.g. video cameras and having a computing unit designed for performing a method according to claim 1.

4. A method for object and object configuration tracking based on sensory input data (3), the method comprising the steps of:

(4.1) Basic recruiting (16);

Detecting interesting parts in sensory input data (3) which are not yet covered by already tracked objects and incrementally initializing basic tracking models (17) for these parts to continuously estimate the states of the not-yet tracked objects,

(4.2) Tracking model complexity adjustment (19);

Testing, during runtime, more complex and more simple prediction and/or measurement models on the tracked objects, and

(4.3) Basic release (29);

Releasing trackers from parts of the sensory data (3) where the tracker prediction and measurement processes do not get sufficient sensory support for some time.

5. The method according to claim 4, wherein step (4.2) comprises:

choosing new prediction and/or measurement model candidates from a suitable, situation-dependent set of models, the performance evaluation and comparison and the final decision if an alternative prediction and/or measurement model will be used for the further tracking of an object or object configuration.

6. The method of claim 5, where the choice of the situation-dependent set of prediction and/or confirmation models is supported by a graph-like structure comprising the steps of:

(6.1) Storing a basic graph structure of tracking models (comprising prediction and/or measurement models) that describes the models themselves and neighborhood relationships between the models. This graph structure may have been created by design or acquired by a learning system,

(6.2) During object tracking (step

     1), selecting prediction and/or measurement models from the models available in the graph (6.1);

each tracked object retaining a memory of which model(s) from the graph it is using (i.e., their position in the graph), and (6.3) During tracking model complexity adjustment (steps 1 and 4.2), using the neighborhood relationships of the tracking models and the position in the graph of a tracked object to systematically reduce the set of model candidates that are inspected.

7. The method of claim 6, where the graph from (6.1) gets adjusted, improved and extended by evaluating the statistics of the tracked objects and/or supervised data from a tutoring instance (man or machine).

8. The method of claim 6, where the graph structure of prediction and/or measurement models is hierarchical and has the form of directed acyclical graphs.

9. The method of claim 6, where parts of the graph structure represent compositionality information, meaning that several subordinate trackers can be merged by virtue of a superordinate (higher-level) tracker that constrains/influences the subordinate trackers.

10. The method of claim 6, where parts of the graph structure represent information on how to combine different subordinate trackers from different sensory modalities, e.g. combining objects from different cameras by exploiting the facts that their sensory cause is a single physical object, represented by a superordinate tracker

11. The method of claim 10, where the constraint/influence of a superordinate tracker on subordinate trackers involves a state transformation step to adjust to the state description of the subordinate trackers, e.g. as in the case of a superordinate tracker operating in 3D and that constrains a subordinate tracker working in 2D, making a 3D to 2D coordinate transformation step necessary when passing information from the 3D to the 2D tracker.