Modality fusion for object tracking with training system and method

• US 6,502,082 B1
• Filed: 10/12/1999
• Issued: 12/31/2002
• Est. Priority Date: 06/01/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toyama, Kentaro, Horvitz, Eric J.
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): Holmes, Michael B.
• Application Number:
  US09/416,189
• Time in Patent Office: 1,176 Days
• Field of Search: 342/64, 89/11.1, 706/23, 706/24, 706/16, 706/15, 706/20, 367/131
• US Class Current: 706/16
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

The present invention is embodied in a system and method for training a statistical model, such as a Bayesian network, to effectively capture probabilistic dependencies between the true state of an object being tracked and evidence from various tracking modalities to achieve robust digital vision tracking. The model can be trained and structured offline using data collected from sensors, that may be either vision or non-vision-based, in conjunction with position estimates from the sensing modalities. Both the individual reports about targets provided by visual processing modalities and inferences about the context-sensitive accuracies of the reports are considered. Dependencies among variables considered in the model can be restructured with Bayesian learning methods that revise the dependencies considered in the model. In use, the learned models for fusing multiple modalities of visual processing provide real-time position estimates by making inferences from reports from the modalities and by inferences about the context-specific reliabilities of one or more modalities.

Claims

1. In a tracking system having multiple modalities for tracking an object with changing states, a method for training the system comprising:

automatically constructing a set of training cases;

converting the cases into a probabilistic model;

continually inputting a current state of the object being tracked and corresponding values of variables considered in the probabilistic model; and

inferring an object position estimate using each modality.

2. The method for training of claim 1 wherein constructing the set of training cases comprises:

providing object ground truth data from at least one sensor;

providing position estimates produced by each modality;

computing reliability indicators for each modality; and

computing a ground-truth reliability measurement for each modality.

3. The method of claim 1 wherein a structure of the probabilistic model is automatically inferred from a set of possible models using the current state of the object being tracked and the corresponding values of the variables.

4. The method of claim 1 wherein a structure of the probabilistic model is automatically inferred using the set of training cases.

5. The method of claim 1 wherein a structure of the probabilistic model is computed using object ground truth data, position estimates produced by each modality, reliability indicators for each modality, and ground-truth reliability for each position estimate.

6. The method of claim 3 wherein the structure of the probabilistic model inferred is the model most likely to produce position data approximating the current state of the object being tracked.

7. The method of claim 1 further comprising automatically learning model parameters by using training cases and model variables to compute prior and conditional probability densities of variables considered in a structured probabilistic model.

8. The method of claim 7 wherein the probability densities are used to automatically produce a multi-modality tracking model for accurately approximating the current state of the object being tracked.

9. The method of claim 1 wherein the variables are automatically selected using the current state of the object being tracked and position estimates produced by the multiple modalities.

10. The method of claim 1 wherein fusing the multiple modalities is based on an inferred reliability of position estimates produced by each of the multiple modalities.

11. A system for tracking objects comprising:

a processor receiving state data associated with the objects and processing the data with a plurality of modalities to produce plural position estimates and estimate reliability results indicators;

a training module for training the system with predefined data to assess probabilities of effects of the indicators on the results; and

a sensor fusion processor that receives the estimates and reliability results indicators to selectively combine the results and estimates to produce synthesized assessments of the data influenced by the reliability results indicators.

12. The system of claim 11 wherein the training module has an automatically created training data set comprising object ground truth data, position estimates produced by each modality, the reliability results indicators, and ground-truth reliability measurements.

13. The system of claim 11 wherein the training module is capable of automatically inferring a probabilistic dependency structure among variables in a probabilistic network model.

14. The system of claim 11 wherein the training module is capable of automatically learning prior and conditional probabilities for variables in a probabilistic network model.

15. The system of claim 11 wherein the training module is capable of automatically inferring a probabilistic dependency structure among variables in a probabilistic network model, and probability densities characterizing the dependencies.

16. The system of claim 11 wherein the training module is capable of using probabilistic learning methods to infer hidden variables, dependencies, and probability densities of variables in a probabilistic network model.

17. A method for structuring a probabilistic network for tracking moving objects comprising:

constructing a set of training data representing properties of the moving objects;

selecting variables of the network;

automatically inferring the network structure from the training data; and

automatically learning parameters of the network variables from the set of training data.

18. The method of claim 17 wherein the network is capable of being trained using the set of training data to assess the probabilistic reliability of one or more vision modalities.

19. The method of claim 17 wherein constructing the set of training data comprises using a sensor to provide object ground truth data which is compared to position estimates generated by one or more vision modalities to compute reliability measurements and reliability indicators.

20. The method of claim 17 wherein probabilistic learning methods are used to automatically infer visible and hidden variables.

21. The method of claim 17 wherein automatically inferring the network structure comprises using the training data to infer a probabilistic dependency structure among variables in the network model by automatically inferring a model which is most probable to accurately track the moving objects.

22. A computer-readable medium having computer-executable modules for structuring and training probabilistic network models for tracking objects, comprising:

a data collection processor that collects data representing properties of the objects;

a variable selection processor that selects variables for the probabilistic network models; and

.

a model structure inference processor that infers the structure of the probabilistic network models.

23. The computer-readable medium of claim 22 wherein the data collection processor automatically constructs a set of training data comprising object ground truth positions, position estimates generated by one or more vision modalities, and modality position estimate reliability measurements and indicators.

24. The computer-readable medium of claim 22 wherein the variable selection processor automatically infers visible and hidden variables from a training data set generated by the data collection processor.

25. The computer-readable medium of claim 22 wherein the model structure inference processor automatically infers the network structure which is most probable to accurately track the moving objects using a training data set generated by the data collection processor to infer a probabilistic dependency structure among variables in the network model.

26. The computer-readable medium of claim 22 further comprising a model parameter processor that computes prior and conditional probability densities of variables considered in a structured probabilistic model.

27. The computer-readable medium of claim 22 wherein the network model is dynamically updated as new position data becomes available.

28. A method for training a manually created probabilistic model for tracking an object comprising:

automatically collecting data detailing the current state of the object;

automatically determining a measure of confidence in the manually created model using the data;

automatically learning model parameters from the data;

automatically determining new dependencies among variables considered in the probabilistic model based upon the measure of confidence and the model parameters;

automatically training the probabilistic model using the learned model parameters and the new variable dependencies.

29. The method for training of claim 28 wherein collecting data comprises:

providing object ground truth data from at least one sensor;

providing position estimates produced by at least one modality;

computing reliability indicators for each modality; and

computing a ground-truth reliability measurement for each modality.

30. The method of claim 28 wherein training the probabilistic model automatically produces a model which is most likely to accurately approximate a current state of the object being tracked.

31. The method of claim 28 wherein learning model parameters comprises computing prior and conditional probability densities of variables considered in the model.

32. The method of claim 31 wherein the probability densities are used to automatically produce a multi-modality tracking model for accurately approximating a current state of the object being tracked.

33. The method of claim 29 further comprising fusing the at least one modality based on an inferred reliability of position estimates produced by each of the modalities.