System and method for tracking objects by fusing results of multiple sensing modalities

• US 6,499,025 B1
• Filed: 06/01/1999
• Issued: 12/24/2002
• Est. Priority Date: 06/01/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toyama, Kentaro, Horvitz, Eric J.
• Primary Examiner(s): Black, Thomas
• Assistant Examiner(s): Holmes, Michael B.
• Application Number:
  US09/323,724
• Time in Patent Office: 1,302 Days
• Field of Search: 706/25, 706/51, 706/52, 250/332, 342/64, 700/90
• US Class Current: 706/52
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

The present invention is embodied in a system and method for efficiently and efficiently performing automated vision tracking, such as tracking human head movement and facial movement. The system and method of the present invention fuses results of multiple sensing modalities to achieve robust digital vision tracking The system and method effectively fuses together the results of multiple vision processing modalities for performing tracking tasks in order to achieve robust vision tracking. The approach integrates reports from several distinct vision processing procedures in a probabilistically coherent manner by performing inferences about the location and/or motion of objects that considers both the individual reports about targets provided by visual processing modalities, as well as inferences about the context-sensitive accuracies of the reports. The context-sensitive accuracies are inferred by observing evidence with relevance to the reliabilities of the different methods.

Claims

1. A method for tracking objects comprising:

dynamically observing data with a plurality of modalities that sense characteristic results of the objects;

selectively combining the results of the plural modalities;

detecting changes in expected reliabilities of the plural modalities influenced by the observations; and

producing synthesized assessments of a probability distribution over a ground truth represented by an unknown state of the objects and desired inferences.

2. The method of claim 1, wherein the synthesized assessments are indicative of maximum likelihood positions of target objects.

3. The method of claim 1, wherein the plurality of modalities comprises a plurality of visual processing procedures.

4. The method of claim 3, wherein selectively combining the results is based on a consideration of relevant evidence extracted directly from reliability reports of the visual processing procedures.

5. The method of claim 4, further comprising using the relevant evidence to fuse together reliability reports of the visual processing procedures for producing synthesized assessments of the ground truth.

6. The method of claim 3, wherein selectively combining the results is based on a consideration of relevant evidence extracted from an operating context of the visual processing procedures on reliability reports of the visual processing procedures.

7. The method of claim 6, further comprising using the relevant evidence to fuse together reliability reports of the visual processing procedures and producing synthesized assessments of the ground truth.

8. The method of claim 1, further comprising training on known data to assess probabilities of effects of indicators on modality results.

9. The method of claim 1, wherein the results are biased based on changing information related to the modalities.

10. The method of claim 1, further comprising creating dependencies between actual characteristics of the object and the results obtained from the plural modalities and restructuring the dependencies with Bayesian models that identify new sets of dependencies.

11. The method of claim 3, further comprising training with data collected from a non-visual sensor that reports accurate characteristics of the object in order to enhance the assessments.

12. The method of claim 1, further comprising identifying conditional probabilities with a learning module.

13. The method of claim 1, further comprising using a Bayesian learning module to infer probabilistic dependency structure among variables and conditional probabilities characterizing the dependencies.

14. The method of claim 1, further comprising using a Bayesian learning model to infer hidden variables, dependencies, and conditional probabilities.

15. The method of claim 1, wherein the synthesized assessments are indicative of at least one of location and motion of the objects for tracking the objects.

16. A tracking device for tracking objects comprising:

a processor receiving original data associated with the objects and processing the data with a plurality of modalities to produce plural estimates and reliability indicators; and

a sensor fusion processor receiving the estimates and reliability indicators to selectively combine the estimates and produce synthesized assessments of the data influenced by the reliability indicators.

17. The tracking device of claim 16, wherein the plurality of modalities are a plurality of visual processing procedures.

18. The tracking device of claim 17, wherein the sensor fusion processor receives and uses relevant evidence extracted directly from reliability reports of the visual processing procedures to produce the synthesized assessments.

19. The tracking device of claim 18, wherein the sensor fusion processor uses the relevant evidence to fuse together reliability reports of the visual processing procedures to produce the synthesized assessments of a ground truth represented by an unknown state of the objects and desired inferences.

20. The tracking device of claim 17, further comprising a combine module capable of combining the estimates produced by the plural modalities based on relevant evidence extracted from an operating context of the visual processing procedures.

21. The tracking device of claim 16, further comprising a training module capable of being trained on known data to assess probabilities of effects of indicators on modality results.

22. The tracking device of claim 16, wherein the estimates are biased based on changing information related to the modalities.

23. The tracking device of claim 16, further comprising dependencies created between actual characteristics of the object and the results obtained from the plural modalities, wherein the sensor fusion processor is capable of restructuring the dependencies with Bayesian models that identify new sets of dependencies.

24. The tracking device of claim 23, wherein the sensor fusion processor is capable of training the model from data collected from a non-visual sensor that reports an accurate measure of object position.

25. The tracking device of claim 16, further comprising a Bayesian learning module capable of inferring probabilistic dependency structure among variables and conditional probabilities characterizing the dependencies.

26. The tracking device of claim 16, further comprising a Bayesian learning model capable of inferring at least one of hidden variables, dependencies and conditional probabilities.

27. The tracking device of claim 16, wherein the synthesized assessments are indicative of at least one of location and motion of the objects for tracking the objects.

28. A computer-readable medium having computer-executable components for tracking objects, comprising:

a modality component capable of processing original data associated with the objects with a plurality of modalities to produce plural estimates and reliability indicators; and

a sensor fusion component capable of selectively combining the estimates and using the reliability indicators to produce synthesized assessments of the data influenced by the reliability indicators.

29. The computer-readable medium of claim 28, wherein the plurality of modalities is a plurality of visual processing procedures.

30. The computer-readable medium of claim 29, wherein the sensor fusion component receives and uses relevant evidence extracted directly from reliability reports of the visual processing procedures to produce the synthesized assessments.

31. The computer-readable medium of claim 30, wherein the sensor fusion component uses the relevant evidence to fuse together reliability reports of the visual processing procedures to produce the synthesized assessments of a ground truth represented by an unknown state of the objects and desired inferences.

32. The computer-readable medium of claim 29, further comprising a component capable of combining the estimates produced by the plural modalities based on relevant evidence extracted from an operating context of the visual processing procedures.

33. The computer-readable medium of claim 28, further comprising a training component capable of being trained on known data to assess probabilities of effects of indicators on modality results.

34. The computer-readable medium of claim 28, wherein the estimates are biased based on changing information related to the modalities.

35. The computer-readable medium of claim 28, further comprising dependencies created between actual characteristics of the object and the results obtained from the plural modalities, wherein the sensor fusion component is capable of restructuring dependencies with Bayesian models that identify new sets of dependencies.

36. The computer-readable medium of claim 35, wherein the sensor fusion component is capable of training the model from data collected from a non-visual sensor that reports an accurate measure of object position.

37. The computer-readable medium of claim 28, further comprising a Bayesian learning module capable of inferring probabilistic dependency structure among variables and conditional probabilities characterizing the dependencies.

38. The computer-readable medium of claim 28, further comprising a Bayesian learning component model capable of inferring at least one of hidden variables, dependencies, and conditional probabilities.

39. The computer readable medium of claim 28, wherein the synthesized assessments are indicative of location of the objects tracked.

40. The computer readable medium of claim 28, wherein the synthesized assessments are indicative of motion of the objects tracked.