System and process for tracking an object state using a particle filter sensor fusion technique

• US 6,882,959 B2
• Filed: 05/02/2003
• Issued: 04/19/2005
• Est. Priority Date: 05/02/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Chen, Yunqiang
• Primary Examiner(s): Barlow, John
• Assistant Examiner(s): LE, JOHN H
• Application Number:
  US10/428,470
• Publication Number:
  US 20040220769A1
• Time in Patent Office: 718 Days
• Field of Search: 702/150, 702179-182, 702186-188, 706/15, 706/16, 706/20, 706/23, 706/24, 706/25, 706/51, 706/52, 342/64
• US Class Current: 702/179
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming

Abstract

A system and process for tracking an object state over time using particle filter sensor fusion and a plurality of logical sensor modules is presented. This new fusion framework combines both the bottom-up and top-down approaches to sensor fusion to probabilistically fuse multiple sensing modalities. At the lower level, individual vision and audio trackers can be designed to generate effective proposals for the fuser. At the higher level, the fuser performs reliable tracking by verifying hypotheses over multiple likelihood models from multiple cues. Different from the traditional fusion algorithms, the present framework is a closed-loop system where the fuser and trackers coordinate their tracking information. Furthermore, to handle non-stationary situations, the present framework evaluates the performance of the individual trackers and dynamically updates their object states. A real-time speaker tracking system based on the proposed framework is feasible by fusing object contour, color and sound source location.

Claims

1. A computer-implemented process for tracking an object state over time using particle filter sensor fusion and a plurality of logical sensor modules, comprising using a computer to perform the following process actions for each iteration of the tracking process:

inputting an object state estimate into a fuser module from each logical sensor module as determined by an object state tracker of that sensor module, wherein the object state estimates are in the form of a Gaussian distribution;

using the fuser module to combine the object state estimate distributions to form a proposal distribution, and then sampling the proposal distribution to produce a series of particles which are provided to each of the logical sensor modules;

using an object state verifier of each logical sensor module to estimate the likelihood of each provided particle;

inputting the likelihood estimates from each logical sensor into the fuser module and computing a combined likelihood model for the particles;

using the fuser module to compute a weight for each particle from the combined likelihood model, the proposal distribution, an object dynamics model which models the changes in the object state over time, and the weight associated with a corresponding particle in the last tracking iteration; and

using the fuser module to compute a final estimate of the object state for the current tracking iteration using the particles and the particle weights, wherein said final object state estimate takes the form of a distribution.

2. The process of claim 1, wherein the process action of combining the object state estimate distributions to form a proposal distribution comprises the action of weighting each object state estimate distribution using a reliability factor associated with the object state tracker providing the distribution.

3. The process of claim 2, wherein the process action of weighting each object state estimate distribution using a reliability factor associated with the object state tracker providing the distribution, comprises the actions of:

dynamically computing the reliability factor during each tracking iteration by quantifying the degree of similarity between the object state estimate distribution computed by a tracker and the final distribution computed for that iteration; and

employing the last computed reliability factor in the next iteration.

4. The process of claim 1, wherein the tracking process is used to track a speaker and the object state is the location of that speaker.

5. The process of claim 4, wherein the plurality of logical sensor modules comprises a vision-based object contour sensor which can track and verify the location of human heads within an image using head shape.

6. The process of claim 4, wherein the plurality of logical sensor modules comprises a vision-based object interior color sensor which can track and verify the location of human heads within an image using head color.

7. The process of claim 4, wherein the plurality of logical sensor modules comprises an audio-based sound source location sensor which can track and verify the location of a source of human speech using a microphone array.

8. The process of claim 1 further comprising a process action for dynamically adapting the object state tracker of each logical sensor module during each tracking iteration by providing each tracker with a revised object state estimate which is used in the next tracking iteration to compute an updated object state in lieu of using the object state computed by the tracker in the current tracking iteration.

9. The process of claim 8, wherein the process action of providing each tracker with a revised object state estimate, comprises the actions of:

dynamically computing a reliability factor which ranges between 0 and 1 for each object state tracker during each tracking iteration by quantifying the degree of similarity between the object state estimate distribution computed by a tracker and the final distribution computed for that iteration; and

computing the revised object state estimate as the sum of the object state estimate distribution computed by the tracker multiplied by the reliability factor and the final distribution multiplied by 1 minus the reliability factor.

10. A computer-readable medium having computer-executable instructions for performing the process actions recited in any one of claim 1, 2 or 8.

11. A two-level, closed-loop, particle filter sensor fusion system for tracking an object state over time, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

access an object state estimate from each of a plurality of logical sensors as determined by an object state tracker of the sensor,

combine the object state estimates to form a proposal distribution,

sample the proposal distribution to produce a series of particles which are provided to each of the logical sensors to estimate the likelihood of each particle using an object state verifier,

access the likelihood estimates from each logical sensor and compute a combined likelihood model for the particles,

compute a weight for each particle,

compute a final estimate of the object state for the current tracking iteration using the particles and the particle weights.

12. The system of claim 11, wherein the program module for computing a weight for each particle comprises computing the weight from the combined likelihood model, the proposal distribution, an object dynamics model which models the changes in the object state over time, and the weight associated with a corresponding particle in the last tracking iteration.

13. The system of claim 11, wherein the object state trackers employ a likelihood function that is different than the likelihood function used by the object state verifiers.

14. The system of claim 13, wherein the likelihood function employed by the object state trackers is less precise than the likelihood function used by the object state verifiers.

15. The system of claim 11, wherein the sensor fusion system is used to track a speaker and the object state is the location of that speaker.

16. The system of claim 15, wherein the plurality of logical sensors comprises a vision-based logical sensor which can track and verify the location of a speaker within an image of a scene containing the speaker.

17. The system of claim 15, wherein the plurality of logical sensors comprises an audio-based logical sensor which can track and verify the location of a speaker within a space using speech coming from the speaker.

18. The system of claim 15, wherein the plurality of logical sensors comprises at least two vision-based logical sensor each of which can track and verify the location of a speaker within an image of a scene containing the speaker using complementary visual cues.

19. The system of claim 15, wherein the plurality of logical sensors comprises both a vision-based logical sensor which can track and verify the location of a speaker within an image of a scene containing the speaker and an audio-based logical sensor which can track and verify the location of a speaker within a space using speech coming from the speaker.

20. The system of claim 11, wherein the object state tracker of each logical sensor employs a likelihood function that is the same as the likelihood function used by the object state verifier of that logical sensor.

21. A computer-readable medium having computer-executable instructions for tracking an object state over time using particle filter sensor fusion and a plurality of logical sensors, said computer-executable instructions comprising for each tracking iteration:

inputting an object state estimate from each logical sensor;

combining the object state estimates to form a proposal distribution;

sampling the proposal distribution to produce a series of particles;

using the logical sensors to estimate the likelihood of each particle;

inputting the likelihood estimates from each logical sensor;

computing a weight for each particle based in part on the likelihood estimates input from each logical sensor; and

computing a final estimate of the object state for the current tracking iteration using the particles and the particle weights.

22. The computer-readable medium of claim 21, wherein the instruction for inputting the likelihood estimates from each logical sensor comprises a sub-instruction for computing a combined likelihood model for the particles from the likelihood estimates input from each logical sensor, and wherein the instruction for computing a weight for each particle based in part on the likelihood estimates input from each logical sensor comprises a sub-instruction for employing the combined likelihood model to compute the particle weights.

23. A process for tracking an object state over time using particle filter sensor fusion and a plurality of logical sensor modules, comprising the following process actions for each iteration of the tracking process:

inputting an object state estimate into a fuser module from each logical sensor module as determined by an object state tracker of that sensor module;

using the fuser module to combine the object state estimates to form a proposal distribution, and then sampling the proposal distribution to produce a series of particles which are provided to each of the logical sensor modules;

using an object state verifier of each logical sensor module to estimate the likelihood of each provided particle;

inputting the likelihood estimates from each logical sensor into the fuser module and computing a combined likelihood model for the particles;

using the fuser module to compute a weight for each particle based in part on the combined likelihood model; and

using the fuser module to compute a final estimate of the object state for the current tracking iteration using the particles and the particle weights.

24. The process of claim 23 further comprising a process action for dynamically adapting the object state tracker of each logical sensor module during each tracking iteration by providing each tracker with a revised object state estimate which is used in the next tracking iteration to compute an updated object state in lieu of using the object state computed by the tracker in the current tracking iteration.

25. The process of claim 24, wherein the process action of providing each tracker with a revised object state estimate, comprises the actions of:

dynamically computing a reliability factor which ranges between 0 and 1 for each object state tracker during each tracking iteration by quantifying the degree of similarity between the object state estimate distribution computed by a tracker and the final distribution computed for that iteration; and

computing the revised object state estimate for each object state tracker using the reliability factor associated with that tracker.

26. The process of claim 25, wherein the process action of computing the revised object state estimate for each object state tracker using the reliability factor associated with that tracker, comprises the action of computing the revised object state estimate for each object state tracker as the sum of the object state estimate distribution computed by the tracker multiplied by the reliability factor and the final distribution multiplied by 1 minus the reliability factor.

27. A process for tracking an object state over time using particle filter sensor fusion and a plurality of logical sensor modules, comprising the following process actions for each iteration of the tracking process:

inputting an object state estimate from each logical sensor;

combining the object state estimates to form a proposal distribution;

sampling the proposal distribution to produce a series of particles;

using the logical sensors to estimate the likelihood of each particle;

computing a weight for each particle based in part on the likelihood estimates from each logical sensor; and

computing a final estimate of the object state for the current tracking iteration using the particles and the particle weights.

28. The process of claim 27, wherein the process action of computing a weight for each particle based in part on the likelihood estimates from each logical sensor, comprises the actions of:

computing a combined likelihood model for the particles using the likelihood estimates from each logical sensor; and

employing the combined likelihood model to compute the particle weights.

29. A two-level, closed-loop, particle filter sensor fusion system for tracking an object state over time, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

access an object state estimate from each of a plurality of logical sensors,

combine the object state estimates to form a proposal distribution,

sample the proposal distribution to produce a series of particles which are provided to each of the logical sensors to estimate the likelihood of each particle,

access the likelihood estimates from each logical sensor,

compute a weight for each particle base in part on the likelihood estimates,

compute a final estimate of the object state for the current tracking iteration using the particles and the particle weights.

30. The system of claim 29, wherein the program module for computing a weight for each particle comprises sub-modules for:

computing a combined likelihood model for the particles using the likelihood estimates from each logical sensor; and

employing the combined likelihood model to compute the particle weights.