SPEAKER DETECTION AND TRACKING USING AUDIOVISUAL DATA

• US 20100194881A1
• Filed: 03/31/2010
• Published: 08/05/2010
• Est. Priority Date: 06/27/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jojic, Nebojsa, Attias, Hagai, Beal, Matthew James
• Granted Patent: US 8,842,177 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 2218/22 Source localisation; Inverse modelling
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Object tracking includes an audio model that receives at least two audio input signals and a video model that receives a video input. The audio model and the video model employ probabilistic generative models which are combined to facilitate object tracking. Expectation maximization can be employed to modify trainable parameters of the audio model and the video model.

Claims

1. One or more processor-accessible storage media comprising processor-executable instructions for object tracking that, when executed, direct a device to perform actions comprising:

receiving at least two audio input signals associated to an object;

receiving a video input signal associated to the object;

modeling a location of the object based at least in part on the two audio input signals and the video input signal; and

calculating an error in modeling the location of the object via a zero mean Gaussian distribution associated with a precision matrix, wherein the zero mean Gaussian distribution associated with the precision matrix is further based at least in part on a precision matrix of an approximation error modeled by a zero mean Gaussian.

2. The one or more processor-accessible storage media of claim 1, wherein the zero mean Gaussian distribution associated with the precision matrix is further based at least in part on a product of a vertical position of the object and a difference in vertical position of a first audio input device and a second audio input device.

3. The one or more processor-accessible storage media of claim 1, wherein the zero mean Gaussian distribution associated with the precision matrix is further based at least in part on a product of a horizontal position of the object and a difference in horizontal position of a first audio input device and a second audio input device.

4. The one or more processor-accessible storage media of claim 1, wherein the location of the object is based at least in part on a probabilistic generative model of the video input signals associated to the object.

5. The one or more processor-accessible storage media of claim 1, wherein the location of the object is based at least in part on a probabilistic generative model of the two audio input signals associated to the object.

6. The one or more processor-accessible storage media of claim 5, wherein the audio model employs a hidden Markov model.

7. The one or more processor-accessible storage media of claim 1, wherein the location of the object is based at least in part on a probabilistic generative model of the two audio input signals and a probabilistic generative model of the video signal.

8. An object tracker system, comprising:

a processor that executes the following computer executable components stored on a computer readable medium;

an audio model component that models an original audio signal of an object;

a video model component that models a location of the object; and

an audio video tracker component that models the location of the object based, at least in part on the audio model and the video model, wherein the audio video tracker provides an output associated with the location of the object based at least in part on a linear mapping that approximates the location of the object, wherein error in approximating the location of the object is modeled by a zero mean Gaussian distribution associated with a precision matrix, and wherein the zero mean Gaussian distribution associated with the precision matrix is based at least in part on a product of a horizontal position of the object and a difference in horizontal position of a first audio input device and a second audio input device.

9. The system of claim 8, wherein the zero mean Gaussian distribution associated with the precision matrix is based further at least in part on a product of a vertical position of the object and a difference in vertical position of the first audio input device and the second audio input device.

10. The system of claim 8, wherein the zero mean Gaussian distribution associated with the precision matrix is based further on a precision matrix of an approximation error modeled by a zero mean Gaussian.

11. The system of claim 8, further comprising a video input device.

12. The system of claim 11, wherein the video input device comprises a camera.

13. The system of claim 8, further comprising at least one audio input device.

14. The system of claim 13, wherein the audio input device includes at least one of a microphone, a telephone, or a speaker phone.

15. The system of claim 8, wherein the original audio signal of the object, a time delay between at least two audio input signals, and a variability component of the original audio signal comprise unobserved variables of the audio model and wherein the audio model further includes trainable parameters.

16. The system of claim 15, wherein the audio video tracker component employs an expectation maximization algorithm to modify the trainable parameters of the audio model.

17. The system of claim 8, wherein the location of the object, an original image of the object, and a variability component of the original image comprise unobserved variables of the video model;

and wherein the video model further includes trainable parameters.

18. The system of claim 17, wherein the audio video tracker component employs an expectation maximization algorithm to modify the trainable parameters of the video model.

19. One or more processor-accessible storage media comprising processor-executable instructions for object tracking that, when executed, direct a device to perform actions comprising:

updating a posterior distribution over unobserved variables of a probabilistic generative audio model and a probabilistic generative video model;

providing the probabilistic generative audio model with trainable parameters;

providing the probabilistic generative video model with trainable parameters;

updating trainable parameters of the probabilistic generative audio model and the probabilistic generative video model;

determining a location of an object based at least in part by combining the probabilistic generative audio model and the probabilistic generative video model using a probabilistic generative model; and

providing an output associated with the location of the object.

20. The one or more processor-accessible storage media of claim 19, wherein the error in approximating the location of the object is modeled by a zero mean Gaussian distribution associated with a precision matrix, and wherein the zero mean Gaussian distribution associated with the precision matrix is based on, at least in part:

a product of a vertical position of the object and a difference in vertical position of a first audio input device and a second audio input device.