Voice-body identity correlation

• US 8,781,156 B2
• Filed: 09/10/2012
• Issued: 07/15/2014
• Est. Priority Date: 01/25/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dernis, Mitchell, Leyvand, Tommer, Klein, Christian, Li, Jinyu
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US13/608,725
• Publication Number:
  US 20120327193A1
• Time in Patent Office: 673 Days
• Field of Search: 382/100, 382/115, 382/116, 382/117, 382/118, 382/181, 382/207, 704/200, 704/208, 704246-257, 704/260, 381/110, 379/67.1, 379/80, 379/88.01
• US Class Current: 382/100
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 17/10 Multimodal systems, i.e. based on the integration of multiple recognition engines or fusion of expert systems
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming

Abstract

A system and method are disclosed for tracking image and audio data over time to automatically identify a person based on a correlation of their voice with their body in a multi-user game or multimedia setting.

Claims

1. A method of identifying a correlation between a user and user voice, the method comprising the steps of:

(a) receiving images of one or more users taken over a plurality of time periods;

(b) receiving audio of voices for a plurality of time periods; and

(c) correlating a voice identified in said step (b) to a user of the one or more users based on a plurality of samplings of determined positions of the user in different images and determined source locations of the voice at different times, said step (c) comprising the step of performing a first sampling of the plurality of samplings to derive a scored confidence level of an association between the voice and a user, the scored confidence level obtained by examining one or more of the following factors;

i. how close the estimated position of the voice source is to the one or more users;

ii. the number of voices which are being heard;

iii. the closeness of the one or more users to an estimated source of the voice;

iv. whether the source of the voice is estimated to be centered within a field of view of the image or closer to edges of the field of view.

2. The method recited in claim 1, said step (c) comprising the step of a sampling of the plurality of samplings being formed by determining a location of the one or more users from examination of an image of the plurality of images and being formed by determining a location of the voice using an acoustic source localization technique.

3. The method recited in claim 1, said step (c) comprising the step of performing a first sampling of the plurality of samplings to obtain a confidence level in an association between the voice and the user, a confidence level above a predefined threshold resulting in the voice and the user being associated together in memory.

4. The method recited in claim 3, said step (c) comprising the step of the confidence level going up in subsequent samplings of the plurality of samplings if the subsequent samplings decrease the number of possible users to whom the voice can belong to.

5. The method recited in claim 4, further comprising the step of unambiguously correlating the voice to a user upon eliminating all other users to whom the voice could belong to in the plurality of samplings.

6. The method recited in claim 5, further comprising the step of performing additional samplings in the plurality of samplings after the correlation between the voice and user has been unambiguously associated together.

7. The method recited in claim 3, further comprising the step of removing the correlation if the additional samplings are unable to remove an ambiguity with respect to which user the voice belongs or if the additional samplings show the voice belongs to a second user of the one or more users.

8. The method recited in claim 1, said step (b) of receiving images of one or more users taken over a plurality of time periods comprising the step of measuring locations of at least portions of the users skeletal joints.

9. The method of claim 8, said step (c) of correlating a voice received in said step (b) to a user based in part on a determined source locations of the voice comprising the step of determining source locations of a voice by time difference of arrivals.

10. In a multi-user application where correlation of a voice to a user may require more than a single sampling of the voice and user location, a method of identifying a correlation between a user and user voice, the method comprising the steps of:

(a) receiving a plurality of images including one or more users;

(b) receiving audio for a plurality of time periods covering the plurality of images;

(c) performing an initial sampling examining a location of one or more users and a location of a voice, the initial sampling determining the voice is ambiguously correlated to a user of the one or more users; and

(d) performing subsequent samplings, the subsequent samplings removing ambiguity as to the correlation determined in said step (c) or confirming the ambiguity to remove the correlation determined in said step (c).

11. The method of claim 10, further comprising the step (d) of performing additional samplings examining locations of the one or more users and locations of the voice, the additional samplings confirming the correlation of the voice with the user or the additional samplings reducing a likelihood that the voice is correlated to the user.

12. The method of claim 11, wherein said steps (c) and (d) unambiguously correlate the voice to the user by identifying a correlation between a location of the user and a source of the voice and by eliminating all other users as possible sources of the voice.

13. The method of claim 10, wherein said step (c) further comprises the step of examining physical traits of the user to distinguish the user from other users and examining acoustic qualities of the voice to distinguish the voice from other voices.

14. The method of claim 10, further comprising the step of removing the correlation between the voice and user determined in said step (c) where the additional samplings in said (f) are unable to disambiguate the correlation between the voice and the user.

15. The method of claim 10, wherein said step (c) determines whether the voice is correlated to a user of the one or more users above a threshold confidence level by examining one or more of the following factors:

i. how close the estimated position of the voice source is to the one or more users;

ii. the number of voices which are being heard;

iii. the closeness of the one or more users to an estimated source of the voice;

iv. whether the source of the voice is estimated to be centered within a field of view of the image or closer to edges of the field of view.

16. A system for correlating a voice to user in a multi-user application, the system comprising:

an image camera component capable of providing a depth image of one or more users in a field of view of the image camera component;

a microphone array capable of receiving audio within range of the microphone array, the microphone array capable of localizing a source of a voice; and

a computing environment in communication with both the image capture component and microphone array, the computing environment capable of distinguishing between different users in the field of view, the computing environment further performing additional samplings of data from the image camera and data from the microphone array, the additional samplings confirming the correlation of the voice with the user or the additional samplings reducing a likelihood that the voice is correlated to the user, wherein the computing environment distinguishes between different users in the field of view by detecting locations of joints of the one or more users.

17. The system of claim 16, wherein the computing environment executes a gaming application involving the one or more users while performing the initial and additional samplings.

18. The system of claim 16, wherein the microphone array uses two microphones to localize a source of the voice by time difference of arrivals of the voice to the two microphones.