Semi-private communication in open environments

• US 10,726,861 B2
• Filed: 11/15/2010
• Issued: 07/28/2020
• Est. Priority Date: 11/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Flaks, Jason S., Bar-Zeev, Avi
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Tarko, Asmamaw G
• Application Number:
  US12/946,701
• Publication Number:
  US 20120120218A1
• Time in Patent Office: 3,543 Days
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/028 using properties of sound source
• CPC Class Codes: H04M 2203/1066 Game playing
• CPC Class Codes: H04M 2203/65 related to applications where calls are combined with other types of communication
• CPC Class Codes: H04R 2203/12 Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A system and method providing semi-private conversation using an area microphone between one local user in a group of local users and a remote user. The local and remote users may be in different physical environments, using devices coupled by a network. A conversational relationship is defined between a local user and a remote user. The local user'"'"'s voice is isolated from other voices in the environment, and transmitted to the remote user. Directional output technology may be used to direct the local user'"'"'s utterances to the remote user in the remote environment.

Claims

1. A method of providing a semi-private conversation between a local user and a remote user, comprising:

receiving by way of each of an array of plural microphones, both of a first voice output from a respective first local user and a second voice output from a respective second local user, said first and second local users being situated in a sound-sharing first physical environment in which the first and second local users are in a two-way sound cross-talk relationship with one another and in which the microphone array is also situated;

using the microphone array to localize a respective origin position in the first physical environment for each of the first and second received voice outputs, the respective origin positions corresponding to respective locations of the first and second local users in the first physical environment;

associating the localized voice outputs with the correspondingly located first and second local users;

using the microphone array to isolate the respective voice output of a selected one of the first and second local users from other sounds present in the sound-sharing first physical environment;

directing the isolated voice output of the selected local user to a selected third user in a second physical environment, where sounds in the first physical environment are not propagated to the second physical environment;

receiving isolated utterances from the third user in the second physical environment; and

routing the utterances to the selected local user in the first physical environment.

2. The method of claim 1 further including the step of tracking user locations in the first physical environment and focusing a sound isolating function of the microphone array on a tracked one of the tracked user locations.

3. The method of claim 2 further including tracking user location by detecting user location in a field of view of a depth image camera.

4. The method of claim 3 further including the step of recording a voice print of a tracked user and associating the voice print with a skeletal model of the tracked user.

5. The method of claim 1 further including the step of determining a conversational relationship between the selected local user and another user in the second physical environment.

6. The method of claim 5 wherein the conversational relationship comprises the selected local user and the other user in the second physical environment acting as teammates in a game.

7. The method of claim 1 wherein routing comprises providing isolated utterances from the third user to a directional output aimed at the selected local user.

8. The method of claim 1 wherein the step of localization includes combining spatial filtering with regularization on the received voice output to thereby provide at least two corresponding outputs.

9. A system including at least a motion capture subsystem and an audio input subsystem, the system comprising:

a motion capture device including at least one array of plural microphones arranged in a housing including the plural microphones, the housing positioned in a sound-sharing first physical environment and configured to receive speech input from any of a plurality of users in the first physical environment;

a processing device including instructions causing the processing device to;

use the microphone array to localize a source location within the first physical environment of a voice received via the microphones from a first user of the plurality of users in the first physical environment;

associate the voice of the localized source location with the first user;

use the microphone array to isolate utterances made in the localized source location by the voice that has been associated with the first user in the first environment; and

direct the isolated utterances of the first user in the first environment to a selected second user in a second environment, where sounds in the first physical environment are not propagated to the second physical environment; and

a directional audio output device coupled to the processing device and configured to direct audio output from a remote user specifically to the localized source location associated with the first user.

10. The system of claim 9 wherein the instructions further include ones causing a tracking of at least a first local user and a second local user in a first physical environment, each local user having a conversational relationship with a remote user in a second physical environment.

11. The system of claim 9 wherein the instructions further include ones causing a receiving of isolated utterances from the second user in the second physical environment and a routing of the utterances to the first user in the first physical environment.

12. The system of claim 9 wherein the directional audio output device comprises an array of ultrasonic speakers.

13. The system of claim 9 wherein the directional audio output device comprises at least a first movable speaker and a second movable speaker.

14. A method providing a semi-private communications between local users situated in a sound-sharing first physical environment and remote users situated in a second physical environment that is not in a sound-sharing relation with the first physical environment, the local users and remote users connected via a network, the method comprising:

tracking at least a first local user and a second local user in the first physical environment, each local user having a conversational relationship with a remote user in the second physical environment;

receiving voice input from at least the first local user and the second local user in the first physical environment via a corresponding one or more of multiple microphones in a microphone array situated in the sound-sharing first physical environment, each microphone in the array configured to receive and to participate in respective acoustic localization of respective speech input from a respective one of the first local user and the second local user;

using the microphone array of the first physical environment for associating an acoustically localized voice as being sourced from a respective one of the first local user and the second local user;

using the microphone array of the first physical environment for isolating utterances of the first local user;

routing isolated utterances of the first user in the first environment to a selected first remote user in the second environment;

using the microphone array of the first physical environment for isolating utterances of the second local user; and

routing the isolated utterances of the second user in the first environment to a selected second remote user in the second environment,

wherein the steps of isolating include combining spatial filtering with regularization on the input to provide an isolated output.

15. The method of claim 14 wherein the method further includes the step of tracking user locations in the physical environment and focusing a sound sensitivity area of the microphone array on the user location.

16. The method of claim 15 wherein routing comprises providing the isolated utterances from the second user to a directional output aimed at the first user.

17. The method of claim 16 further including the step of receiving isolated utterances from the first remote user in the second physical environment and routing the utterances to the first local user in the first physical environment.

18. The method of claim 17 wherein at least one of the first remote user and the second remote user utilizes a headset, whereby the headset utilizing user is not in an equal sound cross-talk relationship with other users of the same remote physical environment;

and

neither of at least first and second local users situated in the sound-sharing first physical environment utilizes a headset, whereby the first and second local users are in an equal sound cross-talk relationship with one another.