Interest determination for auditory enhancement

• US 8,416,715 B2
• Filed: 06/15/2009
• Issued: 04/09/2013
• Est. Priority Date: 06/15/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rosenfeld, Daniel A., Liu, Zicheng, Cutler, Ross G., Chou, Philip A., Huitema, Christian, Quinn, Kori
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US12/484,906
• Publication Number:
  US 20100315482A1
• Time in Patent Office: 1,394 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-356, 709201-207, 709217-248, 37920201-20701
• US Class Current: 370/267
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04M 2201/20 Scanners
• CPC Class Codes: H04M 2203/2044 Group features, e.g. closed user group
• CPC Class Codes: H04M 2203/5018 Initiating a conference during a two-party conversation, i.e. three-party-service or three-way-call computer conference organisation arrangements H04L12/1818
• CPC Class Codes: H04M 2203/5072 Multiple active speakers conducting a computer conference, e.g. Admission detection H04L12/1822
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

Gaze tracking or other interest indications are used during a video conference to determine one or more audio sources that are of interest to one or more participants to the video conference, such as by determining a conversation from among multiple conversations that a subset of participants are participating in or listening to, for enhancing the audio experience of one or more of the participants.

Claims

1. Computer readable storage media having computer readable program code embodied therein, the computer-readable program code adapted to be executed to implement a method comprising:

connecting a computing device of a user to a plurality of participants participating in a video conference;

displaying video images of the plurality of participants on a graphic user interface on a display of the computing device of the user;

receiving user interest information determined by tracking a gaze of the user viewing the display to determine a video image of a particular participant of the plurality of participants that eyes of the user are focused on;

receiving participant interest information from the plurality of participants indicating which participants each of the plurality of participants are looking at;

determining from the user interest information and the participant interest information a subset of one or more participants of the plurality of participants for inclusion in a conversation group with the user, the conversation group comprising the subset of participants determined to be participating in or listening to a particular conversation of multiple conversations;

displaying a panoramic view of a table including the interest information and personal information for each of the plurality of participants on the display of the computing device of the user; and

increasing audio gain for audio received from the subset of participants included in the conversation group with the user for providing the audio with increased audio gain to the user.

2. The computer readable storage media according to claim 1, wherein the increasing the audio gain is carried by the computing device of the user.

3. The computer readable storage media according to claim 1, wherein the increasing the audio gain is carried out on a management computing device managing communication for the video conference prior to sending the audio with increased audio gain to the computing device of the user.

4. A method implemented by a user computing device having one or more processors executing instructions stored in computer-readable media, the method comprising:

displaying video images of a plurality of participants on a graphic user interface on a display, wherein the plurality of participants are displayed in a panorama view of the plurality of participants present at an actual conference table in an environment having multiple microphones present for the plurality of participants;

determining, for a first participant to a video conference, one or more second participants to a conversation of interest to the first participant from among the plurality of participants to the video conference by tracking a gaze of the first participant viewing the display to determine that eyes of the first participant are focused on a video image corresponding to one of the one or more second participants to a conversation of primary interest; and

augmenting audio received by the first participant from the one or more second participants to the conversation of interest to the first participant to increase audibility of the audio received from the one or more second participants relative to audio received from the other participants of the plurality of participants of the video conference by increasing a gain on one or more microphones closest to the one or more second participants out of the multiple microphones.

5. The method according to claim 4, further comprising:

equalizing the amplitude of audio received by the first participant from the plurality of participants of the video conference.

6. The method according to claim 4, wherein determining the one or more second participants to the conversation of interest to the first participant further comprises:

displaying video images of the plurality of participants participating in the video conference on the graphic user interface on the display of the user computing device; and

determining the one or more second participants to the conversation of interest from among the plurality of participants includes receiving a manual indication from the first participant using a pointing device on the graphic user interface of the user computing device indicating a selection of one of the one or more second participants.

7. The method according to claim 4, further comprising:

determining a particular video image displayed on the graphic user interface as being a focus of the gaze of the first participant for a predetermined length of time;

identifying a particular participant corresponding to the particular video image as being one of the one or more second participants to the conversation of interest to the first participant; and

providing a visual indication on the graphic user interface to the first participant that the particular participant corresponding to the particular video image has been identified as corresponding to one of the one or more second participants from which augmented audio is to be received.

8. The method according to claim 4, further comprising a management computing device that receives video and audio communications from each of the participants to the video conference for distribution to the other participants of the video conference, wherein the determining for the first participant to a video conference, one or more second participants to the conversation of interest to the first participant from among the plurality of participants to the video conference further comprises:

receiving interest information from each of the participants to the video conference at the management computing device, the interest information indicating one or more participants each of the participants is interested in receiving augmented audio from;

determining from the received interest information the one or more second participants to the conversation of interest to the first participant; and

enhancing the audio from the one or more second participants prior to transmitting the audio to the first participant.

9. The method according to claim 8, wherein the audio from the one or more second participants is enhanced by increasing a gain of the audio from the one or more second participants by a predetermined amount over a gain of audio received from the other participants prior to mixing the audio from the one or more second participants and the audio from the other participants in a mixer at the management computing device.

10. The method according to claim 8, further comprising:

determining from the received interest information a conversation from among multiple ongoing conversations that each of the plurality of participants is interested in; and

enhancing audio sent to each of the plurality of participants to the video conference based upon the received interest information by grouping each of the participants into one of a plurality of conversation groups based upon the interest information, wherein audio to and from members of a particular conversation group is augmented for other members of the particular conversation group, but not augmented for participants that are members of a different conversation group.

11. The method according to claim 4,

wherein a particular second participant is identified as being included in the conversation of interest to the first participant following a determination that the first participant is looking at the particular second participant, and

wherein another second participant is determined to be included in the conversation of interest to the first participant following a determination that the particular second participant is looking at the other second participant, or the other second participant is looking at the particular second participant.

12. The method according to claim 4, further comprising:

receiving an audio input from the first participant; and

augmenting the audio input from the first participant to the one or more second participants to the conversation of interest to the first participant to increase the audibility of the audio input from the first participant to the one or more second participants relative to audio received from the other participants of the plurality of participants of the video conference.

13. The method according to claim 4, wherein the augmenting the audio received by the first participant from the one or more second participants to the conversation of interest to the first participant to increase the audibility of the audio is accomplished at least in part by increasing a gain of the audio received from the one or more second participants by a predetermined amount over gain of audio received from the other participants of the video conference.

14. Computer readable storage media having computer readable program code embodied therein, the computer-readable program code adapted to be executed by one or more processors to implement the method of claim 4.

15. A computing device comprising:

a communication interface;

a display; and

one or more processors coupled to computer readable storage media storing instructions adapted to be executed by the one or more processors for configuring the computing device to;

connect, via the communication interface, to a plurality of participants for participating in a video conference;

display video images of the plurality of participants on a graphic user interface on the display;

receive user interest information identifying a particular participant as being of interest;

distinguish and highlight the particular participant of interest on the display; and

in response to the particular participant remaining of interest for more than a predetermined threshold period of time, augmenting audio received from the particular participant identified as being of interest.

16. The computing device according to claim 15, wherein the computing device is further configured to:

receive participant interest information from the plurality of participants;

group each of the participants into one of a plurality of conversation groups based upon the participant interest information and the user interest information, wherein audio to and from members of a particular conversation group of which a user of the computing device is a member is augmented for other members of the particular conversation group, but not augmented for participants that are members of a different conversation group.

17. The computing device according to claim 15, wherein the computing device is further configured to determine user interest information by tracking a gaze of a user of the computing device using one or more gaze tracking sensors.

18. The computing device according to claim 15, wherein the computing device is further configured to determine user interest information by receiving an input from a pointing device on the graphic user interface.

19. The computing device according to claim 15, wherein the computing device is further configured to augment the audio received from the particular participant by increasing a gain of the audio received from the particular participant by a predetermined amount over gain of audio received from the other participants of the video conference.