System And Method For Determining The Active Talkers In A Video Conference

• US 20110093273A1
• Filed: 10/16/2009
• Published: 04/21/2011
• Est. Priority Date: 10/16/2009
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Lee, Bowon, Mukherjee, Debargha
• Granted Patent: US 9,154,730 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/440245 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

The present invention describes a method of determining the active talker for display on a video conferencing system, including the steps of: for each participant, capturing audio data using an audio capture sensor and video data using a video capture sensor; determining the probability of active speech (p_A, p_B . . . p_N), where the probability of active speech is a function of the probability of soft voice detection captured by the audio capture sensor and the probability of lip motion detection captured by the video capture sensor; and automatically displaying at least the participant that has the highest probability of active speech.

Claims

1. A method of determining the active talker for display on a video conferencing system, comprising the steps of:

for each participant, capturing audio data using an audio capture sensor and video data using a video capture sensor;

determining the probability of active speech (p_A, p_B . . . p_N), where the probability of active speech is a function of the probability of soft voice detection captured by the audio capture sensor and the probability of lip motion detection captured by the video capture sensor; and

automatically displaying at least the participant that has the highest probability of active speech.

2. The method recited in claim 1 further including the step of denoising the probability of active speech by applying a smoothing filter, wherein the smoothed probability is represented by p*=(p_A*, p_B* . . . p_N*),

3. The method recited in claim 2 wherein the smoothing filter is a nonlinear filter.

4. The method recited in claim 2 wherein the smoothing filter is a linear filter.

5. The method recited in claim 2 further including the step of applying a median filter to the output of the smoothing filter.

6. The method recited in claim 2, further including the step of determining the maximum value of the smoothed probability values (p_A*, p_B*, . . . p_N*),

7. The method recited in claim 6, further including the step of comparing the maximum value of the smoothed probability value to a threshold value.

8. The method recited in claim 7, wherein if the maximum value of the smoothed probability value is greater than the threshold value, it is assumed that the participant with the maximum value of the smoothed probability value is the active talker.

9. The method recited in claim 8, wherein if the maximum value of the smoothed probability value is less than the threshold value, it is assumed that there is no active talker and that the state diagram is in the silent state.

10. The method recited in claim 2, further including the step of ranking the smoothed probability values (p_A*, p_B*, . . . p_N*),

11. The method recited in claim 10, further including the step of comparing the lowest ranked smoothed probability value to a threshold value.

12. The method recited in claim 11, wherein if the lowest ranked smoothed probability value is greater than the threshold value, it is assumed that the lowest ranked participant is the active talker.

13. The method recited in claim 12, wherein if the lowest ranked smoothed probability value is less than the threshold value, it is assumed that there is no active talker and that the state diagram is in the silent state.

14. The method recited in claim 2, further including the step of determining the probabilities of p*, given all the possible states Pr (p*|s) where s is all of the possible 2N+1 states.

15. The method recited in claim 14, further including the step of finding the optimal sequence of states using dynamic programming.

16. The method recited in claim 15 wherein a truncated Viterbi algorithm is used.

17. The method recited in claim 16 wherein the state transition probabilities are modulated based on the duration in the state.

18. A computer readable storage medium having computer-readable program instructions stored thereon for causing a computer system to perform a method of providing feedback to a participant in a video conference, the method comprising the steps of:

determining the probability of active speech (p_A, p_B . . . p_N), where the probability of active speech is a function of the probability of soft voice detection captured by the audio capture sensor and the probability of lip motion detection captured by the video capture sensor; and

automatically displaying at least the participant that has the highest probability of active speech.

19. An apparatus for providing feedback to a participant in a video conference, the apparatus comprising:

a computer readable storage medium having computer-readable program instructions stored thereon for causing a computer system to perform a method of providing feedback to a participant in a video conference, the method comprising the steps of;

determining the probability of active speech (p_A, p_B . . . p_N), where the probability of active speech is a function of the probability of soft voice detection captured by the audio capture sensor and the probability of lip motion detection captured by the video capture sensor; and

automatically displaying at least the participant that has the highest probability of active speech.