COMPUTER SYSTEM EMPLOYING SPEECH RECOGNITION FOR DETECTION OF NON-SPEECH AUDIO

• US 20150002611A1
• Filed: 06/27/2013
• Published: 01/01/2015
• Est. Priority Date: 06/27/2013
• Status: Active Grant

• Current Assignee: GOTO Group Incorporated (LogMeIn, Inc.)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Thapliyal, Ashish V., Alexandrov, Albert
• Granted Patent: US 9,595,271 B2
• Time in Patent Office: Days
• US Class Current: 348/14.08
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G10L 25/84 for discriminating voice from noise
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04M 2203/2027 Live party detection
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A computer system executing a computer audio application such as video conferencing applies audio detection and speech recognition to an input audio stream to generate respective audio detection and speech recognition signals. A function is applied to the audio detection and speech recognition signals to generate a non-speech audio detection signal identifying presence of non-speech audio in the input audio stream when the audio detection signal is asserted and the speech recognition signal is not asserted. A control or indication action is performed in the computer system based on assertion of the non-speech audio detection signal.

Claims

1. A method of operating a computer system, comprising:

applying audio detection and speech recognition to an input audio stream to generate respective audio detection and speech recognition signals;

applying a function to the audio detection and speech recognition signals to generate a non-speech audio detection signal identifying presence of non-speech audio in the input audio stream when the audio detection signal is asserted and the speech recognition signal is not asserted; and

performing a control or indication action in the computer system based on assertion of the non-speech audio detection signal.

2. A method according to claim 1, wherein:

the computer system implements at least part of a video conferencing system having an associated graphical user interface;

the method is performed to discriminate between speech and non-speech audio in each of a plurality of audio streams from respective participants in a video conference; and

the control or indication action includes operating the graphical user interface to reflect the discriminating between speech and non-speech audio in the audio streams, including (a) providing a first graphical identification of one or more first participants as speaking participants based on a discrimination of speech in the respective audio streams, and (b) providing a second graphical identification of one or more second participants as non-speaking participants based on a discrimination of non-speech audio in the respective audio streams.

3. A method according to claim 2, further including, upon the discrimination of non-speech audio for the one or more second participants, generating a notification for one or more of the participants that the non-speech audio has been detected.

4. A method according to claim 3, wherein the notification is shown to the one or more second participants.

5. A method according to claim 3, further including muting or reducing amplitude of the audio streams in which non-speech audio is detected.

6. A method according to claim 2, wherein providing the graphical indications includes applying corresponding different treatments to respective camera viewing windows of the speakers and non-speakers, the treatments being selected from relative sizes, relative positions, and relative highlighting.

7. A method according to claim 2, wherein the video conferencing system includes a conference server coupled to a plurality of conference clients by a network, and wherein the discriminating between speech and non-speech audio is performed at the conference server for each of the audio streams as received from the conference clients.

8. A method according to claim 2, wherein the video conferencing system includes a conference server coupled to a plurality of conference clients by a network, and wherein the discriminating between speech and non-speech audio is performed at each conference client for the audio stream of a local participant.

9. A method according to claim 2, wherein the speech recognition provides a speech output and a separate confidence output indicating a level of confidence in accuracy of the speech output, and wherein a condition of no speech being recognized is based on the confidence output indicating a level of confidence below a predetermined threshold.

10. A method according to claim 1, wherein applying speech recognition is a secondary use of the speech recognition in the computer system, and wherein the computer system makes a distinct primary use of the speech recognition for obtaining speech content.

11. A method according to claim 10, wherein the primary use includes one or more of making a transcription of a speech-carrying session and identifying spoken control commands for the computer system.

12. A non-transitory computer-readable medium storing computer program instructions, the instructions being executable by a computer system to cause the computer system to perform a method including:

applying audio detection and speech recognition to an input audio stream to generate respective audio detection and speech recognition signals;

applying a function to the audio detection and speech recognition signals to generate a non-speech audio detection signal identifying presence of non-speech audio in the input audio stream when the audio detection signal is asserted and the speech recognition signal is not asserted; and

performing a control or indication action in the computer system based on assertion of the non-speech audio detection signal.

13. A non-transitory computer-readable medium according to claim 12, wherein:

the computer system is configured as a conference server computer operable as part of a video conferencing system including a plurality of conference client computers, the video conferencing system having a graphical user interface;

the method is performed to discriminate between speech and non-speech audio in each of a plurality of audio streams from respective participants in a video conference; and

the control or indication action includes operating the graphical user interface to reflect the discriminating between speech and non-speech audio in the audio streams, including (a) providing a first graphical identification of one or more first participants as speaking participants based on a discrimination of speech in the respective audio streams, and (b) providing a second graphical identification of one or more second participants as non-speaking participants based on a discrimination of non-speech audio in the respective audio streams.

14. A non-transitory computer-readable medium according to claim 13, wherein the method further includes, upon the discrimination of non-speech audio for the one or more second participants, generating a notification for one or more of the participants that the non-speech audio has been detected.

15. A non-transitory computer-readable medium according to claim 13, wherein providing the graphical indications includes applying corresponding different treatments to respective camera viewing windows of the speakers and non-speakers, the treatments being selected from relative sizes, relative positions, and relative highlighting.

16. A non-transitory computer-readable medium according to claim 12, wherein:

the computer system is configured as a conference client computer operable as part of a video conferencing system including a conference server computer and other conference client computers, the video conferencing system having a graphical user interface;

the audio stream is from a local participant in a video conference, the local participant being one of a plurality of participants using respective distinct conference client computers; and

the control or indication action includes operating the graphical user interface of the video conference to reflect a discriminating between speech and non-speech audio in the audio streams of the participants, including (a) providing a first graphical identification of one or more first participants as speaking participants based on a discrimination of speech in the respective audio streams, and (b) providing a second graphical identification of one or more second participants as non-speaking participants based on a discrimination of non-speech audio in the respective audio streams.

17. A non-transitory computer-readable medium according to claim 16, wherein the method further includes, upon the discrimination of non-speech audio for the local participant, sending an indication to the conference server that the non-speech audio has been detected.

18. A non-transitory computer-readable medium according to claim 16, wherein providing the graphical indications includes applying corresponding different treatments to respective camera viewing windows of the speakers and non-speakers, the treatments being selected from relative sizes, relative positions, and relative highlighting.

19. A non-transitory computer-readable medium according to claim 12, wherein applying speech recognition is a secondary use of the speech recognition in the computer system, and wherein the computer system makes a distinct primary use of the speech recognition for obtaining speech content.

20. A non-transitory computer-readable medium according to claim 19, wherein the primary use includes one or more of making a transcription of a speech-carrying session and identifying spoken control commands for the computer system.