Methods and systems for participant sourcing indication in multi-party conferencing and for audio source discrimination

• US 20050076081A1
• Filed: 10/01/2003
• Published: 04/07/2005
• Est. Priority Date: 10/01/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Gupta, Anoop
• Granted Patent: US 7,916,848 B2
• Time in Patent Office: Days
• US Class Current: 709/204
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/4046 with distributed floor control
• CPC Class Codes: H04M 2201/38 Displays
• CPC Class Codes: H04M 2201/41 using speaker recognition speaker recognition per se G10L17/00
• CPC Class Codes: H04M 3/56 Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities video conference systems H04N7/15
• CPC Class Codes: H04Q 2213/13034 A/D conversion, code compression/expansion
• CPC Class Codes: H04Q 2213/13091 CLI, identification of calling line
• CPC Class Codes: H04Q 2213/1324 Conference call
• CPC Class Codes: H04Q 2213/13248 Multimedia
• CPC Class Codes: H04Q 2213/13333 Earth satellites
• CPC Class Codes: H04Q 2213/13337 Picturephone, videotelephony
• CPC Class Codes: H04Q 2213/13389 LAN, internet

Abstract

Indications of which participant is providing information during a multi-party conference. Each participant has equipment to display information being transferred during the conference. A sourcing signaler residing in the participant equipment provides a signal that indicates the identity of its participant when this participant is providing information to the conference. The source indicators of the other participant equipment receive the signal and cause a UI to indicate that the participant identified by the received signal is providing information (e.g. the UI can causes the identifier to change appearance). An audio discriminator is used to distinguish between an acoustic signal that was generated by a person speaking from that generated in a band-limited manner. The audio discriminator analyzes the spectrum of detected audio signals and generates several parameters from the spectrum and from past determinations to determine the source of an audio signal on a frame-by-frame basis.

Claims

1. A method, comprising:

receiving an identifier of a first participant of a plurality of participants of a multi-party conference;

receiving a first signal indicating that the first participant is providing information to the multi-party conference; and

displaying a first indication that the first participant is providing information to the multi-party conference in response to the received first signal.

2. The method of claim 1, further comprising terminating the first signal when the first participant is no longer providing information to the multi-party conference.

3. The method of claim 2, further comprising terminating display of the first indication in response to termination of the first signal.

4. The method of claim 1, further comprising:

while displaying the first indication, displaying an identifier of a second participant of the plurality of participants without indicating that the second participant is providing information to the multi-party conference.

5. The method of claim 1, wherein the first indication comprises displaying the identifier so as to flash.

6. The method of claim 1, wherein the first indication comprises displaying the identifier using a preselected color scheme.

7. The method of claim 1, wherein the first indication comprises displaying the identifier as an animated graphic.

8. The method of claim 1, wherein the first indication comprises displaying the identifier using video.

9. The method of claim 1, wherein the first indication comprises displaying the identifier as a photograph.

10. The method of claim 1, wherein the first indication comprises displaying the identifier with a graphic proximate to the identifier of the first participant.

11. The method of claim 10, wherein the graphic proximate to the identifier of the first participant comprises text.

12. The method of claim 1, wherein the first indication comprises displaying the identifier at a position within a list of identifiers of the plurality of participants designated for a participant providing information to the multi-party conference.

13. The method of claim 1, wherein the first indication comprises displaying the identifier of the first participant in an area of a display designated for a participant providing information to the multi-party conference.

14. The method of claim 1, wherein the identifier of the first participant comprises a location associated with the first participant.

15. The method of claim 1, wherein the first signal is received when the first participant is speaking over an audio link.

16. The method of claim 15, wherein the audio link comprises a telephone link.

17. The method of claim 15, wherein the audio link comprises a satellite link.

18. The method of claim 15, wherein the audio link comprises a voice over Internet Protocol (VoIP) link.

19. The method of claim 1, further comprising receiving information from a second participant of the plurality of participants via a network while the first participant is providing information to the multi-party conference.

20. The method of claim 19, wherein receiving information from the second participant further comprises receiving information via client-server communication over the network.

21. The method of claim 19, wherein receiving information from the second participant further comprises receiving information via peer-to-peer communication over the network.

22. The method of claim 19, wherein the network comprises the Internet.

23. The method of claim 19, wherein the network comprises a local area network.

24. The method of claim 19, wherein the network comprises a wireless network.

25. The method of claim 1, further comprising:

concurrently with the first indication, displaying a second indication that a second participant of the plurality of participants is providing information to the multi-party conference in response to receiving a second signal concurrently with the first signal, the second signal indicating that the second participant is providing information to the multi-party conference.

26. The method of claim 1, further comprising suppressing an indication that a second participant of the plurality of participants in response to receiving a signal concurrently indicating that the other providing information to the multi-party conference.

27. A computer-accessible medium having computer-executable instructions to perform operations comprising:

receiving an identifier of a first participant of a plurality of participants of a multi-party conference;

receiving a first signal indicating that the first participant is providing information to the multi-party conference; and

displaying a first indication that the first participant is providing information to the multi-party conference in response to the received first signal.

28. The computer-accessible medium of claim 27, wherein the operations further comprise:

while displaying the first indication, displaying an identifier of a second participant of the plurality of participants without indicating that the second participant is providing information to the multi-party conference.

29. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier so as to flash.

30. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier using a preselected color scheme.

31. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier as an animated graphic.

32. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier using video.

33. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier as a photograph.

34. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier with a graphic proximate to the identifier of the first participant.

35. The computer-accessible medium of claim 34, wherein the graphic proximate to the identifier of the first participant comprises text.

36. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier at a position within a list of identifiers of the plurality of participants designated for a participant providing information to the multi-party conference.

37. The computer-accessible medium of claim 27, wherein the first indication comprises displaying the identifier of the first participant in an area of a display designated for a participant providing information to the multi-party conference.

38. The computer-accessible medium of claim 27, wherein the first signal is received when the first participant is speaking over an audio link.

39. The computer-accessible medium of claim 27, wherein the operations further comprise:

concurrently with the first indication, displaying a second indication that a second participant of the plurality of participants is providing information to the multi-party conference in response to receiving a second signal concurrently with the first signal, the second signal indicating that the second participant is providing information to the multi-party conference.

40. The computer-accessible medium of claim 27 wherein the operations further comprise:

suppressing an indication that a second participant of the plurality of participants in response to receiving a signal concurrently indicating that the other providing information to the multi-party conference.

41. A system for multi-party conferencing for a plurality of participants, the system comprising:

a user interface (UI); and

an information source indicator to cause the UI to display an identifier corresponding to a participant in response to receiving a signal that indicates that the participant is providing information to the multi-party conference, wherein the displayed identifier has a property that indicates that the participant is providing information to the multi-party conference.

42. The system of claim 41, wherein the signal terminates in response to the participant no longer providing information to the multi-party conference.

43. The system of claim 42, wherein the information source indicator is to cause the UI to display the identifier without the property in response to the signal terminating.

44. The system of claim 41, wherein the information source indicator is further to cause the UI to display a plurality of identifiers corresponding to the plurality of participants without the property that indicates that the plurality of corresponding participants are providing information to the multi-party conference.

45. The system of claim 41, wherein the information provided by the participant is the participant'"'"'s speech over an audio link.

46. The system of claim 41, wherein the UI is to selectively display additional information received from the network while the participant is providing information to the multi-party conference.

47. A method, comprising:

detecting an acoustic signal;

determining whether the detected acoustic signal was generated by a person speaking; and

providing a signal indicating that the detected acoustic signal was generated by the person.

48. The method of claim 47, wherein determining whether the detected acoustic signal was generated by a person further comprises:

determining whether the detected acoustic signal was speech from an audio transducer device.

49. The method of claim 47, wherein the signal is transmitted over a network to participants of a multi-party conference.

50. The method of claim 47 wherein determining whether the detected acoustic signal was generated by a person further comprises:

receiving a frame of audio data derived from the detected acoustic signal;

classifying the received frame based on spectral data of the received frame; and

determining a source of a portion of the detected acoustic signal used to derive the frame based on the classification of the frame and a prior determination of a source of a detected acoustic signal portion.

51. The method of claim 50, wherein the spectral data is obtained by performing a frequency transform on the frame of audio data.

52. The method of claim 51, wherein the spectral data is obtained by performing a fast Fourier transform (FFT) on the frame of audio data.

53. The method of claim 51, wherein the spectral data is obtained by performing a modulated complex lapped transform (MCLT) on the frame of audio data.

54. The method of claim 50, further comprising determining a first frequency band'"'"'s energy and a second frequency band'"'"'s energy from the spectral data.

55. The method of claim 54, wherein the first frequency band corresponds to a frequency range for consonants and the second frequency band corresponds to a frequency range for vowels.

56. The method of claim 55, further comprising classifying the frame as being generated by a person when the ratio of the energies of the first and second frequency bands exceeds a first predetermined threshold.

57. The method of claim 56, further comprising selectively classifying the frame as being from a different source when the ratio of the energies of the first and second frequency bands is below a second predetermined threshold.

58. The method of claim 57, further comprising selectively classifying the frame as having an unknown source when the ratio of the energies of the first and second frequency bands exceeds the second predetermined threshold and is below the first predetermined threshold, the second predetermined threshold being less than the first predetermined threshold.

59. The method of claim 55, further comprising determining whether the frame was derived from speech, wherein speech includes acoustic signals generated by a person speaking and acoustic signals generated by an audio transducer device.

60. The method of claim 59, further comprising determining noise floor energies of the first and second frequency bands using the spectral data, wherein a frame is selectively classified as being derived from speech in response to the energy of the first frequency band exceeding the noise floor energy of the first frequency band or the energy of the second frequency band exceeding the noise floor of the second frequency band, or both.

61. The method of claim 60, wherein classifying the received frame further comprises:

determining whether a frame received within a predetermined number of frames relative to the received frame has substantially all of its energy in the second frequency band.

62. The method of claim 60, wherein classifying the received frame further comprises:

determining whether a frame adjacent to the received frame was classified as derived from speech.

63. The method of claim 50, wherein classifying the received frame further comprises classifying the received frame as one of the plurality of predetermined frame types comprising a live-type frame;

a phone-type frame; and

an unsure-type frame, wherein live-type frames represent frames determined to be derived from acoustic signals generated by a person speaking, and phone-type frames represent frames determined to be derived from acoustic signals generated by an audio transducer device.

64. The method of claim 63, further comprising determining the source of the detected acoustic signal to be an acoustic signal generated by a person, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an audio transducer device;

the frame is classified as a live-type frame; and

a predetermined number of prior frames includes live-type frames that exceed a predetermined live-type frame count threshold.

65. The method of claim 63, further comprising determining the source of the detected acoustic signal to be unsure, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an audio transducer device;

the frame is classified as a live-type frame;

a predetermined number of most recent frames do not include enough live-type frames to exceed a predetermined threshold; and

an elapsed time since receiving a previous frame derived from speech exceeds a predetermined first time threshold.

66. The method of claim 63, further comprising determining the source of the detected acoustic signal to be an audio transducer device, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an acoustic signal generated by a person speaking;

the frame is classified as a phone-type frame;

an elapsed time since receiving a previous live-type frame exceeds a predetermined second time threshold; and

a counter value does not exceed a predetermined count threshold, the counter value to track a number of consecutive non-live-type frames received after receiving a live-type frame of most recent frames do not include enough live-type frames to exceed a predetermined threshold.

67. The method of claim 63, further comprising determining the source of the detected acoustic signal to be unsure, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an acoustic signal generated by a person speaking;

the frame is classified as a phone-type frame;

an elapsed time since receiving a previous live-type frame exceeds a predetermined second time threshold; and

the counter value is below a predetermined count threshold, the counter value to track a number of consecutive non-live-type frames received after receiving a live-type frame of most recent frames do not include enough live-type frames to exceed a predetermined threshold.

68. The method of claim 63, further comprising determining the source of the detected acoustic signal to be an acoustic signal generated by a person speaking, if:

the prior determination of a source of a detected acoustic signal portion is unsure;

the frame is classified as a live-type frame; and

a predetermined number of most recent prior frames includes live-type frames that exceed in number a predetermined live-type frame count threshold.

69. The method of claim 63, further comprising determining the source of the detected acoustic signal to be an acoustic transducer device, if:

the prior determination of a source of a detected acoustic signal portion is unsure;

the frame is classified as a phone-type frame; and

a predetermined number of most recent prior frames includes phone-type frames that exceed in number a predetermined phone-type frame count threshold.

70. A computer-accessible medium having computer-executable instructions to perform operations comprising:

detecting an acoustic signal;

determining whether the detected acoustic signal was generated by a person speaking; and

providing a signal indicating that the detected acoustic signal was generated by the person.

71. The computer-accessible medium of claim 70, wherein determining whether the detected acoustic signal was generated by a person further comprises:

determining whether the detected acoustic signal was speech from an audio transducer device.

72. The computer-accessible medium of claim 70, wherein the signal is transmitted over a network to participants of a multi-party conference.

73. The computer-accessible medium of claim 70, wherein determining whether the detected acoustic signal was generated by a person further comprises:

receiving a frame of audio data derived from the detected acoustic signal;

classifying the received frame based on spectral data of the received frame; and

determining a source of a portion of the detected acoustic signal used to derive the frame based on the classification of the frame and a prior determination of a source of a detected acoustic signal portion.

74. The computer-accessible medium of claim 73, wherein the spectral data is obtained by performing a frequency transform on the frame of audio data.

75. The computer-accessible medium of claim 73, wherein the operations further comprise determining a first frequency band'"'"'s energy and a second frequency band'"'"'s energy from the spectral data.

76. The computer-accessible medium of claim 75, wherein the first frequency band corresponds to a frequency range for consonants and the second frequency band corresponds to a frequency range for vowels.

77. The computer-accessible medium of claim 76, wherein the operations further comprise:

classifying the frame as being generated by a person when the ratio of the energies of the first and second frequency bands exceeds a first predetermined threshold.

78. The computer-accessible medium of claim 77, wherein the operations further comprise:

selectively classifying the frame as being from another source when the ratio of the energies of the first and second frequency bands is below a second predetermined threshold.

79. The computer-accessible medium of claim 78, wherein the operations further comprise:

selectively classifying the frame as having an unknown source when the ratio of the energies of the first and second frequency bands exceeds the second predetermined threshold and is below the first predetermined threshold, the second predetermined threshold being less than the first predetermined threshold.

80. The computer-accessible medium of claim 76, wherein the operations further comprise:

determining whether the frame was derived from speech, wherein speech includes acoustic signals generated by a person speaking and acoustic signals generated by an audio transducer device.

81. The computer-accessible medium of claim 80, wherein the operations further comprise:

determining noise floor energies of the first and second frequency bands using the spectral data, wherein a frame is selectively classified as being derived from speech in response to the energy of the first frequency band exceeding the noise floor energy of the first frequency band or the energy of the second frequency band exceeding the noise floor of the second frequency band, or both.

82. The computer-accessible medium of claim 81, wherein classifying the received frame further comprises:

determining whether a frame adjacent to the received frame was classified as derived from speech.

83. The computer-accessible medium of claim 81, wherein classifying the received frame further comprises:

determining whether a frame within a predetermined number of frames relative to the received frame has substantially all of its energy in the second frequency band.

84. The computer-accessible medium of claim 74, wherein classifying the received frame further comprises:

classifying the received frame as one of the plurality of predetermined frame types comprising a live-type frame;

a phone-type frame; and

an unsure-type frame, wherein live-type frames represent frames determined to be derived from acoustic signals generated by a person speaking, and phone-type frames represent frames determined to be derived from acoustic signals generated by an audio transducer device.

85. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be an acoustic signal generated by a person, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an audio transducer device;

the frame is classified as a live-type frame; and

a predetermined number of prior frames includes live-type frames that exceed a predetermined live-type frame count threshold.

86. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be unsure, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an audio transducer device;

the frame is classified as a live-type frame;

a predetermined number of most recent frames do not include enough live-type frames to exceed a predetermined threshold; and

an elapsed time since receiving a previous frame derived from speech exceeds a predetermined first time threshold.

87. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be an audio transducer device, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an acoustic signal generated by a person speaking;

the frame is classified as a phone-type frame;

an elapsed time since receiving a previous live-type frame exceeds a predetermined second time threshold; and

a counter value does not exceed a predetermined count threshold, the counter value to track a number of consecutive non-live-type frames received after receiving a live-type frame of most recent frames do not include enough live-type frames to exceed a predetermined threshold.

88. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be unsure, if:

the prior determination of a source of a detected acoustic signal portion is that the source was an acoustic signal generated by a person speaking;

the frame is classified as a phone-type frame;

an elapsed time since receiving a previous live-type frame exceeds a predetermined second time threshold; and

the counter value is below a predetermined count threshold, the counter value to track a number of consecutive non-live-type frames received after receiving a live-type frame of most recent frames do not include enough live-type frames to exceed a predetermined threshold.

89. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be an acoustic signal generated by a person speaking, if:

the prior determination of a source of a detected acoustic signal portion is unsure;

the frame is classified as a live-type frame; and

a predetermined number of most recent prior frames includes live-type frames that exceed in number a predetermined live-type frame count threshold.

90. The computer-accessible medium of claim 84, wherein the operations further comprise determining the source of the detected acoustic signal to be an acoustic transducer device, if:

the prior determination of a source of a detected acoustic signal portion is unsure;

the frame is classified as a phone-type frame; and

a predetermined number of most recent prior frames includes phone-type frames that exceed in number a predetermined phone-type frame count threshold.

91. A system, comprising:

an audio discriminator to determine whether a detected acoustic signal was generated by a person in speaking; and

a sourcing signaler to selectively provide a signal indicating that the person is speaking in response to a determination by the audio discriminator that the detected acoustic signal was generated by the person in speaking.

92. The system of claim 91, wherein the sourcing signaler is to provide the signal to participants of a multi-party conference over a network.

93. The system of claim 91, wherein the audio discriminator is a component of the sourcing signaler.

94. The system of claim 91, wherein the audio discriminator comprises:

a spectrum analyzer to obtain spectrum data from a frame of audio data acquired from the detected acoustic signal;

a frame classifier to determine which of a set of frame-types the frame matches, based on the spectrum data obtained by the spectrum analyzer; and

a hidden Markov model (HMM) machine to determine whether the detected acoustic signal was sourced by an acoustic signal generated by a person speaking using the classification of the frame-type.

95. The system of claim 91, wherein the audio discriminator comprises:

a spectrum analyzer to obtain spectrum data from a frame of audio data acquired from the detected acoustic signal;

a frame classifier to determine which of a set of frame-types the frame matches, based on the spectrum data obtained by the spectrum analyzer; and

a finite state machine (FSM) to determine whether the detected acoustic signal was sourced by an acoustic signal generated by a person speaking using the classification of the frame-type.