Scalable audio in a multi-point environment

• US 8,831,932 B2
• Filed: 11/11/2011
• Issued: 09/09/2014
• Est. Priority Date: 07/01/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (Plantronics, Inc.)
• Inventors: Feng, Jinwei, Chu, Peter L., Botzko, Stephen
• Primary Examiner(s): Vo, Huyen X.
• Application Number:
  US13/294,471
• Publication Number:
  US 20120290305A1
• Time in Patent Office: 1,033 Days
• Field of Search: 704/208, 704/210, 704/214, 704/215, 704500-504, 704/207, 704/229, 704/230, 348/14.07
• US Class Current: 704/207
• CPC Class Codes: G10L 19/002 Dynamic bit allocation for perceptual audio coders G10L19/032
• CPC Class Codes: G10L 19/0212 using orthogonal transformation
• CPC Class Codes: G10L 19/24 Variable rate codecs, e.g. for generating different qualities using a scalable representation such as hierarchical encoding or layered encoding
• CPC Class Codes: G10L 25/18 the extracted parameters being spectral information of each sub-band
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 7/0072 Speech codec negotiation H04L65/1069 takes precedence

Abstract

Use of a scalable audio codec to implement distributed mixing and/or sender bit rate regulation in a multipoint conference is disclosed. The scalable audio codec allows the audio signal from each endpoint to be split into one or more frequency bands and for the transform coefficients within such bands to be prioritized such that usable audio may be decoded from a subset of the entire signal. The subset may be created by omitting certain frequency bands and/or by omitting certain coefficients within the frequency bands. By providing various rules for each endpoint in a conference, the endpoint can determine the importance of its signal to the conference and can select an appropriate bit rate, thereby conserving bandwidth and/or processing power throughout the conference.

Claims

1. A scalable audio processing method for a conference, the method comprising:

determining a state of a first endpoint in the conference, the state indicative of the first endpoint being a primary source or a secondary source of conferencing audio;

assessing the state by applying one or more rules to the state;

processing audio for the conference with a scalable audio codec, wherein at least one parameter of the scalable audio codec is selected based on the assessed state of the first endpoint, wherein processing the audio with the scalable audio codec at least includes stripping bits from an encoded bit stream of the audio, the encoded bit stream having the bits adjustably arranged based on different frequency bands of the audio; and

transmitting the processed audio for the conference.

2. The method of claim 1, wherein the processing and transmitting are implemented by a given endpoint receiving the encoded bit stream of the audio in the conference, the given endpoint including the first endpoint or a second endpoint in the conference, the processing and transmitting comprising at least one of:

decoding the encoded bit stream into output audio and sending the output audio for output at the given endpoint; and

stripping the bits from the encoded bit stream, decoding the stripped bit stream into the output audio, and sending the output audio for output at the given endpoint.

3. The method of claim 1, wherein the processing and transmitting are implemented by a given endpoint in the conference obtaining input audio at the given endpoint, the given endpoint including the first endpoint or a second endpoint in the conference, the processing and transmitting comprising at least one of:

encoding the input audio into the encoded bit stream and sending the encoded bit stream via a network; and

encoding the input audio in the encoded bit stream, stripping the bits from the encoded bit stream, and sending the stripped bit stream via the network.

4. The method of claim 1, wherein the processing and transmitting of the audio are implemented by a multipoint control unit in communication with a given endpoint in the conference via a network, the given endpoint including the first endpoint or a second endpoint, the processing and transmitting comprising at least one of:

receiving the encoded bit stream of the audio via the network and relaying the encoded bit stream to the given endpoint via the network; and

receiving the encoded bit stream of the audio via the network, stripping the bits from the encoded bit stream, and sending the stripped bit stream to the given endpoint via the network.

5. The method of claim 1, wherein the state is further indicative of the endpoint being a quiet source.

6. The method of claim 1, wherein processing the audio for the conference with the scalable audio codec comprises stripping the bits from the encoded bit stream of the audio using the scalable audio codec when the state of the first endpoint is indicative of a secondary source.

7. The method of claim 1, wherein the at least one parameter of the scalable audio codec comprises an indication of where to strip the bits in the encoded bit stream.

8. The method of claim 1, wherein assessing the state by applying one or more rules to the state comprises:

changing the state from a primary source to a secondary source when (i) a first time interval of the first endpoint in the state has elapsed and (ii) a second endpoint in the conference has a state of a primary source;

or

changing the state from a secondary source to a quiet source when (i) a second time interval of the first endpoint in the state has elapsed and (ii) a second endpoint in the conference has a state of a secondary source.

9. The method of claim 1, wherein assessing the state by applying one or more rules to the state comprises:

maintaining the state of a primary source when the first endpoint changes from not having speech audio to having speech audio;

changing the state from a secondary source to a primary source when (i) the first endpoint changes from not having speech audio to having speech audio and (ii) a second endpoint in the conference has a state of a primary source and does not have speech audio;

or

changing the state from a quiet source to a primary source when the first endpoint changes from not having speech audio to having speech audio.

10. The method of claim 1, wherein assessing the state by applying one or more rules to the state comprises:

changing the state from a primary source to a secondary source when (i) the first endpoint changes from having speech audio to not having speech audio and (ii) a first time interval of the first endpoint in the state has elapsed;

or

changing the state from a secondary source to a quiet source when (i) the first endpoint changes from having speech audio to not having speech audio and (ii) a second time interval of the first endpoint in the state has elapsed.

11. The method of claim 1, wherein determining the state of the first endpoint comprises detecting a presence of speech in audio of the first endpoint, wherein the state is further indicative of the first endpoint having speech audio or not having speech audio.

12. The method of claim 1, further comprising determining, after determining the state, that a time interval associated with the state has elapsed, wherein assessing the state is performed at least after the time interval has elapsed.

13. The method of claim 12, wherein the time interval associated with the state comprises separate time intervals for the state of primary source and the state of secondary source.

14. The method of claim 1, wherein processing the audio with the scalable audio codec comprises:

transform coding the different frequency bands of the audio into different sets of transform coefficients; and

packetizing the different sets of transform coefficients.

15. The method of claim 14, wherein the at least one parameter of the scalable audio codec comprise a plurality of bit allocations allocated for the different frequency bands.

16. The method of claim 15, wherein packetizing comprises packetizing the different sets of transform coefficients with the corresponding bit allocations, whereby the encoded bit stream has the bits adjustably allocated into the different frequency bands based on the corresponding bit allocations.

17. The method of claim 15, wherein the bit allocations associated with the assessed state for the primary source emphasize lower frequencies of the different frequency bands.

18. The method of claim 15, wherein the bit allocations associated with the assessed state for the secondary source deemphasizes higher frequencies of the different frequency bands.

19. The method of claim 14, wherein the at least one parameter of the scalable audio codec comprise frequency regions of the different frequency bands in which the transform coefficients are arranged, and wherein packetizing comprises packetizing the different sets of transform coefficients based on an ordered importance of the frequency regions, whereby the encoded bit stream has the bits adjustably ordered into the frequency regions based on the ordered importance.

20. The method of claim 19, wherein the ordered importance of the frequency regions emphasizes frequency regions with greater energy levels or lower frequencies.

21. A programmable storage device having program instructions stored thereon for causing a programmable control device to perform a scalable audio processing method, the method comrpising:

determining a state of a first endpoint in the conference, the state indicative of the first endpoint being a primary source or a secondary source of conferencing audio;

assessing the state by applying one or more rules to the state;

processing audio for the conference with a scalable audio codec, wherein at least one parameter of the scalabel audio codec is selected based on the assessed state of the first endpoint, wherein processing the audio with the scalable audio codec at least includes stripping bits from encoded bit stream of the audio, the encoded bit stream having the bits adjustably arranged based on different frequency bands of the audio; and

transmitting the processed audio for the conference.

26. The programmable storage device of claim 21, wherein the processing and transmitting are implemented by a given endpoint receiving the encoded bit stream of the audio in the conference, the given endpoint including the first endpoint or a second endpoint in the conference;

and wherein the processing and transmitting comprise at least one of;

decoding the encoded bit stream into output audio and sending the output audio for output at the given endpoint; and

stripping the bits from the encoded bit stream, decoding the stripped bit stream into the output audio, and sending the output audio for output at the given endpoint.

27. The programmable storage device of claim 21, wherein the processing and transmitting are implemented by a given endpoint in the conference obtaining input audio at the given endpoint, the given endpoint including the first endpoint or a second endpoint in the conference;

and wherein the processing and transmitting comprise at least one of;

encoding the input audio into the encoded bit stream and sending the encoded bit stream via a network; and

encoding the input audio in the encoded bit stream, stripping the bits from the encoded bit stream, and sending the stripped bit stream via the network.

28. The programmable storage device of claim 21, wherein the processing and transmitting of the audio are implemented by a multipoint control unit in communication with a given endpoint in the conference via a network, the given endpoint including the first endpoint or a second endpoint;

and wherein the processing and transmitting comprise at least one of;

receiving the encoded bit stream of the audio via the network and relaying the encoded bit stream to the given endpoint via the network; and

receiving the encoded bit stream of the audio via the network, stripping the bits from the encoded bit stream, and sending the stripped bit stream to the given endpoint via the network.

29. The programmable storage device of claim 21, wherein processing the audio for the conference with the scalable audio codec comprises stripping the bits from the encoded bit stream of the audio using the scalable audio codec when the state of the first endpoint is indicative of a secondary source.

30. The programmable storage device of claim 21, wherein the at least one parameter of the scalable audio codec comprises an indication of where to strip the bits in the encoded bit stream.

31. The programmable storage device of claim 21, wherein assessing the state by applying one or more rules to the state comprises:

changing the state from a primary source to a secondary source when (i) a first time interval of the first endpoint in the state has elapsed and (ii) a second endpoint in the conference has a state of a primary source;

changing the state from a secondary source to a quiet source when (i) a second time interval of the first endpoint in the state has elapsed and (ii) a second endpoint in the conference has a state of a secondary source;

maintaining the state of a primary source when the first endpoint changes from not having speech audio to having speech audio;

changing the state from a secondary source to a primary source when (i) the first endpoint changes from not having speech audio to having speech audio and (ii) a second endpoint in the conference has a state of a primary source and does not have speech audio;

changing the state from a quiet source to a primary source when the first endpoint changes from not having speech audio to having speech audio;

changing the state from a primary source to a secondary source when (i) the first endpoint changes from having speech audio to not having speech audio and (ii) a first time interval of the first endpoint in the state has elapsed;

or

changing the state from a secondary source to a quiet source when (i) the first endpoint changes from having speech audio to not having speech audio and (ii) a second time interval of the first endpoint in the state has elapsed.

32. The programmable storage device of claim 21, wherein determining the state of the first endpoint comprises detecting a presence of speech in audio of the first endpoint, wherein the state is further indicative of the first endpoint having speech audio or not having speech audio.

33. The programmable storage device of claim 21, wherein processing the audio with the scalable audio codec comprises:

transform coding the different frequency bands of the audio into different sets of transform coefficients; and

packetizing the different sets of transform coefficients.

34. The programmable storage device of claim 33, wherein the at least one parameter of the scalable audio codec comprise a plurality of bit allocations allocated for the different frequency bands.

35. The programmable storage device of claim 34, wherein packetizing comprises packetizing the different sets of transform coefficients with the corresponding bit allocations, whereby the encoded bit stream has the bits adjustably allocated into the different frequency bands based on the corresponding bit allocations.

36. The programmable storage device of claim 34, wherein the bit allocations associated with the assessed state for the primary source emphasize lower frequencies of the different frequency bands;

and wherein the bit allocations associated with the assessed state for the secondary source deemphasize higher frequencies of the different frequency bands.

37. The programmable storage device of claim 33, wherein the at least one parameter of the scalable audio codec comprise frequency regions of the different frequency bands in which the transform coefficients are arranged, and wherein packetizing comprises packetizing the different sets of transform coefficients based on an ordered importance of the frequency regions, whereby the encoded bit stream has the bits adjustably ordered into the frequency regions based on the ordered importance.

22. A processing device, comprising:

a network interface; and

a processor communicatively coupled to the network interface and obtaining audio signals, the processor having a scalable audio codec and configured to;

determine a state of a first audio device in a distributed audio network, the state indicative of the first audio device being a primary source or a secondary source of network audio,

apply one or more rules to assess the state,

process audio for the distributed audio network with the scalable audio codec, wherein at least one parameter of the scalable audio codec is selected based on the assessed state of the first audio device, wherein to process the audio with the scalable audio codec, the processor is at least configured to strip bits from an encoded bit stream of the audio, the encoded bit stream having the bits adjustably arranged based on different frequency bands of the audio, and

transmit the processed audio for the distributed audio network.

23. The processing device of claim 22, wherein the processing device comprises one of:

the first audio device for which the state is determined, a multipoint control unit in the distributed audio network, or a second audio device in the distributed audio network.

24. The processing device of claim 22, wherein the distributed audio network comprises a multipoint conference;

and wherein the first audio device comprises a first endpoint of the multipoint conference.

38. The device of claim 22, wherein the processing device receives the encoded bit stream;

and wherein to process and transmit, the processor is configured to at least one of;

decode the encoded bit stream into output audio and send the output audio for output; and

strip the bits from the encoded bit stream, decode the stripped bit stream into the output audio, and send the output audio for output.

39. The device of claim 22, wherein the processing device obtains the audio as input audio;

and wherein to process and transmit, the processor is configured to at least one of;

encode the input audio into the encoded bit stream and send the encoded bit stream via the distributed audio network; and

encode the input audio in the encoded bit stream, strip the bits from the encoded bit stream, and send the stripped bit stream via the distributed audio network.

40. The device of claim 22, wherein to process and transmit, the processor is configured to at least one of:

receive the encoded bit stream of the audio via the distributed audio network and relay the encoded bit stream via the distributed audio network; and

receive the encoded bit stream of the audio via the distributed audio network, strip the bits from the encoded bit stream, and send the stripped bit stream via the distributed audio network.

41. The device of claim 22, wherein to process the audio for the distributed audio with the scalable audio codec, the processor is configured to strip the bits from the encoded bit stream of the audio using the scalable audio codec when the state of the first audio device is indicative of a secondary source.

42. The device of claim 22, wherein the at least one parameter of the scalable audio codec comprises an indication of where to strip the bits in the encoded bit stream.

43. The device of claim 22, wherein to apply one or more rules to assess the state, the processor is configured to:

change the state from a primary source to a secondary source when (i) a first time interval of the first audio device in the state has elapsed and (ii) a second audio device in the conference has a state of a primary source;

change the state from a secondary source to a quiet source when (i) a second time interval of the first audio device in the state has elapsed and (ii) a second audio device in the conference has a state of a secondary source;

maintain the state of a primary source when the first endpoint changes from not having speech audio to having speech audio;

change the state from a secondary source to a primary source when (i) the first audio device changes from not having speech audio to having speech audio and (ii) a second audio device in the conference has a state of a primary source and does not have speech audio;

change the state from a quiet source to a primary source when the first audio device changes from not having speech audio to having speech audio;

change the state from a primary source to a secondary source when (i) the first audio device changes from having speech audio to not having speech audio and (ii) a first time interval of the first audio device in the state has elapsed;

or

change the state from a secondary source to a quiet source when (i) the first audio device changes from having speech audio to not having speech audio and (ii) a second time interval of the first endpoint in the state has elapsed.

44. The device of claim 22, wherein to determine the state of the first audio device, the processor is configured to detect a presence of speech in the audio of the first audio device, wherein the state is further indicative of the first audio device having speech audio or not having speech audio.

45. The device of claim 22, wherein to process the audio with the scalable audio codec, the processor is configured to:

transform code the different frequency bands of the audio into different sets of transform coefficients; and

packetize the different sets of transform coefficients.

46. The device of claim 45, wherein the at least one parameter of the scalable audio codec comprise a plurality of bit allocations allocated for the different frequency bands.

47. The device of claim 46, wherein to packetize, the processor is configured to packetize the different sets of transform coefficients with the corresponding bit allocations, whereby the encoded bit stream has the bits adjustably allocated into the different frequency bands based on the corresponding bit allocations.

48. The device of claim 46, wherein the bit allocations associated with the assessed state for the primary source emphasize lower frequencies of the different frequency bands;

and wherein the bit allocations associated with the assessed state for the secondary source deemphasize higher frequencies of the different frequency bands.

49. The device of claim 45, wherein the at least one parameter of the scalable audio codec comprise frequency regions of the different frequency bands in which the transform coefficients are arranged, and wherein to packetize, the processor is configured to packetize the different sets of transform coefficients based on an ordered importance of the frequency regions, whereby the encoded bit stream has the bits adjustably ordered into the frequency regions based on the ordered importance.

25. A scalable audio processing method, comprising:

determining a state of a first audio device in a distributed audio network, the state indicative of the first audio device being a primary source or a secondary source of network audio;

assessing the state by applying one or more rules to the state;

processing audio for the distributed audio network with a scalable audio codec,

wherein at least one parameter of the scalable audio codec is selected based on the assessed state of the first audio device, wherein processing the audio with the scalable audio codec at least includes stripping bits from an encoded bit stream of the audio, the encoded bit stream having the bits adjustably arranged based on different frequency bands of the audio; and

transmitting the processed audio for the distributed audio network.