Video game system with mixing of independent pre-encoded digital audio bitstreams

• US 8,194,862 B2
• Filed: 07/31/2009
• Issued: 06/05/2012
• Est. Priority Date: 07/31/2009
• Status: Active Grant

• Current Assignee: ActiveVideo Networks, Inc. (CommScope Holding Company, Inc.)
• Original Assignee: ActiveVideo Networks, Inc. (CommScope Holding Company, Inc.)
• Inventors: Herr, Stefan, Sigmund, Ulrich
• Primary Examiner(s): Phan, Hai
• Application Number:
  US12/534,016
• Publication Number:
  US 20110028215A1
• Time in Patent Office: 1,040 Days
• Field of Search: 381/23, 381/1, 381/17, 381/19, 704/201, 704/229, 704500-504, 704/E21.001, 463/35, 463/43
• US Class Current: 381/23
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 19/0204 using subband decomposition
• CPC Class Codes: G10L 19/173 Transcoding, i.e. converting between two coded representations avoiding cascaded coding-decoding

Abstract

A computer-implemented method of encoding audio includes accessing a plurality of independent audio source streams, each of which includes a sequence of source frames. Respective source frames of each sequence include respective pluralities of pulse-code modulated audio samples. Each of the plurality of independent audio source streams is separately encoded to generate a plurality of independent encoded streams, each of which corresponds to a respective independent audio source stream. The encoding includes, for respective source frames, converting respective pluralities of pulse-code modulated audio samples to respective pluralities of floating-point frequency samples that are divided into a plurality of frequency bands. An instruction to mix the plurality of independent encoded streams is received; in response, respective floating-point frequency samples of the independent encoded streams are combined. An output bitstream is generated that includes the combined respective floating-point frequency samples.

Claims

1. A method of encoding audio, comprising:

at an audio encoding system including one or more processors and memory, during execution of a video game by a computer system;

receiving an instruction to mix a first independent encoded audio stream with a second independent encoded audio stream, the first and second independent encoded audio streams each comprising a sequence of frames, wherein respective frames of each sequence comprise floating-point frequency samples divided into a plurality of frequency bands, the floating-point frequency samples of a respective frequency band of a respective frame of the first independent encoded audio stream being scaled by a first scale factor, the floating-point frequency samples of a respective frequency band of a respective frame of the second independent encoded audio stream being scaled by a second scale factor;

in response to the instruction to mix the first independent encoded audio stream with the second independent encoded audio stream, combining respective floating-point frequency samples of the first and second independent encoded audio streams, the combining comprising;

calculating an adjusted scale factor as a first function of a difference between the first and second scale factors;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the first independent encoded audio stream by a first ratio of the first scale factor to the adjusted scale factor;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the second independent encoded audio stream by a second ratio of the second scale factor to the adjusted scale factor; and

adding respective floating-point frequency samples of the first independent encoded audio stream, as scaled by the first ratio, to respective floating-point frequency samples of the second independent encoded audio stream, as scaled by the second ratio; and

generating an output bitstream comprising the combined respective floating-point frequency samples.

2. The method of claim 1, further comprising transmitting the output bitstream to a client device for decoding and playback.

3. The method of claim 1, wherein the combining further comprises:

determining that a combined floating-point frequency sample, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, exceeds a predefined limit; and

in response to the determination, assigning the combined floating-point frequency sample to equal the predefined limit.

4. The method of claim 1, wherein respective mantissas of combined floating-point frequency samples, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, are stored in respective single bytes.

5. The method of claim 1, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits.

6. The method of claim 1, wherein the first function comprises addition of an offset to the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the first and second scale factors.

7. The method of claim 1, wherein:

the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table; and

the difference between the first and second scale factors is calculated by subtracting the lower of the indices corresponding to the first and second scale factors from the larger of the indices corresponding to the first and second scale factors.

8. The method of claim 7, wherein the first function comprises subtraction of an offset from the lower of the indices encoding the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the indices encoding the first and second scale factors.

9. The method of claim 7, wherein each of the indices encoding the first, second, and adjusted scale factors is stored in a single byte.

10. The method of claim 1, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the combining further comprising:

scaling the floating-point frequency samples of the respective frequency band and respective frame of the first independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and first scale factors;

scaling the floating-point frequency samples of the respective frequency band and respective frame of the second independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and second scale factors; and

adding respective floating-point frequency samples, as scaled, of the first and second independent encoded bitstreams.

11. The method of claim 10, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits, the combining further comprising:

dividing the index encoding the adjusted scale factor to produce a divided scale factor index being represented by six bits; and

writing the divided scale factor index to the encoded bitstream.

12. The method of claim 1, wherein the combining comprises calculating respective sums of respective floating-point frequency samples and dividing the respective sums by a constant value.

13. The method of claim 12, wherein the constant value equals 2 or √

{square root over (2)}.

14. The method of claim 1, wherein:

the first and second independent encoded streams of the plurality of independent encoded streams each comprises a left channel and a right channel; and

the combining comprises;

mixing the left channels of the first and second independent encoded streams to generate a left channel of the output bitstream; and

mixing the right channels of first and second independent encoded streams to generate a right channel of the output bitstream.

15. The method of claim 1, wherein:

the first independent encoded stream comprises a left channel and a right channel;

the second independent encoded stream comprises a mono channel; and

the combining comprises;

mixing the left channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a left channel of the output bitstream; and

mixing the right channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a right channel of the output bitstream.

16. The method of claim 1, wherein:

the first and second independent encoded streams each comprises first and second stereo channels for frequency bands below a predefined limit and a mono channel for frequency bands above the predefined limit; and

the combining comprises separately mixing the first stereo channels, second stereo channels, and mono channels of the first and second independent encoded streams.

17. The method of claim 1, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a continuous source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises an episodic source of non-silent audio data.

18. The method of claim 1, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a first episodic source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises a second episodic source of non-silent audio data.

19. A system for encoding audio, comprising:

memory;

one or more processors;

one or more programs stored in the memory and configured for execution by the one or more processors, the one or more programs including instructions for;

receiving an instruction to mix a first independent encoded audio stream with a second independent encoded audio stream, the first and second independent encoded audio streams each comprising a sequence of frames, wherein respective frames of each sequence comprise floating-point frequency samples divided into a plurality of frequency bands, the floating-point frequency samples of a respective frequency band of a respective frame of the first independent encoded audio stream being scaled by a first scale factor, the floating-point frequency samples of a respective frequency band of a respective frame of the second independent encoded audio stream being scaled by a second scale factor;

in response to the instruction to mix the first independent encoded audio stream with the second independent encoded audio stream, combining the respective floating-point frequency samples of the first and second independent encoded audio streams, the combining comprising;

calculating an adjusted scale factor as a first function of a difference between the first and second scale factors;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the first independent encoded audio stream by a first ratio of the first scale factor to the adjusted scale factor;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the second independent encoded audio stream by a second ratio of the second scale factor to the adjusted scale factor; and

adding respective floating-point frequency samples of the first independent encoded audio stream, as scaled by the first ratio, to respective floating-point frequency samples of the second independent encoded audio stream, as scaled by the second ratio; and

generating an output bitstream comprising the combined respective floating-point frequency samples.

20. The system of claim 19, wherein the instructions for combining further comprise instructions for:

determining that a combined floating-point frequency sample, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, exceeds a predefined limit; and

in response to the determination, assigning the combined floating-point frequency sample to equal the predefined limit.

21. The system of claim 19, wherein respective mantissas of combined floating-point frequency samples, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, are stored in respective single bytes.

22. The system of claim 19, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits.

23. The system of claim 19, wherein the first function comprises addition of an offset to the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the first and second scale factors.

24. The system of claim 19, wherein:

the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table; and

the difference between the first and second scale factors is calculated by subtracting the lower of the indices corresponding to the first and second scale factors from the larger of the indices corresponding to the first and second scale factors.

25. The system of claim 24, wherein the first function comprises subtraction of an offset from the lower of the indices encoding the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the indices encoding the first and second scale factors.

26. The system of claim 24, wherein each of the indices encoding the first, second, and adjusted scale factors is stored in a single byte.

27. The system of claim 19, wherein the one or more programs further comprise instructions for transmitting the output bitstream to a client device for decoding and playback.

28. The system of claim 19, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, and the instructions for combining further comprise instructions for:

scaling the floating-point frequency samples of the respective frequency band and respective frame of the first independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and first scale factors;

scaling the floating-point frequency samples of the respective frequency band and respective frame of the second independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and second scale factors; and

adding respective floating-point frequency samples, as scaled, of the first and second independent encoded bitstreams.

29. The system of claim 28, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits, and the instructions for combining further comprise instructions for:

dividing the index encoding the adjusted scale factor to produce a divided scale factor index being represented by six bits; and

writing the divided scale factor index to the encoded bitstream.

30. The system of claim 19, wherein the instructions for combining further comprise instructions for calculating respective sums of respective floating-point frequency samples and dividing the respective sums by a constant value.

31. The system of claim 30, wherein the constant value equals 2 or √

{square root over (2)}.

32. The system of claim 19, wherein:

the first and second independent encoded streams of the plurality of independent encoded streams each comprises a left channel and a right channel; and

the instructions for combining further comprise instructions for;

mixing the left channels of the first and second independent encoded streams to generate a left channel of the output bitstream; and

mixing the right channels of first and second independent encoded streams to generate a right channel of the output bitstream.

33. The system of claim 19, wherein:

the first independent encoded stream comprises a left channel and a right channel;

the second independent encoded stream comprises a mono channel; and

the instructions for combining further comprise instructions for;

mixing the left channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a left channel of the output bitstream; and

mixing the right channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a right channel of the output bitstream.

34. The system of claim 19, wherein:

the first and second independent encoded streams each comprises first and second stereo channels for frequency bands below a predefined limit and a mono channel for frequency bands above the predefined limit; and

the instructions for combining further comprise instructions for separately mixing the first stereo channels, second stereo channels, and mono channels of the first and second independent encoded streams.

35. The system of claim 19, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a continuous source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises an episodic source of non-silent audio data.

36. The system of claim 19, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a first episodic source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises a second episodic source of non-silent audio data.

37. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computer system, cause the computer system to:

receive an instruction to mix a first independent encoded audio stream with a second independent encoded audio stream, the first and second independent encoded audio streams each comprising a sequence of frames, wherein respective frames of each sequence comprise floating-point frequency samples divided into a plurality of frequency bands, the floating-point frequency samples of a respective frequency band of a respective frame of the first independent encoded audio stream being scaled by a first scale factor, the floating-point frequency samples of a respective frequency band of a respective frame of the second independent encoded audio stream being scaled by a second scale factor;

in response to the instruction to mix the first independent encoded audio stream with the second independent encoded audio stream, combine the respective floating-point frequency samples of the first and second independent encoded audio streams the combining comprising;

calculating an adjusted scale factor as a first function of a difference between the first and second scale factors;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the first independent encoded audio stream by a first ratio of the first scale factor to the adjusted scale factor;

scaling the floating-point frequency samples of the respective frequency band of the respective frame of the second independent encoded audio stream by a second ratio of the second scale factor to the adjusted scale factor; and

adding respective floating-point frequency samples of the first independent encoded audio stream, as scaled by the first ratio, to respective floating-point frequency samples of the second independent encoded audio stream, as scaled by the second ratio; and

generate an output bitstream comprising the combined respective floating-point frequency samples.

38. The non-transitory computer readable storage medium of claim 37, wherein the one or more programs further comprise instructions which, when executed by the computer system, cause the computer system to:

determine that a combined floating-point frequency sample, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, exceeds a predefined limit; and

in response to the determination, assign the combined floating-point frequency sample to equal the predefined limit.

39. The non-transitory computer readable storage medium of claim 37, wherein respective mantissas of combined floating-point frequency samples, generated by adding respective floating-point frequency samples of the first and second encoded bitstreams, are stored in respective single bytes.

40. The non-transitory computer readable storage medium of claim 37, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits.

41. The non-transitory computer readable storage medium of claim 37, wherein the first function comprises addition of an offset to the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the first and second scale factors.

42. The non-transitory computer readable storage medium of claim 37, wherein:

the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table; and

the difference between the first and second scale factors is calculated by subtracting the lower of the indices corresponding to the first and second scale factors from the larger of the indices corresponding to the first and second scale factors.

43. The non-transitory computer readable storage medium of claim 42, wherein the first function comprises subtraction of an offset from the lower of the indices encoding the first or second scale factor, the offset being a monotonic second function of the magnitude of the difference between the indices encoding the first and second scale factors.

44. The non-transitory computer readable storage medium of claim 42, wherein each of the indices encoding the first, second, and adjusted scale factors is stored in a single byte.

45. The non-transitory computer readable storage medium of claim 37, wherein the one or more programs further comprise instructions which, when executed by the computer system, cause the computer system to transmit the output bitstream to a client device for decoding and playback.

46. The non-transitory computer readable storage medium of claim 37, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, and the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to:

scale the floating-point frequency samples of the respective frequency band and respective frame of the first independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and first scale factors;

scale the floating-point frequency samples of the respective frequency band and respective frame of the second independent encoded bitstream by a scale factor value having an index corresponding to a difference between indices encoding the adjusted and second scale factors; and

add respective floating-point frequency samples, as scaled, of the first and second independent encoded bitstreams.

47. The non-transitory computer readable storage medium of claim 46, wherein the first, second, and adjusted scale factors are encoded as indices referencing scale factor values stored in a table, the indices each being represented with more than six bits, and the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to:

divide the index encoding the adjusted scale factor to produce a divided scale factor index being represented by six bits; and

write the divided scale factor index to the encoded bitstream.

48. The non-transitory computer readable storage medium of claim 37, wherein the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to calculate respective sums of respective floating-point frequency samples and dividing the respective sums by a constant value.

49. The non-transitory computer readable storage medium of claim 48, wherein the constant value equals 2 or √

{square root over (2)}.

50. The non-transitory computer readable storage medium of claim 37, wherein:

the first and second independent encoded streams of the plurality of independent encoded streams each comprises a left channel and a right channel; and

the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to;

mix the left channels of the first and second independent encoded streams to generate a left channel of the output bitstream; and

mix the right channels of first and second independent encoded streams to generate a right channel of the output bitstream.

51. The non-transitory computer readable storage medium of claim 37, wherein:

the first independent encoded stream comprises a left channel and a right channel;

the second independent encoded stream comprises a mono channel; and

the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to;

mix the left channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a left channel of the output bitstream; and

mix the right channel of the first independent encoded stream with the mono channel of the second independent encoded stream to generate a right channel of the output bitstream.

52. The non-transitory computer readable storage medium of claim 37, wherein:

the first and second independent encoded streams each comprises first and second stereo channels for frequency bands below a predefined limit and a mono channel for frequency bands above the predefined limit; and

the instructions to combine further comprise instructions which, when executed by the computer system, cause the computer system to separately mix the first stereo channels, second stereo channels, and mono channels of the first and second independent encoded streams.

53. The non-transitory computer readable storage medium of claim 37, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a continuous source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises an episodic source of non-silent audio data.

54. The non-transitory computer readable storage medium of claim 37, wherein:

the first independent encoded audio stream is generated from a first independent audio source stream that comprises a first episodic source of non-silent audio data; and

the second independent encoded audio stream is generated from a second independent audio source stream that comprises a second episodic source of non-silent audio data.