Constrained dynamic amplitude panning in collaborative sound systems

• US 9,131,298 B2
• Filed: 03/14/2013
• Issued: 09/08/2015
• Est. Priority Date: 11/28/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Lae-Hoon, Xiang, Pei
• Primary Examiner(s): BLOUIN, MARK S
• Application Number:
  US13/830,894
• Publication Number:
  US 20140146984A1
• Time in Patent Office: 908 Days
• Field of Search: 381/307
• US Class Current: 1/1
• CPC Class Codes: H04R 2205/024 Positioning of loudspeaker enclosures for spatial sound reproduction
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 5/00 Stereophonic arrangements stereophonic pick-ups H04R9/16, H04R11/12, H04R17/08, H04R19/10
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence
• CPC Class Codes: H04S 7/308 Electronic adaptation dependent on speaker or headphone connection

Abstract

In general, techniques are described for performing constrained dynamic amplitude panning in collaborative sound systems. A headend device comprising one or more processors may perform the techniques. The processors may be configured to identify, for a mobile device participating in a collaborative surround sound system, a specified location of a virtual speaker of the collaborative surround sound system and determine a constraint that impacts playback of audio signals rendered from an audio source by the mobile device. The processors may be further configure to perform dynamic spatial rendering of the audio source with the determined constraint to render audio signals that reduces the impact of the determined constraint during playback of the audio signals by the mobile device.

Claims

1. A method comprising:

identifying two or more mobile devices of a plurality of mobile devices participating in a collaborative surround sound system capable of representing a virtual speaker of the collaborative surround sound system;

determining a constraint that impacts playback of audio signals rendered from audio source data by at least one of the identified two or more mobile devices;

determining, based on the constraint, a gain for the at least one of the identified two or more mobile devices; and

rendering the audio source data using the gain to generate audio signals that reduce the impact of the determined constraint during playback of the audio signals by the identified two or more mobile devices.

2. The method of claim 1, wherein determining the constraint comprises:

determining an expected power duration that indicates an expected duration that the at least one of the identified two or more mobile device will have sufficient power to playback the audio signals rendered from the audio source data;

determining a source audio duration that indicates a playback duration of the audio signals rendered from the audio source data; and

when the source audio duration exceeds the expected power duration, determining the expected power duration as the constraint.

3. The method of claim 2, wherein rendering the audio source data using the gain comprises rendering the audio source data using the gain to generate the audio signals such that an expected power duration to playback the audio signals is less than the source audio duration.

4. The method of claim 1,

wherein determining the constraint comprises determining a frequency dependent constraint, and

wherein rendering the audio source data using the at least one gain comprises rendering the audio source data using the at least one gain to generate the audio signals such that an expected power duration to playback the audio signals by the at least one of the identified two or more mobile devices is less than a duration of the audio source data.

5. The method of claim 1,

wherein rendering the audio source data comprises rendering the audio source data using an expected power duration, as the constraint to generate the audio signals, to playback the audio signals by the at least one of the identified two or more mobile devices such that the expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the audio source data.

6. The method of claim 1,

wherein the plurality of mobile devices comprise a first mobile device, a second mobile device and a third mobile device,

wherein the virtual speaker comprises one of a plurality of virtual speakers of the collaborative surround sound system,

wherein the constraint comprises one or more expected power durations, the one or more expected power duration each indicating an expected duration for which one of the plurality of mobile devices will have sufficient power to playback audio signals rendered from the audio source data, and

wherein determining the gain for the at least one of the identified two or more mobile devices comprises:

computing volume gains g₁, g₂ and g₃ for the first mobile device, the second mobile device and the third mobile device, respectively, in accordance with the following equation;

7. The method of claim 1, wherein rendering the audio source data using the gain comprises performing a constrained vector based dynamic amplitude panning with respect to the audio source data to generate the audio signals so as to reduce the impact of the determined constraint on playback of the audio signals by the at least one of the two or more mobile devices.

8. The method of claim 1, wherein the virtual speaker of the collaborative surround sound system appears to be placed in a location different than a location of at least one of the two or more mobile devices.

9. The method of claim 1, wherein the audio source data comprises one of a higher order ambisonic audio source data, a multi-channel audio source data and an object-based audio source data.

10. A headend device comprising:

one or more processors configured to identify two or more mobile devices of a plurality of mobile devices participating in a collaborative surround sound system capable of representing a virtual speaker of the collaborative surround sound system, determine a constraint that impacts playback of audio signals rendered from audio source data by at least one of the identified two or more mobile devices, determine, based on the constraint, a gain for the at least one of the identified two or more mobile devices, and render the audio source data using the gain to generate audio signals that reduce the impact of the determined constraint during playback of the audio signals by the identified two or more mobile devices; and

a memory configured to store the audio signals.

11. The headend device of claim 10, wherein the one or more processors are further configured to, when determining the constraint, determine an expected power duration that indicates an expected duration that the at least one of the identified two or more mobile devices will have sufficient power to playback the audio signals rendered from the audio source data, determine a source audio duration that indicates a playback duration of the audio signals from the audio source data, and, when the source audio duration exceeds the expected power duration, determining the expected power duration as the constraint.

12. The headend device of claim 11, wherein the one or more processors are configured to render the audio source data using the gain to generate the audio signals such that an expected power duration to playback the audio signals is less than the source audio duration.

13. The headend device of claim 10,

wherein the one or more processors are configured to determine a frequency dependent constraint, and

wherein the one or more processors are configured to render the audio source data using the determined frequency dependent constraint to generate the audio signals such that an expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the source audio data indicating a playback duration of the audio signals.

14. The headend device of claim 10,

wherein the virtual speaker comprises one of a plurality of virtual speakers of the collaborative surround sound system,

wherein the at least one of the identified two or more mobile devices comprises one of a plurality of mobile devices configured to support the plurality of virtual speakers,

wherein the one or more processors are configured to render the audio source data using an expected power duration, as the constraint to generate the audio signals, to playback the audio signals by the at least one of the identified two or more mobile devices such that the expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the source audio.

15. The headend device of claim 10,

wherein the plurality of mobile devices comprise a first mobile device, a second mobile device and a third mobile device,

wherein the virtual speaker comprises one of a plurality of virtual speaker of the collaborative surround sound system,

wherein the constraint comprises one or more expected power duration, the one or more expected power durations each indicating an expected duration that one of the plurality of mobile devices will have sufficient power to playback audio signals rendered from the audio source, and

wherein the one or more processors are configured to compute volume gains g₁, g₂ and g₃ for the first mobile device, the second mobile device and the third mobile device, respectively, in accordance with the following equation:

16. The headend device of claim 10, wherein the one or more processors are configured to perform a constrained vector based dynamic amplitude panning with respect to the audio source data to generate the audio signals so as to reduce the impact of the determined constraint on playback of the audio signals by the at least one of the identified two or more mobile device.

17. The headend device of claim 10, wherein the virtual speaker of the collaborative surround sound system appears to be placed in a location different than a location of the identified two or more mobile devices.

18. The headend device of claim 10, wherein the audio source data comprises one of a higher order ambisonic audio source data, a multi-channel audio source data and an object-based audio source data.

19. A headend device comprising:

means for identifying two or more mobile devices of a plurality of mobile devices participating in a collaborative surround sound system capable of representing a virtual speaker of the collaborative surround sound system;

means for determining a constraint that impacts playback of audio signals rendered from audio source data by at least one of the identified two or more mobile devices;

means for determining, based on the constraint, a gain for the at least one of the identified two or more mobile devices; and

means for rendering the audio source data using the gain to generate audio signals that reduce the impact of the determined constraint during playback of the audio signals by the identified two or more mobile devices.

20. The headend device of claim 19, wherein the means for determining the constraint comprises:

means for determining an expected power duration that indicates an expected duration that the at least one of the identified two or more mobile devices will have sufficient power to playback the audio signals rendered from the audio source data;

means for determining a source audio duration that indicates a playback duration of the audio signals rendered from the audio source data; and

means for determining, when the source audio duration exceeds the expected power duration, the expected power duration as the constraint.

21. The headend device of claim 20, wherein the means for rendering the audio source data comprises means for rendering the audio source data using the gain to generate the audio signals such that an expected power duration to playback the audio signals is less than a duration of the audio source data.

22. The headend device of claim 20,

wherein the means for determining the constraint comprise means for determining a frequency dependent constraint, and

wherein the means for rendering comprises means for rendering the audio source data using the at least one gain to generate the audio signals such that an expected power duration to playback the audio signals by the at least one of the identified two or more mobile devices is less than a duration of the audio source data.

23. The headend device of claim 19,

wherein the means for rendering comprises means for performing dynamic spatial rendering of the audio source data using an expected power duration, as the constrain to generate the audio signals, to playback the audio signals by the at least one of the identified two or more mobile devices such that the expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the source audio data.

24. The headend device of claim 19,

wherein the plurality of mobile devices comprise a first mobile device, a second mobile device and a third mobile device,

wherein the virtual speaker comprises one of a plurality of virtual speakers of the collaborative surround sound system,

wherein the constraint comprises one or more expected power durations, the one or more expected power durations each indicating an expected duration that one of the plurality of mobile devices will have sufficient power to playback audio signals rendered from the audio source, and

wherein the means for determining the gain for the at least one of the identified two or more mobile devices comprises:

means for computing volume gains g₁, g₂ and g₃ for the first mobile device, the second mobile device and the third mobile device, respectively, in accordance with the following equation;

25. The headend device of claim 19, wherein the means for rendering comprises means for performing a constrained vector based dynamic amplitude panning with respect to the audio source data to generate the audio signals so as to reduce the impact of the determined constraint on playback of the audio signals by the at least one of the identified two or more mobile devices.

26. The headend device of claim 19, wherein the virtual speaker of the collaborative surround sound system appears to be placed in a location different than a location of at least one of the two or more mobile devices.

27. The headend device of claim 19, wherein the audio source data comprises one of higher order ambisonic audio source data, a multi-channel audio source data and an object-based audio source data.

28. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed cause one or more processors to:

identify two or more mobile devices of a plurality of mobile devices participating in a collaborative surround sound system capable of representing a virtual speaker of the collaborative surround sound system;

determine a constraint that impacts playback of audio signals rendered from audio source data by at least one of the identified two or more mobile devices;

determine, based on the constraint, a gain for the at least one of the identified two or more mobile devices; and

render the audio source data using the gain to generate audio signals that reduce the impact of the determined constraint during playback of the audio signals by the plurality of mobile devices.

29. The non-transitory computer-readable storage medium of claim 28, wherein the instructions further cause, when executed, the one or more processors to, when determining the constraint, determine an expected power duration that indicates an expected duration that the at least one of the identified two or more mobile devices will have sufficient power to playback the audio signals rendered from the audio source data, determine a source audio duration that indicates a playback duration of the audio signals rendered from the audio source data, and, when the source audio duration exceeds the expected power duration, determining the expected power duration as the constraint.

30. The non-transitory computer-readable storage medium of claim 29, wherein the instructions further cause, when executed, the one or more processors to, when rendering the audio source data with the determined constraint, render the audio source data using the gain to generate the audio signals such that an expected power duration to playback the audio signals is less than a duration of the audio source data.

31. The non-transitory computer-readable storage medium of claim 28,

wherein the instructions further cause, when executed, the one or more processors to, when determining the constraint, determine a frequency dependent constraint, and

wherein the instructions further cause, when executed, the one or more processors to, when rendering, render the audio source data using the gain to generate the audio signals such that an expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the audio source data.

32. The non-transitory computer-readable storage medium of claim 28,

wherein the instructions further cause, when executed, the one or more processors to, when rendering, render the audio source data using an expected power duration, as the constraint to render the audio signals, to playback the audio signals by the at least one of the identified two or more mobile devices such that the expected power duration to playback the audio signals by the at least one of the identified two or more of the mobile devices is less than a duration of the audio source data.

33. The non-transitory computer-readable storage medium of claim 28,

wherein the plurality of mobile devices comprise a first mobile device, a second mobile device and a third mobile device,

wherein the virtual speaker comprises one of a plurality of virtual speakers of the collaborative surround sound system,

wherein the constraint comprises one or more expected power duration, the one or more expected power duration each indicating an expected duration that one of the plurality of mobile devices will have sufficient power to playback audio signals rendered from the audio source data, and

wherein the instructions further cause, when executed, the one or more processors to, when determining the gain for the at least one of the two or more mobile devices, compute volume gains g₁, g₂ and g₃ for the first mobile device, the second mobile device and the third mobile device, respectively, in accordance with the following equation:

34. The non-transitory computer-readable storage medium of claim 28, wherein the instructions further cause, when executed, the one or more processors to, when rendering the audio source data using the gain, perform a constrained vector based dynamic amplitude panning with respect to the audio source data to generate the audio signals so as to reduce the impact of the determined constraint on playback of the audio signals by the at least one of the two or more mobile devices.

35. The non-transitory computer-readable storage medium of claim 28, wherein the virtual speaker of the collaborative surround sound system appears to be placed in a location different than a location of at least one of the two or more mobile devices.

36. The non-transitory computer-readable storage medium of claim 28, wherein the audio source data comprises one of a higher order ambisonic audio source data, a multi-channel audio source data and an object-based audio source data.