Method, system and article of manufacture for processing spatial audio

• US 9,578,439 B2
• Filed: 07/23/2015
• Issued: 02/21/2017
• Est. Priority Date: 01/02/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Lae-Hoon, Peri, Raghuveer, Visser, Erik
• Primary Examiner(s): HUBER, PAUL W
• Application Number:
  US14/807,760
• Publication Number:
  US 20160198282A1
• Time in Patent Office: 579 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/301 Automatic calibration of stereophonic sound system, e.g. with test microphone

Abstract

Techniques for processing directionally-encoded audio to account for spatial characteristics of a listener playback environment are disclosed. The directionally-encoded audio data includes spatial information indicative of one or more directions of sound sources in an audio scene. The audio data is modified based on input data identifying the spatial characteristics of the playback environment. The spatial characteristics may correspond to actual loudspeaker locations in the playback environment. The directionally-encoded audio may also be processed to permit focusing/defocusing on sound sources or particular directions in an audio scene. The disclosed techniques may allow a recorded audio scene to be more accurately reproduced at playback time, regardless of the output loudspeaker setup. Another advantage is that a user may dynamically configure audio data so that it better conforms to the user'"'"'s particular loudspeaker layouts and/or the user'"'"'s desired focus on particular subjects or areas in an audio scene.

Claims

1. A method of processing audio, comprising:

receiving, at a device, audio data corresponding to a scene;

receiving a selection distinguishing one or more enabled regions and one or more disabled regions in the scene;

determining, based on the audio data, spatial information indicative of one or more directions of one or more sound sources in the scene; and

modifying the audio data based on the selection, based on the spatial information, and based on input data identifying one or more spatial characteristics of a playback environment, wherein the modifying includes applying one or more gains based on a masking window function.

2. The method of claim 1, wherein the one or more gains include a cumulative gain for a particular audio channel, and further comprising:

determining the cumulative gain for the particular audio channel, wherein the cumulative gain is based on a sum of gains associated with enabled regions corresponding to the particular audio channel, wherein modifying the audio data includes applying the cumulative gain to a portion of the audio data corresponding to the particular audio channel.

3. The method of claim 1, wherein the selection is based on an operational mode of the device.

4. The method of claim 3, wherein the operational mode of the device is selected from the group consisting of front camera enabled and back camera enabled.

5. The method of claim 1, wherein the device includes a camera and the selection is based on a zoom operation of the camera.

6. The method of claim 1, further comprising:

providing a user interface configured to permit a user to select the one or more enabled regions in the scene, wherein the selection is received via the user interface.

7. The method of claim 1, further comprising:

receiving the input data through a user interface that permits a user to configure the input data according to the one or more spatial characteristics of the playback environment.

8. The method of claim 1, wherein the input data includes a sector definition indicating a region in the playback environment.

9. The method of claim 8, wherein the sector definition corresponds to a loudspeaker location in the playback environment.

10. The method of claim 8, wherein the masking window function is based on the selection and the sector definition.

11. The method of claim 1, wherein the input data identifies a plurality of regions in the playback environment, and further comprising mapping a single region of the enabled regions to multiple regions of the plurality of regions, wherein modifying the audio data based on the selection and based on the input includes modifying the audio data based at least in part on the mapping.

12. An apparatus, comprising:

an interface configured to receive audio data corresponding to a scene; and

a processor configured to;

determine, based on the audio data, spatial information indicative of one or more directions of one or more sound sources in the scene; and

modify the audio data based on a selection distinguishing one or more enabled regions and one or more disabled regions in the scene, based on the spatial information, and based on input data identifying one or more spatial characteristics of a playback environment, wherein the processor is configured to modify the audio data at least in part by applying one or more gains based on a masking window function.

13. The apparatus of claim 12, further comprising a second interface configured to receive the selection.

14. The apparatus of claim 12, wherein the selection is based on an operational mode of the apparatus.

15. The apparatus of claim 12, further comprising a camera, wherein the selection is based on a zoom operation of the camera.

16. The apparatus of claim 12, further comprising a user interface configured to permit a user to select the one or more enabled regions in the scene to provide the selection.

17. The apparatus of claim 12, further comprising:

a user interface to permit a user to configure the input data according to the one or more spatial characteristics of the playback environment.

18. The apparatus of claim 12, wherein the input data includes a sector definition indicating a region in the playback environment, and wherein the sector definition corresponds to a loudspeaker location in the playback environment.

19. The apparatus of claim 18, wherein the masking window function is based on the selection and the sector definition.

20. The apparatus of claim 12, further comprising:

a module configured to render the modified audio data for playback.

21. An apparatus, comprising:

means for receiving audio data corresponding to a scene;

means for receiving a selection distinguishing one or more enabled regions and one or more disabled regions in the scene;

means for determining, based on the audio data, spatial information indicative of one or more directions of one or more sound sources in the scene; and

means for modifying the audio data based on the selection, based on the spatial information, and based on input data identifying one or more spatial characteristics of a playback environment, wherein the means for modifying is configured to modify the audio data at least in part by applying one or more gains based on a masking window function.

22. The apparatus of claim 21, wherein the one or more gains include a cumulative gain for a particular audio channel, and further comprising:

means for determining the cumulative gain for the particular audio channel, wherein the cumulative gain is based on a sum of gains associated with enabled regions corresponding to the particular audio channel, wherein the means for modifying the audio data is configured to apply the cumulative gain to a portion of the audio data corresponding to the particular audio channel.

23. The apparatus of claim 21, wherein the selection is based on an operational mode of the apparatus.

24. The apparatus of claim 21, further comprising a camera, wherein the selection is based on a zoom operation of the camera.

25. The apparatus of claim 21, wherein the means for receiving a selection includes means for providing a user interface configured to permit a user to select the one or more enabled regions in the scene.

26. The apparatus of claim 21, further comprising:

means for receiving the input data through a user interface that permits a user to configure the input data according to the one or more spatial characteristics of the playback environment.

27. The apparatus of claim 21, wherein the input data includes a sector definition indicating a region in the playback environment.

28. The apparatus of claim 27, wherein the sector definition corresponds to a loudspeaker location in the playback environment.

29. The apparatus of claim 27, wherein the masking window function is based on the selection and the sector definition.

30. A non-transient computer-readable medium embodying a set of instructions executable by one or more processors, comprising:

code for receiving audio data corresponding to a scene;

code for receiving a selection distinguishing one or more enabled regions and one or more disabled regions in the scene;

code for determining, based on the audio data, spatial information indicative of one or more directions of one or more sound sources in the scene; and

code for modifying the audio data based on the selection, based on the spatial information, and based on input data identifying one or more spatial characteristics of a playback environment, wherein the modifying includes applying one or more gains based on a masking window function.