THREE-DIMENSIONAL AUDIO RENDERING TECHNIQUES

• US 20150131966A1
• Filed: 06/30/2014
• Published: 05/14/2015
• Est. Priority Date: 11/11/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Zurek, Robert A., Merrell, Thomas Y.
• Granted Patent: US 9,888,333 B2
• Time in Patent Office: Days
• US Class Current: 386/241
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/13 Application of wave-field synthesis in stereophonic audio systems
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02

Abstract

Three-dimensional (3D) audio content creation and rendering systems and methodologies are presented here. A disclosed method of processing 3D audio assigns audio source objects to 3D video objects, links audio tracks to assigned audio source objects, and performs wave field synthesis on the linked audio tracks to generate 3D audio data representing a 3D spatial sound field. A disclosed method of processing 3D audio during playback of 3D video content obtains 3D audio data and 3D video data for a frame of 3D video content, applies device-specific parameters to the 3D audio data to obtain transformed 3D audio data scaled to a presentation device, and processes the transformed 3D audio data to render audio information for an array of speakers associated with the presentation device.

Claims

1. A method of processing three-dimensional (3D) audio for 3D video content having 3D video objects, the method comprising:

assigning at least one audio source object to each 3D video object in the 3D video content;

linking a respective audio track to each assigned audio source object, resulting in a plurality of linked audio tracks; and

performing wave field synthesis on the plurality of linked audio tracks to generate 3D audio data representing a 3D spatial sound field corresponding to the 3D video content.

2. The method of claim 1, further comprising:

calculating a normalized screen size transform matrix for the 3D audio data, based on dimensions of a virtual 3D portal of the 3D video content.

3. The method of claim 1, wherein the wave field synthesis is performed on a frame-by-frame basis to generate 3D audio data for each video frame of the 3D video content.

4. The method of claim 1, further comprising:

writing the 3D audio data in association with corresponding 3D video data.

5. The method of claim 4, wherein the writing comprises writing the 3D audio data to a random access memory (RAM) element to facilitate real-time rendering of the 3D spatial sound field.

6. The method of claim 4, wherein the writing comprises writing the 3D audio data to a nonvolatile memory element to facilitate on-demand rendering of the 3D spatial sound filed.

7. The method of claim 1, further comprising:

determining interactive audio content caused by interaction between at least two of the 3D video objects, wherein the generated 3D audio data is influenced by the interactive audio content.

8. The method of claim 1, further comprising:

compiling and mixing the plurality of linked audio tracks, wherein the generated 3D audio data is influenced by the compiling and mixing.

9. The method of claim 1, further comprising:

compiling and determining audio source locations for the 3D video objects, wherein the generated 3D audio data is influenced by the compiling and determining.

10. The method of claim 1, wherein the 3D audio data corresponds to one audio stream for each audio source object tied to a 3D video object.

11. The method of claim 1, wherein the 3D audio data represents the 3D spatial sound field in a manner that is independent of physical display screen dimensions of a presentation device.

12. The method of claim 1, wherein the 3D audio data represents the 3D spatial sound field in a manner that is independent of speaker configuration of a presentation device.

13. The method of claim 1, further comprising:

defining acoustic characteristics for at least some of the 3D video objects.

14. A tangible and non-transitory computer readable medium having computer-executable instructions stored thereon and capable of performing a method when executed by a processor, the method comprising:

assigning at least one audio source object to each three-dimensional (3D) video object in 3D video content;

linking a respective audio track to each assigned audio source object, resulting in a plurality of linked audio tracks; and

performing wave field synthesis on the plurality of linked audio tracks to generate 3D audio data representing a 3D spatial sound field corresponding to the 3D video content.

15. The computer readable medium of claim 14, wherein the method performed by the computer-executable instructions further comprises:

calculating a normalized screen size transform matrix for the 3D audio data, based on dimensions of a virtual 3D portal of the 3D video content.

16. The computer readable medium of claim 14, wherein the method performed by the computer-executable instructions further comprises:

compiling and determining audio source locations for the 3D video objects, wherein the generated 3D audio data is influenced by the compiling and determining.

17. The computer readable medium of claim 14, wherein the method performed by the computer-executable instructions further comprises:

defining acoustic characteristics for at least some of the 3D video objects.

18. A computing system comprising:

at least one processor; and

memory having computer-executable instructions stored thereon that, when executed by the at least one processor, cause the computing system to;

assign at least one audio source object to each three-dimensional (3D) video object in 3D video content;

link a respective audio track to each assigned audio source object, resulting in a plurality of linked audio tracks; and

perform wave field synthesis on the plurality of linked audio tracks to generate 3D audio data representing a 3D spatial sound field corresponding to the 3D video content.

19. The computing system of claim 18, wherein the computer-executable instructions, when executed by the at least one processor, cause the computing system to:

calculate a normalized screen size transform matrix for the 3D audio data, based on dimensions of a virtual 3D portal of the 3D video content.

20. The computing system of claim 18, wherein the computer-executable instructions, when executed by the at least one processor, cause the computing system to:

compile and determine audio source locations for the 3D video objects, wherein the generated 3D audio data is influenced by the compiling and determining.

21. The computing system of claim 18, wherein the computer-executable instructions, when executed by the at least one processor, cause the computing system to:

define acoustic characteristics for at least some of the 3D video objects.

22. A method of processing three-dimensional (3D) audio for 3D video content having 3D video objects, the method comprising:

obtaining 3D audio data and 3D video data for a frame of the 3D video content;

applying device-specific parameters to the obtained 3D audio data to obtain transformed 3D audio data that is scaled to a host presentation device; and

processing the transformed 3D audio data to render audio information for an array of speakers associated with the host presentation device.

23. The method of claim 22, wherein:

the processing results in a respective channel of audio information for each speaker in the array of speakers.

24. The method of claim 22, wherein the 3D audio data comprises a plurality of wave field synthesis coefficients that represent a 3D spatial sound field.

25. The method of claim 22, further comprising:

obtaining a normalized screen size transform matrix in association with the 3D audio data for the frame;

calculating a device-specific screen size transform matrix from the normalized screen size transform matrix and a physical screen size of the host presentation device; and

applying the calculated screen size transform matrix to the obtained 3D audio data to obtain the transformed 3D audio data.

26. The method of claim 25, wherein the device-specific parameters defines the physical screen size of the host presentation device.

27. The method of claim 25, wherein characteristics of the obtained normalized screen size transform matrix are influenced by dimensions of a virtual 3D portal for the frame of the 3D video content.

28. The method of claim 22, wherein the device-specific parameters defines a speaker configuration for the array of speakers.

29. The method of claim 28, wherein the speaker configuration identifies a number of speakers contained in the array of speakers.

30. The method of claim 28, wherein the speaker configuration identifies positions of speakers contained in the array of speakers.

31. The method of claim 22, wherein processing the transformed 3D audio data comprises:

calculating an audio mixing matrix for each speaker contained in the array of speakers, and rendering audio information for each speaker in accordance with the audio mixing matrix.

32. A tangible and non-transitory computer readable medium having computer-executable instructions stored thereon and capable of performing a method when executed by a processor, the method comprising:

obtaining three-dimensional (3D) audio data and 3D video data for a frame of 3D video content;

applying device-specific parameters to the obtained 3D audio data to obtain transformed 3D audio data that is scaled to a host presentation device; and

processing the transformed 3D audio data to render audio information for an array of speakers associated with the host presentation device.

33. The computer readable medium of claim 32, wherein the 3D audio data comprises a plurality of wave field synthesis coefficients that represent a 3D spatial sound field.

34. The computer readable medium of claim 32, wherein the method performed by the computer-executable instructions further comprises:

obtaining a normalized screen size transform matrix in association with the 3D audio data for the frame;

calculating a device-specific screen size transform matrix from the normalized screen size transform matrix and a physical screen size of the host presentation device; and

applying the calculated screen size transform matrix to the obtained 3D audio data to obtain the transformed 3D audio data.

35. The computer readable medium of claim 32, wherein processing the transformed 3D audio data comprises:

calculating an audio mixing matrix for each speaker contained in the array of speakers, and rendering audio information for each speaker in accordance with the audio mixing matrix.

36. An audio/video presentation device comprising:

an array of speakers;

at least one processor; and

memory having computer-executable instructions stored thereon that, when executed by the at least one processor, cause the audio/video presentation device to;

obtain three-dimensional (3D) audio data and 3D video data for a frame of 3D video content;

apply device-specific parameters to the obtained 3D audio data to obtain transformed 3D audio data that is scaled to the presentation device; and

process the transformed 3D audio data to render audio information for the array of speakers.

37. The audio/video presentation device of claim 36, wherein:

processing the transformed 3D audio data results in a respective channel of audio information for each speaker in the array of speakers.

38. The audio/video presentation device of claim 36, wherein the 3D audio data comprises a plurality of wave field synthesis coefficients that represent a 3D spatial sound field.

39. The audio/video presentation device of claim 36, wherein the computer-executable instructions, when executed by the at least one processor, cause the audio/video presentation device to:

obtain a normalized screen size transform matrix in association with the 3D audio data for the frame;

calculate a device-specific screen size transform matrix from the normalized screen size transform matrix and a physical screen size of the audio/video presentation device; and

apply the calculated screen size transform matrix to the obtained 3D audio data to obtain the transformed 3D audio data.

40. The audio/video presentation device of claim 39, wherein the device-specific parameters define the physical screen size of the audio/video presentation device.

41. The audio/video presentation device of claim 39, wherein characteristics of the obtained normalized screen size transform matrix are influenced by dimensions of a virtual 3D portal for the frame of the 3D video content.

42. The audio/video presentation device of claim 36, wherein the device-specific parameters define a speaker configuration for the array of speakers.

43. The audio/video presentation device of claim 36, wherein processing the transformed 3D audio data comprises:

calculating an audio mixing matrix for each speaker contained in the array of speakers, and rendering audio information for each speaker in accordance with the audio mixing matrix.