System and tools for enhanced 3D audio authoring and rendering

• US 9,204,236 B2
• Filed: 06/27/2012
• Issued: 12/01/2015
• Est. Priority Date: 07/01/2011
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Tsingos, Nicolas R., Robinson, Charles Q., Scharpf, Jurgen W.
• Primary Examiner(s): King, Simon
• Application Number:
  US14/126,901
• Publication Number:
  US 20140119581A1
• Time in Patent Office: 1,252 Days
• Field of Search: 381/300
• US Class Current: 1/1
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 3/00 Systems employing more than two channels, e.g. quadraphonic H04S5/00, H04S7/00 take precedence
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence
• CPC Class Codes: H04S 7/308 Electronic adaptation dependent on speaker or headphone connection
• CPC Class Codes: H04S 7/40 Visual indication of stereophonic sound image

Abstract

Improved tools for authoring and rendering audio reproduction data are provided. Some such authoring tools allow audio reproduction data to be generalized for a wide variety of reproduction environments. Audio reproduction data may be authored by creating metadata for audio objects. The metadata may be created with reference to speaker zones. During the rendering process, the audio reproduction data may be reproduced according to the reproduction speaker layout of a particular reproduction environment.

Claims

1. An apparatus, comprising:

an interface system; and

a logic system configured for;

receiving, via the interface system, audio reproduction data comprising one or more audio objects and associated metadata;

wherein the associated metadata includes trajectory data for at least one of the one or more audio objects indicating a time-variable audio object position of the at least one audio object within the three-dimensional space;

wherein the audio object position is constrained to a two-dimensional surface;

wherein the audio reproduction data has been created with respect to a virtual reproduction environment comprising a plurality of speaker zones at different elevations;

receiving, via the interface system, reproduction environment data comprising an indication of a number of reproduction speakers of an actual three-dimensional reproduction environment and an indication of the location of each reproduction speaker within the actual reproduction environment;

mapping the audio reproduction data created with reference to the plurality of speaker zones of the virtual reproduction environment to the reproduction speakers of the actual reproduction environment; and

rendering the one or more audio objects into one or more speaker feed signals based, at least in part, on the associated metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the actual reproduction environment.

2. The apparatus of claim 1, wherein the rendering involves creating a gain based on one or more of a desired audio object position, a distance from the desired audio object position to a reference position, a velocity of an audio object or an audio object content type.

3. The apparatus of claim 1, wherein the two-dimensional surface comprises one of a spherical surface, an elliptical surface, a conical surface, a cylindrical surface or a wedge.

4. The apparatus of claim 1, wherein the rendering involves imposing speaker zone constraints including data for disabling selected reproduction speakers.

5. The apparatus of claim 1, wherein the actual reproduction environment comprises a screen for the projection of video images;

wherein the audio reproduction data is synchronized with the video images; and

wherein the rendering involves applying screen-to-room balance control according to screen-to-room balance control data received from the user input system.

6. The apparatus of claim 1, wherein the rendering involves controlling a spread in one or more of three dimensions of an audio object over multiple reproduction speakers.

7. The apparatus of claim 1, wherein the rendering involves dynamic object blobbing in response to speaker overload, by directing audio energy to an increased number of neighboring reproduction speakers while maintaining overall constant energy.

8. The apparatus of claim 1, wherein the rendering involves mapping audio object positions to planes of speaker arrays of the actual reproduction environment.

9. The apparatus of claim 1, wherein the logic system is configured to determine whether to apply panning rules for an audio object position to a plurality of speaker locations or to map an audio object position to a single speaker location.

10. The apparatus of claim 9, wherein the logic system is configured to smooth transitions in speaker gains when transitioning from mapping an audio object position from a first single speaker location to a second single speaker location.

11. The apparatus of claim 9, wherein the logic system is configured to smooth transitions in speaker gains when transitioning between mapping an audio object position to a single speaker location and applying panning rules for the audio object position to the plurality of speaker locations.

12. A method, comprising:

receiving audio reproduction data comprising one or more audio objects and associated metadata;

wherein the associated metadata includes trajectory data for at least one of the one or more audio objects indicating a time-variable audio object position of the at least one audio object within the three-dimensional space;

wherein the audio object position is constrained to a two-dimensional surface;

wherein the audio reproduction data has been created with respect to a virtual reproduction environment comprising a plurality of speaker zones at different elevations;

receiving reproduction environment data comprising an indication of a number of reproduction speakers in an actual reproduction environment and an indication of the location of each reproduction speaker of the three-dimensional actual reproduction environment;

mapping the audio reproduction data created with reference to the plurality of speaker zones of the virtual reproduction environment to the reproduction speakers of the actual reproduction environment; and

rendering the one or more audio objects into one or more speaker feed signals based, at least in part, on the associated metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the actual reproduction environment.

13. A non-transitory medium having software stored thereon, the software including instructions for performing the following operations:

receiving audio reproduction data comprising one or more audio objects and associated metadata;

wherein the associated metadata includes trajectory data for at least one of the one or more audio objects indicating a time-variable audio object position of the at least one audio object within the three-dimensional space;

wherein the audio object position is constrained to a two-dimensional surface;

wherein the audio reproduction data has been created with respect to a virtual reproduction environment comprising a plurality of speaker zones at different elevations;

receiving reproduction environment data comprising an indication of a number of reproduction speakers in an actual reproduction environment and an indication of the location of each reproduction speaker of the three-dimensional actual reproduction environment;

mapping the audio reproduction data created with reference to the plurality of speaker zones of the virtual reproduction environment to the reproduction speakers of the actual reproduction environment; and

rendering the one or more audio objects into one or more speaker feed signals based, at least in part, on the associated metadata, wherein each speaker feed signal corresponds to at least one of the reproduction speakers within the actual reproduction environment.

14. An apparatus (2100) for authoring an audio object, the apparatus (2100) comprising:

an interface system (2105);

a user input system (2135);

a display system (2130); and

a logic system (2110) configured for;

receiving audio data via the interface system;

displaying a virtual reproduction environment in a graphical user interface on the display system (2130);

wherein the virtual reproduction environment comprises a plurality of speaker zones at different elevations;

receiving user input regarding a position of an audio object via the user input system;

determining trajectory data indicating a time-variable position of the audio object in a three-dimensional space according to the user input received via the user input system, wherein the determining involves constraining the time-variable position to a two-dimensional surface within the three-dimensional space;

wherein the audio object comprises the audio data;

displaying an audio object trajectory according to the trajectory data in the graphical user interface; and

creating metadata associated with the audio data object;

wherein the metadata includes the trajectory data.

15. The apparatus of claim 14, wherein the two-dimensional surface comprises one of a spherical surface, an elliptical surface, a conical surface, a cylindrical surface or a wedge.

16. The apparatus of claim 14, wherein the trajectory data comprises a set of positions within the three-dimensional space at multiple time instances.

17. The apparatus of claim 14, wherein the plurality of speaker zones correspond to reproduction speakers of an actual three-dimensional reproduction environment comprising reproduction speakers, or wherein the plurality of speaker zones correspond to virtual speakers of a virtual surround sound environment.

18. The apparatus of claim 14, wherein an increased elevation of the audio object is indicated in the graphical user interface by an increase in a diameter of a circle that represents the audio object in the graphical user interface.

19. A method, comprising:

receiving audio data;

displaying a virtual reproduction environment in a graphical user interface on a display system;

wherein the virtual reproduction environment comprises a plurality of speaker zones at different elevations;

receiving user input regarding a position of an audio object;

determining trajectory data indicating a time-variable position of the audio object in a three-dimensional space, wherein the determining involves constraining the position to a two-dimensional surface within the three-dimensional space;

wherein the audio object comprises the audio data;

displaying an audio object trajectory according to the trajectory data in the graphical user interface; and

creating metadata associated with the audio object;

wherein the metadata includes the trajectory data.

20. A non-transitory medium having software stored thereon, the software including instructions for performing the following operations:

receiving audio data;

displaying a virtual reproduction environment in a graphical user interface on a display system;

wherein the virtual reproduction environment comprises a plurality of speaker zones at different elevations;

receiving user input regarding a position of an audio object;

determining trajectory data indicating a time-variable position of the audio object in a three-dimensional space, wherein the determining involves constraining the position to a two-dimensional surface within the three-dimensional space;

wherein the audio object comprises the audio data;

displaying an audio object trajectory according to the trajectory data in the graphical user interface; and

creating metadata associated with the audio object;

wherein the metadata includes the trajectory data.