Smooth rendering of overlapping audio-object interactions

• US 10,531,219 B2
• Filed: 03/20/2017
• Issued: 01/07/2020
• Est. Priority Date: 03/20/2017
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Laaksonen, Lasse Juhani
• Primary Examiner(s): Krzystan, Alexander
• Application Number:
  US15/463,513
• Publication Number:
  US 20180270602A1
• Time in Patent Office: 1,023 Days
• Field of Search: 381306, 381310, 381 59, 381303, 381 22, 381 23
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A method including, detecting an overlap between at least two waveform renderings, wherein the at least two waveform renderings comprise an audio object, determining at least one difference between the at least two waveform renderings for the audio object when the overlap is detected, determining a rendering modification decision for the audio object associated with the at least one difference, processing at least one of the at least two waveform renderings dependent on the rendering modification decision so as to introduce an effect related to the determined at least one difference, and performing a modified rendering with the processed at least one of the at least two waveform renderings comprising the effect for the audio object.

Claims

1. A method comprising:

detecting an overlap between at least two instruction sets, where the at least two instruction sets are simultaneously applicable for determining waveform renderings of a same audio object;

determining at least one difference between at least two of the waveform renderings of the same audio object, where the determining of the at least one difference is determined with the at least two instruction sets when the overlap is detected;

determining a rendering modification decision for the same audio object, where the rendering modification decision is based, at least partially, on the determined at least one difference; and

during rendering of the same audio object with at least one of the at least two instruction sets, applying a modification to a waveform rendering determined with the at least one of the at least two instruction sets, where the modification is dependent on the rendering modification decision so as to introduce an effect related to the determined at least one difference.

2. The method of claim 1, where the determining of the rendering modification decision for the same audio object further comprises:

determining the rendering modification decision based on one of a handover or an interpolation between the at least two waveform renderings, wherein the handover selects one of the at least two waveform renderings, and wherein the interpolation combines effects associated with the at least two waveform renderings.

3. The method of claim 2, where the at least two waveform renderings comprises a first waveform rendering and a second waveform rendering, and where the determining of the rendering modification decision further comprises:

receiving state and parameters based on at least one of an audio object location or an audio object playback time for the same audio object for each of the first waveform rendering and the second waveform rendering;

wherein the determining of the at least one difference between the at least two waveform renderings further comprises at least one of;

determining a difference between a first state for generating the first waveform rendering and a second state for generating the second waveform rendering, or

determining a difference between a first parameter for generating the first waveform rendering and a second parameter for generating the second waveform rendering;

comparing the determined at least one difference to a predetermined threshold;

selecting the handover from one of;

a first instruction set of the at least two instruction sets configured to determine the first waveform rendering and a second instruction set of the at least two instruction sets configured to determine the second waveform rendering, and a third instruction set of the at least two instruction sets configured to determine the first waveform rendering and a fourth instruction set of the at least two instruction sets configured to determine the second waveform rendering,

in response to a determination that the determined at least one difference is greater than the predetermined threshold; and

selecting the interpolation between the first waveform rendering and the second waveform rendering in response to a determination that the determined at least one difference is less than the predetermined threshold.

4. The method of claim 1, where the at least two waveform renderings, further comprise:

a rendering associated with at least one user position; and

a rendering associated with at least one spatial audio extension.

5. The method of claim 4, where an extension point associated with the at least one spatial audio extension is defined relative to the at least one user position.

6. The method of claim 4, where an extension point associated with the at least one spatial audio extension is defined independent of the at least one user position.

7. The method of claim 3, where the parameters for each of the at least two waveform renderings include at least one of audio object size, directivity, or audio waveform filterings.

8. The method of claim 1, further comprising:

detecting an interaction for each of the at least two waveform renderings prior to the detecting of the overlap;

determining an audio object state modification based on a change in the interaction.

9. The method of claim 8, where the change in the interaction comprises a decrease and the audio object state modification comprises an adjustment based on reversibility.

10. The method of claim 8, where the change in the interaction comprises an increase and the audio object state modification comprises an adjustment based on effective distance.

11. The method of claim 8, further comprising determining the audio object state modification based on the rendering modification decision.

12. The method of claim 1, where the at least two waveform renderings of the same audio object comprises one of:

at least two simultaneous waveform renderings, where each of the at least two simultaneous waveform renderings is determined with an instruction set of the at least two instruction sets that is applicable for determining a waveform rendering of a single instance of the same audio object, that are to be fused into a single rendering without discontinuities or artefacts, or

at least two simultaneous waveform renderings, where each of the at least two simultaneous waveform renderings is determined with an instruction set of the at least two instruction sets that is applicable for determining a waveform rendering of one of at least two instances of the same audio object, that are to be fused into a single rendering without discontinuities or artefacts.

13. The method of claim 1, where the determining of the at least one difference between the at least two waveform renderings when the overlap is detected further comprises:

determining the at least one difference based on at least one of a difference in spatial position of the at least two waveform renderings or a difference in playtime of a playback of the at least two waveform renderings.

14. An apparatus comprising:

at least one processor; and

at least one non-transitory memory including computer program code, the at least one non-transitory memory and the computer program code configured to, with the at least one processor, cause the apparatus to;

detect an overlap between at least two instruction sets, where the at least two instruction sets are simultaneously applicable for determining waveform renderings of a same audio object;

determine at least one difference between at least two of the waveform renderings of the same audio object, where the determining of the at least one difference is determined with the at least two instruction sets when the overlap is detected;

determine a rendering modification decision for the same audio object, where the rendering modification decision is based, at least partially, on the determined at least one difference; and

during rendering of the same audio object with at least one of the at least two instruction sets, applying a modification to a waveform rendering determined with the at least one of the at least two instruction sets, where the modification is dependent on the rendering modification decision so as to introduce an effect related to the determined at least one difference.

15. An apparatus as in claim 14, where, when determining the rendering modification decision for the same audio object, the at least one non-transitory memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

determine the rendering modification decision based on one of a handover or an interpolation between the at least two waveform renderings configured to be determined with the at least two instruction sets, wherein the handover selects one of the at least two waveform renderings, and where the interpolation combines effects associated with the at least two waveform renderings.

16. An apparatus as in claim 15, where the at least two waveform renderings comprises a first waveform rendering and a second waveform rendering, and, when determining the rendering modification decision, the at least one non-transitory memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

receive state and parameters based on at least one of an audio object location or an audio object playback time for the same audio object for each of the first waveform rendering and the second waveform rendering;

wherein, to determine the at least one difference between the at least two waveform renderings further comprises at least one of;

to determine a difference between a first state for generating the first waveform rendering and a second state for generating the second waveform rendering, or

to determine a difference between a first parameter for generating the first waveform rendering and a second parameter for generating the second waveform rendering;

compare the determined at least one difference to a predetermined threshold;

select the handover from one of;

a first instruction set of the at least two instruction sets configured to determine the first waveform rendering and a second instruction set of the at least two instruction sets configured to determine the second waveform rendering, and a third instruction set of the at least two instruction sets configured to determine the first waveform rendering and a fourth instruction set of the at least two instruction sets configured to determine the second waveform rendering,

in response to a determination that the determined at least one difference is greater than the predetermined threshold; and

select the interpolation between the first waveform rendering and the second waveform rendering in response to a determination that the determined at least one difference is less than the predetermined threshold.

17. An apparatus as in claim 16, where the at least two waveform renderings further comprise:

a rendering associated with at least one user position; and

a rendering associated with at least one spatial audio extension.

18. An apparatus as in claim 14, where parameters for each of the at least two waveform renderings include at least one of audio object size, directivity, or audio waveform filterings.

19. An apparatus as in claim 14, where the at least one non-transitory memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

detect an interaction for each of the at least two waveform renderings prior to detecting the overlap; and

determine an audio object state modification based on a change in the interaction.

20. A non-transitory program storage device readable with a machine, tangibly embodying a program of instructions executable with the machine for performing operations, the operations comprising:

detecting an overlap between at least two instruction sets, where the at least two instruction sets are simultaneously applicable for determining waveform renderings of a same audio object;

determining at least one difference between at least two of the waveform renderings of the same audio object, where the determining of the at least one difference is determined with the at least two instruction sets when the overlap is detected;

determining a rendering modification decision for the same audio object, where the rendering modification decision is based, at least partially, on the determined at least one difference; and

during rendering of the same audio object with at least one of the at least two instruction sets, applying a modification to a waveform rendering determined with the at least one of the at least two instruction sets, where the modification is dependent on the rendering modification decision so as to introduce an effect related to the determined at least one difference.