Audio object modification in free-viewpoint rendering

• US 10,433,096 B2
• Filed: 03/08/2018
• Issued: 10/01/2019
• Est. Priority Date: 10/14/2016
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Laaksonen, Lasse Juhani
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US15/915,348
• Publication Number:
  US 20190082282A1
• Time in Patent Office: Days
• Field of Search: 381310, 381307, 381 57, 381 17- 18
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/03 Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 H04S2400/01 takes precedence
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/03 Application of parametric coding in stereophonic audio systems
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A method including, during a free-viewpoint rendering to a user, where the free-viewpoint rendering comprises rendering of an audio object, and during relative movement between a location of the user and the audio object in the free-viewpoint rendering, detecting a locational conflict abnormality between the audio object and the location of the user; and based upon the detecting of the locational conflict abnormality, changing the rendering of the audio object, where the changing is based upon at least one predetermined abnormality modification parameter.

Claims

1. An apparatus comprising:

at least one processor; and

at least one non-transitory memory including computer program code, the at least one non-transitory memory and the computer program code configured to, with the at least one processor, cause the apparatus to;

define audio object metadata associated with at least one audio object, wherein the audio object metadata includes at least one instruction for rendering the at least one audio object in a free-viewpoint rendering in response to detection of a locational conflict abnormality based upon at least one predetermined abnormality modification parameter; and

transmit the audio object metadata, wherein the audio object metadata is configured to modify rendering of the at least one audio object.

2. The apparatus as claimed in claim 1, wherein the at least one predetermined abnormality modification parameter comprises a location parameter, wherein the location parameter defines a location modification for the at least one audio object.

3. The apparatus as claimed in claim 1, wherein the at least one predetermined abnormality modification parameter comprises a scaling parameter, wherein the scaling parameter defines a size modification for the at least one audio object.

4. The apparatus as claimed in claim 1, wherein the at least one predetermined abnormality modification parameter comprises a rotation parameter, wherein the rotation parameter defines a rotation modification for the at least one audio object.

5. The apparatus as claimed in claim 1, wherein the at least one predetermined abnormality modification parameter comprises an amplification parameter, wherein the amplification parameter defines a volume of the at least one audio object relative to an original volume of the at least one audio object.

6. The apparatus as claimed in claim 1, wherein the at least one non-transitory memory including computer program code are further configured to, with the at least one processor, cause the apparatus to provide an interface to set up the at least one predetermined abnormality modification parameter for at least one of:

the at least one audio object,

at least one channel, or

at least one track for a detected user interaction.

7. The apparatus as claimed in claim 1, wherein the at least one instruction for rendering the at least one audio object in the free-viewpoint rendering in response to detection of the locational conflict abnormality further comprises at least one instruction for detection of the locational conflict abnormality based on at least one of:

a current position of the rendering,

a previous position of the rendering, or

a next position of the rendering,

wherein the current position, the previous position, and the next position of the rendering each include a viewpoint of a user.

8. The apparatus as claimed in claim 1, wherein the audio object metadata further includes at least one instruction that defines when the rendering of the at least one audio object is to be modified.

9. The apparatus as claimed in claim 1, wherein the audio object metadata further includes at least one instruction that defines an extent to which the rendering of the at least one audio object is to be modified.

10. The apparatus as claimed in claim 1, wherein detection of the locational conflict abnormality is based on a distance parameter.

11. The apparatus as claimed in claim 1, wherein the at least one non-transitory memory including computer program code are further configured to, with the at least one processor, cause the apparatus to transmit the at least one audio object.

12. The apparatus as claimed in claim 1, wherein the audio object metadata includes at least one of:

a channel,

a track,

a set of directional sub-band components,

a modification parameter, or

a rendering parameter.

13. The apparatus as claimed in claim 1, wherein the at least one non-transitory memory including computer program code are further configured to, with the at least one processor, cause the apparatus to transmit the audio object metadata for audio object spatial modification.

14. The apparatus as claimed in claim 13, wherein the at least one non-transitory memory including computer program code are further configured to, with the at least one processor, cause the apparatus to transmit the audio object metadata to an audio object spatial modification engine.

15. A method comprising:

defining audio object metadata associated with at least one audio object, wherein the audio object metadata includes at least one instruction for rendering the at least one audio object in a free-viewpoint rendering in response to detection of a locational conflict abnormality based upon at least one predetermined abnormality modification parameter; and

transmitting the audio object metadata, wherein the audio object metadata is configured to modify rendering of the at least one audio object.

16. The method as claimed in claim 15, wherein the at least one predetermined abnormality modification parameter comprises a location parameter, wherein the location parameter defines a location modification for the at least one audio object.

17. The method as claimed in claim 15, wherein the at least one predetermined abnormality modification parameter comprises a scaling parameter, wherein the scaling parameter defines a size modification for the at least one audio object.

18. The method as claimed in claim 15, wherein the at least one predetermined abnormality modification parameter comprises a rotation parameter, wherein the rotation parameter defines a rotation modification for the at least one audio object.

19. The method as claimed in claim 15, wherein the at least one predetermined abnormality modification parameter comprises an amplification parameter, wherein the amplification parameter defines a volume of the at least one audio object relative to an original volume of the at least one audio object.

20. The method as claimed in claim 15, further comprising:

providing an interface to set up the at least one predetermined abnormality modification parameter for at least one of;

the at least one audio object,

at least one channel, or

at least one track for a detected user interaction.

21. The method as claimed in claim 15, wherein the at least one instruction for rendering the at least one audio object in the free-viewpoint rendering in response to detection of the locational conflict abnormality further comprises at least one instruction for detection of the locational conflict abnormality based on at least one of:

a current position of the rendering,

a previous position of the rendering, or

a next position of the rendering,

wherein the current position, the previous position, and the next position of the rendering each include a viewpoint of a user.

22. A non-transitory program storage device readable with a machine, tangibly embodying a program of instructions executable with the machine for performing operations, the operations comprising:

defining audio object metadata associated with at least one audio object, wherein the audio object metadata includes at least one instruction for rendering the at least one audio object in a free-viewpoint rendering in response to detection of a locational conflict abnormality based upon at least one predetermined abnormality modification parameter; and

transmitting the audio object metadata, wherein the audio object metadata is configured to modify rendering of the at least one audio object.