Audio content modification for playback audio

• US 10,542,368 B2
• Filed: 03/27/2018
• Issued: 01/21/2020
• Est. Priority Date: 03/27/2018
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Vilermo, Miikka Tapani, Lehtiniemi, Arto Juhani, Laaksonen, Lasse Juhani, Tammi, Mikko Tapio
• Primary Examiner(s): Fischer, Mark
• Application Number:
  US15/937,248
• Publication Number:
  US 20190306651A1
• Time in Patent Office: 665 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3278 Power saving in modem or I/O interface
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method including accessing, by a user device, a location corresponding to at least one microphone array in a content of a free-viewpoint audio environment. The method includes tracking user movement of a user based on a user position in the content of the free-viewpoint audio environment, determining a probability of error in a direction estimate from at least one part of an audio signal; and applying audio ambience to the at least one part of the audio signal based on the user displacement from the nearest microphone array of the at least one microphone array and the probability of error in the direction estimate from the at least one part of the audio signal.

Claims

1. A method comprising:

accessing, by a user device, a location corresponding to at least one microphone array in a content of a free-viewpoint audio environment;

tracking user movement of a user based on a user position in the content of the free-viewpoint audio environment;

determining user displacement from a nearest microphone array of the at least one microphone array;

determining a probability of error in a direction estimate from at least one part of an audio signal; and

applying audio ambience to the at least one part of the audio signal based on the user displacement from the nearest microphone array of the at least one microphone array and the probability of error in the direction estimate from the at least one part of the audio signal.

2. The method as in claim 1, wherein applying audio ambience to the at least one part of the audio signal further comprises:

increasing audio ambience with the user displacement and making audio from direction of the user displacement more ambient than audio from an opposite direction.

3. The method as in claim 1, wherein tracking user movement in the content of the free-viewpoint audio environment further comprises:

tracking the user movement based on at least one of;

global positioning system, high accuracy indoor positioning, at least one accelerometer, at least one camera based system, or at least one light based system.

4. The method as in claim 1, further comprising:

making playback audio from high-probability error directions more ambient than playback audio from low-probability error directions.

5. The method as in claim 1, wherein the content of the free-viewpoint audio environment includes a virtual environment and tracking the user movement comprises tracking virtual movement of the user.

6. The method as in claim 1, wherein the content of the free-viewpoint audio environment comprises a real world environment and tracking the user movement comprises tracking movement of the user in the real world environment.

7. The method as in claim 1, further comprising:

presenting audio as a directional signal, an ambient signal and direction parameter for each frequency band.

8. The method as in claim 1, further comprising:

presenting audio via directional audio coding.

9. The method as in claim 1, wherein the applying of the audio ambience comprises determining the audio ambience based on the user displacement, and the method further comprises:

determining an amount of ambiguity based on a near distance and a far distance, wherein the near distance is a closest distance at which 360 stitching starts to work.

10. An apparatus comprising:

at least one processor; and

at least one non-transitory memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to;

access a location corresponding to at least one microphone array in a content of a free-viewpoint audio environment;

track user movement of a user based on a user position in the content of the free-viewpoint audio environment;

determine user displacement from a nearest microphone array of the at least one microphone array;

determine a probability of error in a direction estimate from at least one part of an audio signal; and

apply audio ambience to the at least one part of the audio signal based on the user displacement from the nearest microphone array of the at least one microphone array and the probability of error in the direction estimate from the at least one part of the audio signal.

11. An apparatus as in claim 10, wherein, when determining audio ambience based on the user displacement from the nearest microphone array, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

increase audio ambience with the user displacement and make audio from direction of the user displacement more ambient than audio from opposite direction.

12. An apparatus as in claim 10, wherein, when tracking user movement in the content of the free-viewpoint audio environment, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

track the user movement based on at least one of;

global positioning system, high accuracy indoor positioning, at least one accelerometer, at least one camera based system, or at least one light based system.

13. An apparatus as in claim 10, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:

make playback audio from high-probability error directions more ambient than playback audio from low-probability error directions.

14. An apparatus as in claim 10, wherein the content of the free-viewpoint audio environment comprises a virtual environment and tracking the user movement comprises tracking virtual movement of the user.

15. An apparatus as in claim 10, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:

present audio as a directional signal, an ambient signal and direction parameter for each frequency band.

16. An apparatus as in claim 10, wherein, when determining audio ambience based on the user displacement, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

determine an amount of ambiguity based on a near distance and a far distance, wherein the near distance is a closest distance at which 360 stitching starts to work.

17. An apparatus as in claim 10, wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:

modify audio in tiles in which uncertainty is larger than a threshold.

18. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising:

accessing, by a user device, a location corresponding to at least one microphone array in a content of a free-viewpoint audio environment;

tracking user movement of a user based on a user position in the content of the free-viewpoint audio environment;

determining user displacement from a nearest microphone array of the at least one microphone array;

determining a probability of error in a direction estimate from at least one part of an audio signal; and

applying audio ambience to the at least one part of the audio signal based on the user displacement from the nearest microphone array of the at least one microphone array and the probability of error in the direction estimate from the at least one part of the audio signal.