VR audio superzoom

• US 10,165,386 B2
• Filed: 05/16/2017
• Issued: 12/25/2018
• Est. Priority Date: 05/16/2017
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Lehtiniemi, Arto Juhani, Eronen, Antti Johannes, Leppanen, Jussi Artturi, Mate, Sujeet Shyamsundar
• Primary Examiner(s): Sniezek, Andrew L
• Application Number:
  US15/596,533
• Publication Number:
  US 20180338213A1
• Time in Patent Office: 588 Days
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/03 Application of parametric coding in stereophonic audio systems
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A method including, identifying at least one object of interest (OOI), determining a plurality of microphones capturing sound from the at least one OOI, determining, for each of the plurality of microphones, a volume around the at least one OOI, determining a spatial audio volume based on associating each of the plurality of microphones to the volume around the at least one OOI, and generating a spatial audio scene based on the spatial audio volume for free-listening-point audio around the at least one OOI.

Claims

1. A method comprising:

identifying at least one object of interest;

determining a plurality of microphones capturing sound from the at least one object of interest, wherein at least one of the plurality of microphones is located at a separate position from at least one other of the plurality of microphones in an environment, and wherein determining the at least one of the plurality of microphones and the at least one other of the plurality of microphones comprises determining each said respective microphone is capturing sound from the at least one object of interest relative to a microphone in close proximity to the at least one object of interest;

determining, for each said respective microphone at each of the separate positions in the environment, at least one of an area, a volume, and a point around the at least one object of interest;

determining an audio scene based on associating each of said respective microphones to the at least one of the determined area, volume, and point around the at least one object of interest; and

generating the audio scene based on at least one of the determined audio scene for free-listening-point audio around the at least one object of interest.

2. The method of claim 1, wherein generating the audio scene further comprises:

generating a superzoom audio scene, wherein the superzoom audio scene enables a volumetric audio experience that allows a user to select to experience the at least one object of interest at different levels of detail, and as captured by different devices of the plurality of microphones and from at least one of a different location and a different direction than a first direction and location.

3. The method of claim 1, wherein generating the audio scene further comprises:

generating a sound of the at least one object of interest from a plurality of the separate positions.

4. The method of claim 1, wherein the audio scene further comprises a volumetric six-degrees-of-freedom audio scene.

5. The method of claim 1, wherein the plurality of microphones includes at least one of a microphone array, a stage microphone, and a Lavalier microphone.

6. The method of claim 1, generating the audio scene further comprises:

determining a distance to a user and a direction to the user associated with the at least one object of interest.

7. The method of claim 1, further comprising:

performing, for at least one of the plurality of microphones, beamforming from the at least one object of interest to a user.

8. The method of claim 1, wherein determining, for each of the plurality of microphones, the area around the at least one object of interest further comprises:

determining separate areas associated with each of the plurality of microphones; and

determining a border between each of the separate areas.

9. The method of claim 1, wherein the plurality of microphones includes at least one microphone with a sound signal associated particular section of the at least one object of interest and at least one other microphone with a sound signal associated with an entire area of the at least one object of interest.

10. The method of claim 9, wherein generating the audio scene further comprises:

increasing a proportion of the sound signal associated with the particular section of the at least one object of interest in relation to the sound signal associated with the entire area of the at least one object of interest in response to a user moving closer to the particular section of the at least one object of interest.

11. The method of claim 1, further comprising:

determining a position for each of the plurality of microphones based on a high accuracy indoor positioning tag.

12. The method of claim 1, wherein determining the plurality of microphones capturing sound from the at least one object of interest further comprises:

performing cross-correlation between a microphone in close proximity to the at least one object of interest and each of the others of the plurality of microphones.

13. The method of claim 1, wherein identifying the object of interest is based on receiving an indication from a user.

14. The method of claim 1, wherein generating the audio scene further comprises:

at least one of storing, transmitting and streaming the audio scene.

15. An apparatus comprising:

at least one processor; and

at least one non-transitory memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to;

identify at least one object of interest;

determine a plurality of microphones capturing sound from the at least one object of interest, wherein at least one of the plurality of microphones is located at a separate position from at least one other of the plurality of microphones in an environment, and wherein determining the at least one of the plurality of microphones and the at least one other of the plurality of microphones comprises determining each said respective microphone is capturing sound from the at least one object of interest relative to a microphone in close proximity to the at least one object of interest;

determine, for each said respective microphone at each of the separate positions in the environment, at least one of an area, a volume, and a point around the at least one object of interest;

determine an audio scene based on associating each of said respective microphones to the at least one of the determined area, volume, and point around the at least one object of interest; and

generate the audio scene based on at least one of the determined audio scene for free-listening-point audio around the at least one object of interest.

16. An apparatus as in claim 15, where, when generating the audio scene, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

generate a superzoom audio scene, wherein the superzoom audio scene enables a volumetric audio experience that allows a user to select to experience the at least one object of interest at different levels of detail, and as captured by different devices of the plurality of microphones and from at least one of a different location and a different direction than a first direction and location.

17. An apparatus as in claim 15, wherein the plurality of microphones includes at least one of a microphone array, a stage microphone, and a Lavalier microphone.

18. An apparatus as in claim 15, where, when generating the audio scene, the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:

determine a distance to a user and a direction to the user associated with the at least one object of interest.

19. An apparatus as in claim 15, where the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to:

perform, for at least one of the plurality of microphones, beamforming from the at least one object of interest to a user.

20. A non-transitory program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations, the operations comprising:

identifying at least one object of interest;

determining a plurality of microphones capturing sound from the at least one object of interest, wherein at least one of the plurality of microphones is located at a separate position from at least one other of the plurality of microphones in an environment, and wherein determining the at least one of the plurality of microphones and the at least one other of the plurality of microphones comprises determining each said respective microphone is capturing sound from the at least one object of interest relative to a microphone in close proximity to the at least one object of interest;

determining, for each said respective microphone at each of the separate positions in the environment, at least one of an area, a volume, and a point around the at least one object of interest;

determining an audio scene based on associating each of said respective microphones to the at least one of the determined area, volume, and point around the at least one object of interest; and

generating the audio scene based on at least one of the determined audio scene for free-listening-point audio around the at least one object of interest.