Methods and systems for generating spatialized audio during a virtual experience

• US 10,206,055 B1
• Filed: 12/28/2017
• Issued: 02/12/2019
• Est. Priority Date: 12/28/2017
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Mindlin, Samuel C., Jathal, Kunal, Anis, Shan
• Primary Examiner(s): Lao, Lun-See
• Application Number:
  US15/856,359
• Time in Patent Office: 411 Days
• Field of Search: 381303, 381300, 381310, 381 17, 381 54- 58, 381 74, 381 77, 381 61, 381103, 381105, 381110, 700 94
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 5/005 of the pseudo five- or more-channel type, e.g. virtual surround
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A spatialized audio presentation system identifies an orientation of a virtual avatar, associated with a user engaged in a virtual experience, with respect to a virtual sound source within a virtual space. Within the virtual space, the virtual sound source generates a sound to be presented to the user while the user is engaged in the virtual experience. Based on the identified orientation of the virtual avatar, the system selects a head-related impulse response from a library of head-related impulse responses corresponding to different potential orientations of the virtual avatar with respect to the virtual sound source. The system then generates respective versions of the sound for presentation to the user at the left and right ears of the user by applying, respectively, a left-side component and a right-side component of the selected head-related impulse response to the sound. Corresponding methods are also disclosed.

Claims

1. A method comprising:

identifying, by a spatialized audio presentation system for a virtual avatar of a user engaged in a virtual experience within a virtual space, an orientation of the virtual avatar with respect to a virtual sound source that is located within the virtual space and that generates a sound to be presented to the user while the user is engaged in the virtual experience;

selecting, by the spatialized audio presentation system based on the identified orientation of the virtual avatar with respect to the virtual sound source, a head-related impulse response from a library of head-related impulse responses corresponding to different potential orientations of the virtual avatar with respect to the virtual sound source, the selected head-related impulse response including a left-side component and a right-side component;

generating, by the spatialized audio presentation system for presentation to the user at a left ear of the user while the user is engaged in the virtual experience, a left-side version of the sound by applying the left-side component of the selected head-related impulse response to the sound; and

generating, by the spatialized audio presentation system for presentation to the user at a right ear of the user while the user is engaged in the virtual experience, a right-side version of the sound by applying the right-side component of the selected head-related impulse response to the sound.

2. The method of claim 1, wherein:

the sound to be presented to the user is a voice communication spoken by an additional user engaged in the virtual experience along with the user; and

the virtual sound source that is located within the virtual space and that generates the sound to be presented to the user is an additional virtual avatar of the additional user.

3. The method of claim 1, wherein:

the left-side component of the selected head-related impulse response is recorded at a left ear of a user model in response to impulse stimulation generated at a spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source;

the right-side component of the selected head-related impulse response is recorded at a right ear of the user model in response to the impulse stimulation generated at the spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source; and

the left-side and right-side components of the selected head-related impulse response are collectively configured to model how sounds originating at the spatial location corresponding to the identified orientation are received by the user with respect to a biometric characteristic of the user including at least one of

a distance between the left ear and the right ear,

a distance between the left ear and a left shoulder of the user,

a distance between the right ear and a right shoulder of the user,

a distance from a pinna of the left ear to a canal of the left ear,

a distance from a pinna of the right ear to a canal of the right ear,

a distance from the left ear to a top of a head of the user, and

a distance from the right ear to the top of the head of the user.

4. The method of claim 3, wherein:

the user model is a physical head and torso simulation dummy disposed near the spatial location corresponding to the identified orientation so as to detect the impulse stimulation generated at the spatial location; and

the left-side and right-side components of the selected head-related impulse response are direct recordings of the impulse stimulation generated at the spatial location and received by the physical head and torso simulation dummy in accordance with a biometric characteristic of the physical head and torso simulation dummy that corresponds to the biometric characteristic of the user.

5. The method of claim 3, further comprising determining, by the spatialized audio presentation system, the biometric characteristic of the user;

wherein;

the user model is a physical head and torso simulation dummy disposed near the spatial location corresponding to the identified orientation so as to detect the impulse stimulation generated at the spatial location; and

the left-side and right-side components of the selected head-related impulse response are modified recordings of the impulse stimulation generated at the spatial location and received by the physical head and torso simulation dummy in accordance with a biometric characteristic of the physical head and torso simulation dummy that corresponds to the biometric characteristic of the user, the modified recordings modified to model the determined biometric characteristic of the user in place of the biometric characteristic of the physical head and torso simulation dummy.

6. The method of claim 3, further comprising determining, by the spatialized audio presentation system, the biometric characteristic of the user;

wherein;

the user model is a virtual user model based on the determined biometric characteristic of the user and simulated as being disposed near a virtual spatial location so as to detect virtual impulse stimulation simulated as being generated at the virtual spatial location, the virtual spatial location implementing the spatial location corresponding to the identified orientation and the virtual impulse stimulation implementing the impulse stimulation generated at the spatial location; and

the left-side and right-side components of the selected head-related impulse response are synthesized recordings of the virtual impulse stimulation simulated as being generated at the virtual spatial location and received by the virtual user model in accordance with the determined biometric characteristic of the user.

7. The method of claim 1, further comprising:

determining, by the spatialized audio presentation system, a delay parameter representative of a reflection time of an echo of the sound to be presented to the user; and

determining, by the spatialized audio presentation system, a diffusion parameter for the echo of the sound based on a virtual material within the virtual space from which the echo of the sound reflects;

wherein the generating of the left-side and right-side versions of the sound include applying, along with the left-side and right-side components of the selected head-related impulse response, a reverberation effect to the sound, the reverberation effect applied based on the delay parameter and the diffusion parameter.

8. The method of claim 1, further comprising:

determining, by the spatialized audio presentation system, an occlusion parameter representative of an effect of a virtual occlusion object on the sound to be presented to the user, the virtual occlusion object obstructing a direct sound propagation path between the virtual sound source and the virtual avatar within the virtual space;

wherein the generating of the left-side and right-side versions of the sound include applying, along with the left-side and right-side components of the selected head-related impulse response, an occlusion effect to the sound, the occlusion effect applied based on the occlusion parameter.

9. The method of claim 1, further comprising:

determining, by the spatialized audio presentation system, a source projection parameter representative of an effect of a direction in which the virtual sound source projects the generated sound to be presented to the user;

wherein the generating of the left-side and right-side versions of the sound include applying, along with the left-side and right-side components of the selected head-related impulse response, a source projection effect to the sound, the source projection effect applied based on the source projection parameter.

10. The method of claim 1, further comprising:

identifying, by the spatialized audio presentation system for the virtual avatar of the user, a distance from the virtual avatar to the virtual sound source; and

determining, by the spatialized audio presentation system, an attenuation parameter representative of an amplitude level fall-off of the sound propagating over the identified distance from the virtual avatar to the virtual sound source;

wherein the generating of the left-side and right-side versions of the sound include applying, along with the left-side and right-side components of the selected head-related impulse response, the attenuation parameter representative of the amplitude level fall-off of the sound.

11. The method of claim 1, wherein:

the left-side component of the selected head-related impulse response is recorded at a left ear of a user model in response to impulse stimulation generated at the spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source;

the right-side component of the selected head-related impulse response is recorded at a right ear of the user model in response to the impulse stimulation generated at the spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source; and

the impulse stimulation is implemented by at least one of a white noise signal and a chirp signal that each include a range of frequencies audible to a human ear.

12. The method of claim 1, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

13. A method comprising:

identifying, by a spatialized audio presentation system for a virtual avatar of a user engaged in a virtual experience within a virtual space, a first orientation, at a first point in time, of the virtual avatar with respect to a virtual sound source that is located within the virtual space and that generates a sound to be presented to the user while the user is engaged in the virtual experience;

selecting, by the spatialized audio presentation system based on the first orientation of the virtual avatar with respect to the virtual sound source, a first head-related impulse response from a library of head-related impulse responses corresponding to different potential orientations of the virtual avatar with respect to the virtual sound source, the first head-related impulse response including a left-side component and a right-side component;

generating, by the spatialized audio presentation system for presentation to the user at a left ear of the user while the user is engaged in the virtual experience, a left-side version of the sound by applying the left-side component of the first head-related impulse response to the sound;

generating, by the spatialized audio presentation system for presentation to the user at a right ear of the user while the user is engaged in the virtual experience, a right-side version of the sound by applying the right-side component of the first head-related impulse response to the sound;

identifying, by the spatialized audio presentation system for the virtual avatar, a second orientation, at a second point in time subsequent to the first point in time, of the virtual avatar with respect to the virtual sound source;

selecting, by the spatialized audio presentation system based on the second orientation of the virtual avatar with respect to the virtual sound source, a second head-related impulse response from the library of head-related impulse responses, the second head-related impulse response including a left-side component and a right-side component;

updating, by the spatialized audio presentation system, the left-side version of the sound by cross-fading the application of the left-side component of the first head-related impulse response to an application of the left-side component of the second head-related impulse response to the sound; and

updating, by the spatialized audio presentation system, the right-side version of the sound by cross-fading the application of the right-side component of the first head-related impulse response to an application of the right-side component of the second head-related impulse response to the sound.

14. The method of claim 13, embodied as computer-executable instructions on at least one non-transitory computer-readable medium.

15. A system comprising:

at least one physical computing device that

identifies, for a virtual avatar of a user engaged in a virtual experience within a virtual space, an orientation of the virtual avatar with respect to a virtual sound source that is located within the virtual space and that generates a sound to be presented to the user while the user is engaged in the virtual experience;

selects, based on the identified orientation of the virtual avatar with respect to the virtual sound source, a head-related impulse response from a library of head-related impulse responses corresponding to different potential orientations of the virtual avatar with respect to the virtual sound source, the selected head-related impulse response including a left-side component and a right-side component;

generates, for presentation to the user at a left ear of the user while the user is engaged in the virtual experience, a left-side version of the sound by applying the left-side component of the selected head-related impulse response to the sound; and

generates, for presentation to the user at a right ear of the user while the user is engaged in the virtual experience, a right-side version of the sound by applying the right-side component of the selected head-related impulse response to the sound.

16. The system of claim 15, wherein:

the sound to be presented to the user is a voice communication spoken by an additional user engaged in the virtual experience along with the user; and

the virtual sound source that is located within the virtual space and that generates the sound to be presented to the user is an additional virtual avatar of the additional user.

17. The system of claim 15, wherein:

the left-side component of the selected head-related impulse response is recorded at a left ear of a user model in response to impulse stimulation generated at a spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source;

the right-side component of the selected head-related impulse response is recorded at a right ear of the user model in response to the impulse stimulation generated at the spatial location corresponding to the identified orientation of the virtual avatar with respect to the virtual sound source; and

the left-side and right-side components of the selected head-related impulse responses are collectively configured to model how sounds originating at the spatial location corresponding to the identified orientation are received by the user with respect to a biometric characteristic of the user including at least one of

a distance between the left ear and the right ear,

a distance between the left ear and a left shoulder of the user,

a distance between the right ear and a right shoulder of the user,

a distance from a pinna of the left ear to a canal of the left ear,

a distance from a pinna of the right ear to a canal of the right ear,

a distance from the left ear to a top of a head of the user, and

a distance from the right ear to the top of the head of the user.

18. The system of claim 15, further comprising:

determines a delay parameter representative of a reflection time of an echo of the sound to be presented to the user; and

determines a diffusion parameter for the echo of the sound based on a virtual material within the virtual space from which the echo of the sound reflects;

wherein the at least one physical computing device generates the left-side and right-side versions of the sound by further applying, along with the left-side and right-side components of the selected head-related impulse response, a reverberation effect to the sound, the reverberation effect applied based on the delay parameter and the diffusion parameter.

19. The system of claim 15, wherein the at least one physical computing device further:

determines an occlusion parameter representative of an effect of a virtual occlusion object on the sound to be presented to the user, the virtual occlusion object obstructing a direct sound propagation path between the virtual sound source and the virtual avatar within the virtual space;

wherein the at least one physical computing device generates the left-side and right-side versions of the sound by further applying, along with the left-side and right-side components of the selected head-related impulse response, an occlusion effect to the sound, the occlusion effect applied based on the occlusion parameter.

20. The system of claim 15, wherein the at least one physical computing device further:

determines a source projection parameter representative of an effect of a direction in which the virtual sound source projects the generated sound to be presented to the user;

wherein the at least one physical computing device generates the left-side and right-side versions of the sound by further applying, along with the left-side and right-side components of the selected head-related impulse response, a source projection effect to the sound, the source projection effect applied based on the source projection parameter.