Methods and systems for providing a composite audio stream for an extended reality world

• US 10,484,811 B1
• Filed: 09/10/2018
• Issued: 11/19/2019
• Est. Priority Date: 09/10/2018
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Inventors: Mindlin, Samuel Charles, Khalid, Mohammad Raheel, Jathal, Kunal, Anis, Shan
• Primary Examiner(s): Chin, Vivian C
• Assistant Examiner(s): Fahnert, Friedrich
• Application Number:
  US16/126,289
• Time in Patent Office: 435 Days
• Field of Search: 381 22, 381300
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

An exemplary extended reality audio processing system accesses first and second audio streams representative, respectively, of first and second virtual sounds within an extended reality world being experienced by a user by way of a media player device. The system also accesses acoustic propagation data representative of characteristics that affect the propagation of the first and second virtual sounds to an avatar of the user. Based on the first and second audio streams and the real-time acoustic propagation data, the system renders a composite binaural audio stream representative of a composite virtual sound that comprises a combination of simulated reproductions of the first and second virtual sounds that are customized to account for the characteristics that affect the propagation. The system further transmits the composite binaural audio stream to the media player device for presentation to the user as the user experiences the extended reality world.

Claims

1. A method comprising:

accessing, by an extended reality audio processing system, a first audio stream representative of a first virtual sound that virtually originates from a first virtual audio source within an extended reality world being experienced by a user by way of a media player device the extended reality world generated based on a live capture of a real-world scene and managed by a live capture management system separate from the media player device;

accessing, by the extended reality audio processing system, a second audio stream representative of a second virtual sound that virtually originates from a second virtual audio source within the extended reality world, the second virtual audio source distinct from the first virtual audio source;

accessing, by the extended reality audio processing system as the first and second virtual sounds propagate to an avatar of the user within the extended reality world, real-time acoustic propagation data representative of characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user, the accessed real-time acoustic propagation data including listener propagation data accessed from the media player device and world propagation data accessed from the live capture management system, the world propagation data associated with the real-world scene and tracked and managed by the live capture management system;

rendering, by the extended reality audio processing system as the first and second virtual sounds propagate to the avatar of the user and based on the first and second audio streams and the real-time acoustic propagation data, a composite binaural audio stream representative of a composite virtual sound that comprises a combination of simulated reproductions of the first and second virtual sounds, the simulated reproductions customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user; and

transmitting, by the extended reality audio processing system, the composite binaural audio stream to the media player device for presentation to the user as the user experiences the extended reality world.

2. The method of claim 1, wherein:

the extended reality audio processing system is a network-edge-deployed server separate from the media player device;

the listener propagation data represents a real-time pose of the avatar of the user as the user experiences the extended reality world, the real-time pose including a position and an orientation of the avatar at a first time; and

the transmitting of the composite binaural audio stream by the network-edge-deployed server is performed so as to provide the composite binaural audio stream to the media player device at a second time that is within a predetermined latency threshold after the first time.

3. The method of claim 1, further comprising rendering and transmitting, by the extended reality audio processing system, an additional composite binaural audio stream for presentation to the user as the user experiences a single-user extended reality world managed by the media player device, the rendering of the additional composite binaural audio stream based on additional real-time acoustic propagation data that includes world propagation data and listener propagation data both accessed from the media player device.

4. The method of claim 1, wherein:

the extended reality world being experienced by the user is a multi-user extended reality world managed by the live capture management system separate from the media player device;

the live capture management system manages world propagation data associated with a plurality of users including the user; and

the world propagation data accessed by the extended reality audio processing system from the live capture management system includes the world propagation data associated with the plurality of users including the user.

5. The method of claim 1, wherein:

the extended reality world is a multi-user extended reality world being jointly experienced by a plurality of users including the user and an additional user;

the first virtual audio source is an avatar of the additional user within the multi-user extended reality world;

the first virtual sound represented by the first audio stream is based on a real-time captured sound originating from the additional user as the additional user experiences the multi-user extended reality world;

the second virtual audio source is one of a diegetic media source included within the multi-user extended reality world, an intelligent assistant embodied within the multi-user extended reality world, and a non-player character embodied within the multi-user extended reality world; and

the second virtual sound represented by the second audio stream is based on a prerecorded or synthesized sound originating from a real audio source corresponding to the second virtual audio source.

6. The method of claim 1, wherein the rendering of the composite binaural audio stream includes:

decoding the first audio stream to generate a first raw audio stream;

decoding the second audio stream to generate a second raw audio stream;

generating, based on the first and second raw audio streams, a three-dimensional (3D) audio representation of the first and second virtual sounds that is customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user;

generating a composite binaural raw audio stream representative of the 3D audio representation of the first and second virtual sounds; and

encoding the composite binaural raw audio stream to generate the composite binaural audio stream that is transmitted to the media player device for presentation to the user as the user experiences the extended reality world.

7. The method of claim 1, wherein the accessing of the first audio stream is performed by accessing the first audio stream from a plurality of microphones that are positioned at respective locations with respect to the real-world scene so as to capture audio originating from an audio source within the real-world scene.

8. A system comprising:

at least one physical computing device that

accesses a first audio stream representative of a first virtual sound that virtually originates from a first virtual audio source within an extended reality world being experienced by a user by way of a media player device, the extended reality world generated based on a live capture of a real-world scene and managed by a live capture management system separate from the media player device;

accesses a second audio stream representative of a second virtual sound that virtually originates from a second virtual audio source within the extended reality world, the second virtual audio source distinct from the first virtual audio source;

accesses, as the first and second virtual sounds propagate to an avatar of the user within the extended reality world, real-time acoustic propagation data representative of characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user, the accessed real-time acoustic propagation data including listener propagation data accessed from the media player device and world propagation data accessed from the live capture management system, the world propagation data associated with the real-world scene and tracked and managed by the live capture management system;

renders, as the first and second virtual sounds propagate to the avatar of the user and based on the first and second audio streams and the real-time acoustic propagation data, a composite binaural audio stream representative of a composite virtual sound that comprises a combination of simulated reproductions of the first and second virtual sounds, the simulated reproductions customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user; and

transmits the composite binaural audio stream to the media player device for presentation to the user as the user experiences the extended reality world.

9. The system of claim 8, wherein:

the at least one physical computing device is implemented within a network-edge-deployed server separate from the media player device;

the listener propagation data represents a real-time pose of the avatar of the user as the user experiences the extended reality world, the real-time pose including a position and an orientation of the avatar at a first time; and

the network-edge-deployed server transmits the composite binaural audio stream so as to provide the composite binaural audio stream to the media player device at a second time that is within a predetermined latency threshold after the first time.

10. The system of claim 8, wherein the at least one physical computing device further renders and transmits an additional composite binaural audio stream for presentation to the user as the user experiences a single-user extended reality world managed by the media player device, the rendering of the additional composite binaural audio stream based on additional real-time acoustic propagation data that includes world propagation data and listener propagation data both accessed from the media player device.

11. The system of claim 8, wherein:

the extended reality world being experienced by the user is a multi-user extended reality world managed by the live capture management system separate from the media player device;

the live capture management system manages world propagation data associated with a plurality of users including the user; and

the world propagation data accessed by the at least one physical computing device from the live capture management system includes the world propagation data associated with the plurality of users including the user.

12. The system of claim 8, wherein:

the extended reality world is a multi-user extended reality world being jointly experienced by a plurality of users including the user and an additional user;

the first virtual audio source is an avatar of the additional user within the multi-user extended reality world;

the first virtual sound represented by the first audio stream is based on a real-time captured sound originating from the additional user as the additional user experiences the multi-user extended reality world;

the second virtual audio source is one of a diegetic media source included within the multi-user extended reality world, an intelligent assistant embodied within the multi-user extended reality world, and a non-player character embodied within the multi-user extended reality world; and

the second virtual sound represented by the second audio stream is based on a prerecorded or synthesized sound originating from a real audio source corresponding to the second virtual audio source.

13. The system of claim 8, wherein the at least one physical computing device renders the composite binaural audio stream by:

decoding the first audio stream to generate a first raw audio stream;

decoding the second audio stream to generate a second raw audio stream;

generating, based on the first and second raw audio streams, a three-dimensional (3D) audio representation of the first and second virtual sounds that is customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user;

generating a composite binaural raw audio stream representative of the 3D audio representation of the first and second virtual sounds; and

encoding the composite binaural raw audio stream to generate the composite binaural audio stream that is transmitted to the media player device for presentation to the user as the user experiences the extended reality world.

14. The system of claim 8, wherein the at least one physical computing device accesses the first audio stream from a plurality of microphones that are positioned at respective locations with respect to the real-world scene so as to capture audio originating from an audio source within the real-world scene.

15. A non-transitory computer-readable medium storing instructions that, when executed, direct at least one processor of a computing device to:

access a first audio stream representative of a first virtual sound that virtually originates from a first virtual audio source within an extended reality world being experienced by a user by way of a media player device, the extended reality world generated based on a live capture of a real-world scene and managed by a live capture management system separate from the media player device;

access a second audio stream representative of a second virtual sound that virtually originates from a second virtual audio source within the extended reality world, the second virtual audio source distinct from the first virtual audio source;

access, as the first and second virtual sounds propagate to an avatar of the user within the extended reality world, real-time acoustic propagation data representative of characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user, the accessed real-time acoustic propagation data including listener propagation data accessed from the media player device and world propagation data accessed from the live capture management system, the world propagation data associated with the real-world scene and tracked and managed by the live capture management system;

render, as the first and second virtual sounds propagate to the avatar of the user and based on the first and second audio streams and the real-time acoustic propagation data, a composite binaural audio stream representative of a composite virtual sound that comprises a combination of simulated reproductions of the first and second virtual sounds, the simulated reproductions customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user; and

transmit the composite binaural audio stream to the media player device for presentation to the user as the user experiences the extended reality world.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed, direct the at least one processor to further render and transmit an additional composite binaural audio stream for presentation to the user as the user experiences a single-user extended reality world managed by the media player device, the rendering of the additional composite binaural audio stream based on additional real-time acoustic propagation data that includes world propagation data and listener propagation data both accessed from the media player device.

17. The non-transitory computer-readable medium of claim 15, wherein:

the extended reality world being experienced by the user is a multi-user extended reality world managed by the live capture management system separate from the media player device;

the live capture management system manages world propagation data associated with a plurality of users including the user; and

the world propagation data accessed by the at least one processor from the live capture management system includes the world propagation data associated with the plurality of users including the user.

18. The non-transitory computer-readable medium of claim 15, wherein:

the extended reality world is a multi-user extended reality world being jointly experienced by a plurality of users including the user and an additional user;

the first virtual audio source is an avatar of the additional user within the multi-user extended reality world;

the first virtual sound represented by the first audio stream is based on a real-time captured sound originating from the additional user as the additional user experiences the multi-user extended reality world;

the second virtual audio source is one of a diegetic media source included within the multi-user extended reality world, an intelligent assistant embodied within the multi-user extended reality world, and a non-player character embodied within the multi-user extended reality world; and

the second virtual sound represented by the second audio stream is based on a prerecorded or synthesized sound originating from a real audio source corresponding to the second virtual audio source.

19. The non-transitory computer-readable medium of claim 15, wherein the instructions direct the at least one processor to render the composite binaural audio stream by:

decoding the first audio stream to generate a first raw audio stream;

decoding the second audio stream to generate a second raw audio stream;

generating, based on the first and second raw audio streams, a three-dimensional (3D) audio representation of the first and second virtual sounds that is customized to account for the characteristics that affect the propagation of the first and second virtual sounds to the avatar of the user;

generating a composite binaural raw audio stream representative of the 3D audio representation of the first and second virtual sounds; and

encoding the composite binaural raw audio stream to generate the composite binaural audio stream that is transmitted to the media player device for presentation to the user as the user experiences the extended reality world.

20. The non-transitory computer-readable medium of claim 15, wherein the instructions, when executed, direct the at least one processor to access the first audio stream from a plurality of microphones that are positioned at respective locations with respect to the real-world scene so as to capture audio originating from an audio source within the real-world scene.