Spatial delivery of multi-source audio content

• US 10,237,675 B1
• Filed: 05/22/2018
• Issued: 03/19/2019
• Est. Priority Date: 05/22/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Valavanis, George Michael
• Primary Examiner(s): Kurr, Jason R
• Application Number:
  US15/986,537
• Time in Patent Office: 301 Days
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 3/04 for correcting frequency response
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A system for enabling spatial delivery of multi-source audio data to a user based on a multi-layer audio stack is provided. The multi-layer audio stack includes a central layer located within a predetermined vertical distance from a reference line associated with the user, such as the horizon line of the user. The multi-layer audio stack can also include an upper layer located above the central layer and/or a lower layer located below the central layer. Audio data from multiple sources are collected and prioritized based on context data gathered for the user. Audio data on which the user would like to focus is assigned the highest priority and delivered on the central layer. Audio data that the user does not currently focus on, but would like to visit next, can be assigned a lower priority and be delivered in the upper layer or the lower layer.

Claims

1. A computing device, comprising:

a processor; and

a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to at least;

receive audio data associated with a plurality of audio sources;

assign a priority to audio data associated with each of the plurality of audio sources;

deliver audio data to a user based on a multi-layer audio stack of the user by

rendering audio data having a first priority to the user at a central layer of the multi-layer audio stack, the central layer comprising a center spatial region at a first elevation within a predetermined vertical distance from a reference line associated with the user, wherein rendering the audio data having the first priority comprises generating a first audible sound from the audio data having the first priority, the rendering providing a simulation that the first audible sound is emanating from the center spatial region; and

rendering audio data having a second priority at an upper layer or a lower layer of the multi-layer audio stack, the upper layer comprising an upper spatial region at a second elevation higher than the center spatial region of the central layer and the lower layer comprising a lower spatial region at a third elevation lower than the center spatial region of the central layer, wherein rendering the audio data having the second priority comprises generating a second audible sound from the audio data having the second priority, the second audible sound configured to appear to emanate from the upper spatial region of the upper layer or the lower spatial region of the lower layer.

2. The computing device of claim 1, wherein a size of the center spatial region of the central layer is larger than a size of the upper spatial region of the upper layer and a size of the lower spatial region of the lower layer.

3. The computing device of claim 1, wherein the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to:

receive an instruction to navigate to a selected layer in the multi-layer audio stack;

in response to receiving the instruction, shift the multi-layer audio stack to place the selected layer at the central layer of the multi-layer audio stack, update the priority associated with the audio data of the plurality of audio sources to assign the first priority to audio data being delivered at the selected layer and a second priority to audio data being delivered at other layers of the multi-layer audio stack, and deliver the audio data associated with the plurality of audio sources based on the updated priority and the shifted multi-layer audio stack.

4. The computing device of claim 3, wherein the memory having further computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to

in response to an event occurring at the lower layer or the upper layer, generate a notification of the event at a corresponding layer, wherein the selected layer is the layer where the event occurred, and wherein the instruction to navigate to the selected layer is received in response to the notification of the event.

5. The computing device of claim 1, wherein delivering the audio data in the multi-layer audio stack comprises generating the first audible sound and the second audible sound using spatial audio technology to provide a simulation that the first audible sound and the second audible sound are emanating from respective audio objects located in a corresponding layer of the multi-layer audio stack.

6. The computing device of claim 5, wherein delivering the audio data further comprises moving the respective audio objects from a first location to a second location within the respective layers.

7. The computing device of claim 1, wherein the plurality of audio sources comprise at least one software application generating audio signals.

8. A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a computing device, cause the one or more processors of the computing device to at least:

receive audio data associated with a plurality of audio sources;

assign a priority to each of the plurality of audio sources and a corresponding audio data;

deliver the audio data to a user based on the priority and a multi-layer audio stack comprising a central layer and at least one of a lower layer or a upper layer, the central layer comprising a center spatial region at a first elevation within a predetermined vertical distance from a reference line associated with the user, the upper layer comprising an upper spatial region at a second elevation higher than the center spatial region of the central layer and the lower layer comprising a lower spatial region at a third elevation lower than the center spatial region of the central layer, wherein delivering the audio data comprises;

rendering audio data having a first priority at the central layer of the multi-layer audio stack by generating a first audible sound from the audio data having the first priority, the rendering providing a simulation that the first audible sound is emanating from the center spatial region; and

rendering audio data having a second priority at the upper layer or the lower layer of the multi-layer audio stack by generating a second audible sound from the audio data having the second priority, the rendering providing a simulation that the second audible sound is emanating from the upper spatial region or the lower spatial region.

9. The computer-readable storage medium of claim 8, wherein a size of the center spatial region of the central layer is larger than a size of the upper spatial region of the upper layer and a size of the lower spatial region of the lower layer.

10. The computer-readable storage medium of claim 8, wherein delivering the audio data in the multi-layer audio stack comprises generating the first audible sound and the second audible sound using spatial audio technology to provide a simulation that the first audible sound and the second audible sound are emanating from respective audio objects located in a corresponding layer of the multi-layer audio stack.

11. The computer-readable storage medium of claim 10, wherein a plurality of audio objects are associated with the audio data delivered in the central layer and are distributed with a predetermined minimum distance between any pair of the plurality of the audio objects.

12. The computer-readable storage medium of claim 8, having further computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to:

receive an instruction to navigate to a selected layer in the multi-layer audio stack;

in response to receiving the instruction, shift the multi-layer audio stack to place the selected layer at the central layer of the multi-layer audio stack, update the priority associated with the audio data of the plurality of audio sources to assign the first priority to audio data being delivered at the selected layer and assign a priority lower than the first priority to audio data being delivered at other layers of the multi-layer audio stack, and deliver the audio data associated with the plurality of audio sources based on the updated priority and the shifted multi-layer audio stack.

13. The computer-readable storage medium of claim 8, wherein a size of the center spatial region of the central layer is larger than a size of the upper spatial region of the upper layer and a size of the lower spatial region of the lower layer.

14. A method, comprising:

receiving audio data associated with a plurality of audio sources;

assigning a priority to each of the plurality of audio sources and the corresponding audio data;

delivering the audio data to a user based on the assigned priority and a multi-layer audio stack comprising a central layer and at least one of a lower layer or a upper layer, the central layer comprising a center spatial region at a first elevation within a predetermined vertical distance from a reference line associated with the user, the upper layer comprising an upper spatial region at a second elevation higher than the center spatial region of the central layer and the lower layer comprising a lower spatial region at a third elevation lower than the center spatial region of the central layer, wherein delivering the audio data comprises;

rendering audio data having a first priority to the user at the central layer of the multi-layer audio stack by generating a first audible sound from the audio data having the first priority, the rendering providing a simulation that the first audible sound is emanating from the center spatial region; and

rendering audio data having a second priority at the upper layer or the lower layer of the multi-layer audio stack by generating a second audible sound from the audio data having the second priority, the rendering providing a simulation that the second audible sound is emanating from the upper spatial region or the lower spatial region.

15. The method of claim 14, wherein delivering the audio data at the upper layer or the lower layer of the multi-layer audio stack further comprises pre-processing the audio data before rendering the audio data at the corresponding layer.

16. The method of claim 15, wherein preprocessing the audio data comprises applying a low pass filter on the audio data.

17. The method of claim 14, wherein the center spatial region of the central layer, the upper spatial regions of the upper layer and the lower spatial regions of the lower layer form a first ring shape area, a second ring shape area and a third ring shape area, respectively.

18. The method of claim 17, wherein a first radius of the first ring shape area of the central layer is larger than a second radius of the second ring shape area of the upper layer and a third radius of the third ring shape area of the lower layer.

19. The method of claim 18, wherein the ring shape area of the central layer spans vertically in space into a donut shape area.

20. The method of claim 14, further comprising:

receiving an instruction to navigate to a selected layer in the multi-layer audio stack;

in response to receiving the instruction, shifting the multi-layer audio stack to place the selected layer at the central layer of the multi-layer audio stack and adjusting the size of the layers of the multi-layer audio stack, updating the priority associated with the audio data of the plurality of audio sources to assign the first priority to audio data being delivered at the selected layer and a priority lower than the first priority to audio data being delivered at other layers of the multi-layer audio stack, and delivering the audio data associated with the plurality of audio sources based on the updated priority and the updated multi-layer audio stack.