Spatial audio for three-dimensional data sets

• US 10,264,380 B2
• Filed: 11/14/2017
• Issued: 04/16/2019
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Salume, Diego E., Scott, Jason B., Sinex, Charles Tatsuya, Wifall, Richard
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US15/812,785
• Publication Number:
  US 20180332420A1
• Time in Patent Office: 518 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G06V 20/182 Network patterns, e.g. roads or rivers
• CPC Class Codes: G06V 20/188 Vegetation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A head mounted display (HMD) device is provided that includes a display, a position and orientation sensor, and speakers. The HMD device includes a processor configured to receive map data of a three-dimensional environment, receive a three-dimensional data set, detect a predefined feature in the three-dimensional data set, place a spatial audio emitter anchor in the three-dimensional environment in a location where the feature was detected, the spatial audio emitter having an emitter boundary, determine a current location of a user in the virtual environment based on a signal from the sensor, display the map data of the three-dimensional environment with a graphical representation of the three-dimensional data set based on the determined current location of the user, and responsive to determining that the location of the user is outside of the boundary, play the spatial audio emitter in a spatial audio mode via the near-ear speakers.

Claims

1. A head mounted display device, comprising:

a display;

a sensor configured to detect a position and orientation of a head of a user;

a pair of near-ear speakers; and

a processor configured to;

receive map data of a three-dimensional environment;

receive a three-dimensional data set;

detect a predefined feature in the three-dimensional data set at a location within the three-dimensional environment;

place a spatial audio emitter anchor in the three-dimensional environment in a location where the feature was detected, the spatial audio emitter having an emitter boundary surrounding the location;

determine a current location of the user in the virtual environment based on a signal from the sensor;

display the map data of the three-dimensional environment with a graphical representation of the three-dimensional data set based on the determined current location of the user; and

responsive to determining that the location of the user is outside of the boundary, play the spatial audio emitter in a spatial audio mode via the near-ear speakers.

2. The head mounted display device of claim 1, wherein playing the spatial audio emitter in the spatial audio mode includes playing a mono audio sample back via the near-ear speakers after processing the audio sample using a head related transfer function based on the position and orientation of the head of the user sensed by the sensor.

3. The head mounted display device of claim 1, wherein the three-dimensional data set represents a phenomenon selected from the group consisting of an atmospheric phenomenon, an oceanographic phenomenon, a crowd phenomenon, and a traffic phenomenon.

4. The head mounted display device of claim 1, wherein the processor is configured to:

adjust a characteristic of audio played by the audio emitter.

5. The head mounted display device of claim 1, wherein the predefined feature is a first predefined feature, and wherein the processor is further configured to:

detect an observation of a second predefined feature in the map data at a location proximate to the location of the first predefined feature in the first data set; and

adjust a characteristic of audio played by the audio emitter based upon the detection of the first predefined feature and the second predefined feature.

6. The head mounted display device of claim 5, wherein the second predefined feature is detected from terrain data, building data, vegetation data, or road data in the map data.

7. The head mounted display device of claim 5, wherein the processor is configured to adjust the characteristic of the audio played by the audio emitter by selecting an audio sample from among a plurality of audio samples based upon the detection of the first predefined feature and the second predefined feature.

8. The head mounted display device of claim 7, wherein the processor is configured to select two or more audio samples from among the plurality of audio samples based upon the detection of the first predefined feature and the second predefined feature, and adjust the characteristics of the audio played by the audio emitter based on a mix between the selected two or more audio samples.

9. The head mounted display device of claim 1, wherein the boundary is a first boundary defining a first region encompassing the audio emitter, and the audio emitter further includes a second boundary encompassing the first boundary and defining a second region between the first boundary and the second boundary, and wherein the processor is further configured to:

responsive to determining that the current location of the user is outside of the second boundary, ceasing to play the spatial audio emitter.

10. The head mounted display of claim 1, wherein the processor is further configured to:

responsive to determining that the current location of the user is inside the boundary, ceasing to play the spatial audio emitter in spatial audio mode, and commencing playing the spatial audio emitter in a two-dimensional immersive stereo mode.

11. The head mounted display of claim 10, wherein the ceasing and the commencing are accomplished according to a crossfade between the spatial audio mode and stereo mode that is executed in a crossfade region proximate the boundary.

12. A method, comprising:

at a processor of a computing device;

receiving map data of a three-dimensional environment;

receiving a three-dimensional data set;

detecting a predefined feature in the three-dimensional data set at a location within the three-dimensional environment;

placing a spatial audio emitter anchor in the three-dimensional environment in a location where the feature was detected, the spatial audio emitter having an emitter boundary surrounding the location;

determining a current location of the user in the virtual environment based on a signal from a sensor configured to detect a position and orientation of a head of a user;

displaying the map data of the three-dimensional environment with a graphical representation of the three-dimensional data set based on the determined current location of the user via a display; and

responsive to determining that the location of the user is outside of the boundary, playing the spatial audio emitter in a spatial audio mode via a pair of near-ear speakers.

13. The method of claim 12, wherein playing the spatial audio emitter in the spatial audio mode includes playing a mono audio sample back via the near-ear speakers after processing the audio sample using a head related transfer function based on the position and orientation of the head of the user sensed by the sensor.

14. The method of claim 12, wherein the three-dimensional data set represents a phenomenon selected from the group consisting of an atmospheric phenomenon, an oceanographic phenomenon, a crowd phenomenon, and a traffic phenomenon.

15. The method of claim 12, further comprising adjusting a characteristic of audio played by the audio emitter.

16. The method of claim 12, wherein the predefined feature is a first predefined feature, and wherein the method further comprises:

detecting an observation of a second predefined feature in the map data at a location proximate to the location of the first predefined feature in the first data set; and

adjusting a characteristic of audio played by the audio emitter based upon the detection of the first predefined feature and the second predefined feature.

17. The method of claim 16, wherein the second predefined feature is detected from terrain data, building data, vegetation data, or road data in the map data.

18. The method of claim 16, wherein the method further comprises adjusting the characteristic of the audio played by the audio emitter by selecting an audio sample from among a plurality of audio samples based upon the detection of the first predefined feature and the second predefined feature.

19. The method of claim 12, wherein the boundary is a first boundary defining a first region encompassing the audio emitter, and the audio emitter further includes a second boundary encompassing the first boundary and defining a second region between the first boundary and the second boundary, and wherein the method further comprises:

responsive to determining that the current location of the user is outside of the second boundary, ceasing to play the spatial audio emitter; and

responsive to determining that the current location of the user is inside the boundary, ceasing to play the spatial audio emitter in spatial audio mode, and commencing playing the spatial audio emitter in a two-dimensional immersive stereo mode.

20. A computing device comprising:

a sensor configured to detect a position and orientation of a head of a user;

a pair of near-ear speakers; and

a processor configured to;

receive map data of a three-dimensional environment;

receive a three-dimensional data set;

detect a predefined feature in the three-dimensional data set at a location within the three-dimensional environment;

place a spatial audio emitter anchor in the three-dimensional environment in a location where the feature was detected, the spatial audio emitter having an emitter boundary surrounding the location;

determine a current location of the user in three-dimensional environment based on a signal from the sensor; and

responsive to determining that the location of the user is outside of the boundary, play the spatial audio emitter in a spatial audio mode via the near-ear speakers.