Mixed reality spatial audio

• US 10,616,705 B2
• Filed: 10/17/2018
• Issued: 04/07/2020
• Est. Priority Date: 10/17/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Schmidt, Brian Lloyd, Tajik, Jehangir, Jot, Jean-Marc
• Primary Examiner(s): Holder, Regina N
• Application Number:
  US16/163,529
• Publication Number:
  US 20190116448A1
• Time in Patent Office: 538 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/301 Automatic calibration of stereophonic sound system, e.g. with test microphone
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

A method of presenting an audio signal to a user of a mixed reality environment is disclosed. According to examples of the method, an audio event associated with the mixed reality environment is detected. The audio event is associated with a first audio signal. A location of the user with respect to the mixed reality environment is determined. An acoustic region associated with the location of the user is identified. A first acoustic parameter associated with the first acoustic region is determined. A transfer function is determined using the first acoustic parameter. The transfer function is applied to the first audio signal to produce a second audio signal, which is then presented to the user.

Claims

1. A method of presenting an audio signal to a user of a mixed reality environment, the method comprising:

detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;

determining a location of the user with respect to the mixed reality environment;

detecting a spatial property of an environment of the user;

identifying, based on the detected spatial property, a predefined first acoustic region;

determining a first acoustic parameter associated with the predefined first acoustic region;

determining, using the first acoustic parameter, a transfer function;

applying the transfer function to the first audio signal to produce a second audio signal; and

presenting, to the user, the second audio signal.

2. The method of claim 1, wherein the first audio signal comprises a waveform audio file.

3. The method of claim 1, wherein the first audio signal comprises a live audio stream.

4. The method of claim 1, wherein:

the user is associated with a wearable system comprising one or more sensors and a display configured to present a view of the mixed reality environment, and

identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the predefined first acoustic region is based on the first sensor input.

5. The method of claim 4, wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

6. The method of claim 5, wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

7. The method of claim 5, wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

8. The method of claim 4, wherein the wearable system further comprises a microphone, the microphone is located in the predefined first acoustic region, and the first acoustic parameter is determined based on a signal detected by the microphone.

9. The method of claim 1, wherein the first acoustic parameter corresponds to a reverberation parameter.

10. The method of claim 1, wherein the first acoustic parameter corresponds to a filtering parameter.

11. The method of claim 1, further comprising:

detecting a second spatial property of the environment of the user;

identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and

determining a second acoustic parameter associated with the predefined second acoustic region,

wherein the transfer function is determined using the second acoustic parameter.

12. A system comprising:

a wearable headgear unit including;

a display configured to present a view of a mixed reality environment;

a speaker;

one or more sensors; and

circuitry configured to perform a method comprising;

detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;

determining, based on the one or more sensors, a location of the wearable headgear unit with respect to the mixed reality environment;

detecting a spatial property of an environment of the user;

identifying, based on the detected spatial property, a predefined first acoustic region;

determining a first acoustic parameter associated with the predefined first acoustic region;

determining, using the first acoustic parameter, a transfer function;

applying the transfer function to the first audio signal to produce a second audio signal; and

presenting, to the user via the speaker, the second audio signal.

13. The system of claim 12, wherein the first audio signal comprises a waveform audio file.

14. The system of claim 12, wherein the first audio signal comprises a live audio stream.

15. The system of claim 12, wherein:

identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the first acoustic region is based on the first sensor input.

16. The system of claim 15, wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

17. The system of claim 16, wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

18. The system of claim 16, wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

19. The system of claim 15, wherein the first acoustic parameter is determined based on a signal detected by the microphone.

20. The system of claim 12, wherein the first acoustic parameter corresponds to a reverberation parameter.

21. The system of claim 12, wherein the first acoustic parameter corresponds to a filtering parameter.

22. The system of claim 12, further comprising:

detecting a second spatial property of the environment of the user;

identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and

determining a second acoustic parameter associated with the predefined second acoustic region,

wherein the transfer function is determined using the second acoustic parameter.

23. An augmented reality system comprising:

a localization subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, an identity of a predefined first space in which the augmented reality system is located;

a communication subsystem configured to communicate the identity of the first space in which the augmented reality system is located and further configured to receive an audio parameter associated with the first space; and

an audio output subsystem configured to process an audio segment based on the audio parameter and further configured to output the audio segment.

24. An augmented reality system comprising:

a sensor subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, information associated with an acoustic property of a predefined first space corresponding to a location of the augmented reality system;

an audio processing subsystem configured to process an audio segment based on the information, the audio processing subsystem communicatively coupled to the sensor subsystem; and

a speaker for presenting the audio segment, the speaker coupled to the audio processing subsystem.

25. The augmented reality system of claim 24 wherein the sensor subsystem is further configured to determine geometry information for the predefined first space.

26. The augmented reality system of claim 24 wherein the sensor subsystem comprises a camera.

27. The augmented reality system of claim 24 wherein the sensor subsystem comprises an object recognizer configured to recognize objects having distinct acoustic absorption properties.

28. The augmented reality system of claim 24 wherein the sensor subsystem comprises a microphone.