SYSTEM AND METHOD FOR HIGH-PRECISION 3-DIMENSIONAL AUDIO FOR AUGMENTED REALITY

• US 20120093320A1
• Filed: 10/13/2010
• Published: 04/19/2012
• Est. Priority Date: 10/13/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Flaks, Jason, Bar-Zeev, Avi, Tardif, John
• Granted Patent: US 8,767,968 B2
• Time in Patent Office: Days
• US Class Current: 381/17
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/301 Automatic calibration of stereophonic sound system, e.g. with test microphone
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

Techniques are provided for providing 3D audio, which may be used in augmented reality. A 3D audio signal may be generated based on sensor data collected from the actual room in which the listener is located and the actual position of the listener in the room. The 3D audio signal may include a number of components that are determined based on the collected sensor data and the listener'"'"'s location. For example, a number of (virtual) sound paths between a virtual sound source and the listener may be determined The sensor data may be used to estimate materials in the room, such that the affect that those materials would have on sound as it travels along the paths can be determined In some embodiments, sensor data may be used to collect physical characteristics of the listener such that a suitable HRTF may be determined from a library of HRTFs.

Claims

1. A method comprising:

determining physical characteristics of a room based on sensor data;

determining a location of a listener in the room; and

determining a 3D audio signal based on the physical characteristics of the room and the location of the listener in the room.

2. The method of claim 1, wherein the determining a 3D audio signal includes:

determining a plurality of sound paths between a virtual sound source and the listener based on the physical characteristics of the room and the location of the listener in the room.

3. The method of claim 2, wherein the determining the 3D audio signal includes:

determining a plurality of components of the 3D audio signal, a component is determined for each of the plurality of sound paths; and

applying a head related transfer function (HRTF) for the listener to each of the plurality of components.

4. The method of claim 1, further comprising estimating materials of objects in the room based on the sensor data, the determining a 3D audio signal is based on an effect that the materials will have on sound.

5. The method of claim 1, wherein the determining a location of the listener in the room includes:

collecting image sensor data; and

determining a position and a rotation of the listener in the room based on the image sensor data.

6. The method of claim 1, wherein the determining a location of the listener in the room includes tracking the listener'"'"'s position in the room.

7. The method of claim 1, further comprising:

collecting image sensor data of the listener;

determining physical characteristics of the listener from the image sensor data of the listener; and

determining an HRTF for the listener from a library of HRTFs based on the physical characteristics of the listener, the determining a 3D audio signal is based on the HRTF for the listener.

8. The method of claim 7, further comprising:

storing the physical characteristics of the listener at a first time;

identifying the listener based on biometric information at a second time that is later than the first time, the biometric information is other than the physical characteristics of the listener; and

selecting the HRTF from the library based on the stored physical characteristics for the identified listener.

9. The method of claim 1, wherein the sensor data includes sensor data from one or more microphones on the listener.

10. An apparatus, comprising:

one or more sensors;

a processor coupled to the one or more sensors; and

a computer readable storage medium coupled to the processor, the computer readable storage medium having instructions stored thereon which, when executed on the processor, cause the processor to collect data pertaining to an environment and a listener using the sensors;

the processor determines physical characteristics of the environment based on the collected sensor data;

the processor determines a location of the listener in the environment based on the collected sensor data;

the processor determines a plurality of components for a 3D audio signal based on the physical characteristics of the environment and the location of the listener in the environment;

the processor applies a head related transfer function (HRTF) for the listener to each of the plurality of components; and

the processor provides the 3D audio signal.

11. The apparatus of claim 10, wherein the processor estimates materials of objects in the environment based on the collected sensor data;

and the processor modifies the components of the 3D audio signal based on a predicted effect that the materials will have on sound.

12. The apparatus of claim 10, wherein the processor determines a position and a rotation of the listener in the environment based on collected sensor data, the one or more sensors includes a depth sensor.

13. The apparatus of claim 10, wherein the processor determines physical characteristics of the listener from the collected sensor data;

and the processor determines the HRTF from a library of HRTFs based on the physical characteristics of the listener.

14. The apparatus of claim 13, wherein the processor:

stores either the determined HRTF or the physical characteristics of the listener at a first time;

the processor identifies the listener based on biometric information at a second time that is later than the first time, the biometric information is other than the physical characteristics; and

the processor either accesses the stored HRTF or determines an HRTF from the library based on the stored physical characteristics for the identified user.

15. The apparatus of claim 10, wherein the processor determines a plurality of sound paths between a virtual sound source and the listener based on the data that describes physical characteristics of the environment and the location of the listener in the environment, the processor determines one of the components of the audio signal for each of the plurality of sound paths.

16. A method for providing a 3D audio signal, comprising:

collecting sensor data that includes depth information pertaining to a listener;

determining physical parameters of the listener based on the depth information pertaining to the listener;

determining a head related transfer function (HRTF) for the listener based on a library of HRTFs, determining the HRTF is based on the physical parameters of the listener;

collecting sensor data that includes depth information pertaining to a room;

determining physical parameters of the room based on the depth information pertaining to the room;

determining a location of the listener in the room;

determining a plurality of sound paths between a virtual sound source and the listener based on the physical parameters of the room and the location of the listener in the room;

determining a component of a 3D audio signal for each of the plurality of sound paths, the determining is based on the physical parameters of the room;

applying the HRTF for the listener to each of the components of the 3D audio signal; and

providing the 3D audio signal.

17. The method of claim 16, further comprising determining a reverb tail to apply to the 3D audio signal based on selecting at least one reverb tail from a library of stored reverb tails, the selecting is based on the physical parameters of the room.

18. The method of claim 16, further comprising estimating materials of objects in the room based on the depth information pertaining to the room and/or RGB information pertaining to the room, the determining a component of the 3D audio signal is based on a predicted effect that materials on a given sound path will have on sound.

19. The method of claim 16, wherein the determining a component of a 3D audio signal for each of the plurality of sound paths includes assigning an effect that objects in the room would have on sound if replaced by materials in a virtual environment, the determining a component of the 3D audio signal is based on a predicted effect that objects on a given sound path would have on sound.

20. The method of claim 16, wherein the determining a head related transfer function (HRTF) includes:

playing a virtual 3D sound for the user while applying a first HRTF from the library, the first HRTF is selected based on the physical parameters of the listener;

instructing the listener to indicate the apparent source of the virtual sound;

analyzing effectiveness of the first HRTF; and

repeating the playing, the instructing, and the analyzing for one or more additional HRTFs from the library that are selected based on the physical parameters of the listener.