Audio spatialization using reflective room model

• US 9,107,021 B2
• Filed: 04/30/2010
• Issued: 08/11/2015
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Florencio, Dinei A., Zhang, Cha, Song, Myung-Suk
• Primary Examiner(s): Goins, Davetta W
• Assistant Examiner(s): SELLERS, DANIEL R
• Application Number:
  US12/772,014
• Publication Number:
  US 20110268281A1
• Time in Patent Office: 1,929 Days
• Field of Search: 381/17, 381/26, 381/58, 381/59, 381/66, 381 941- 943, 381/303, 381/310
• US Class Current: 1/1
• CPC Class Codes: G06F 30/20 Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: H04R 2420/01 Input selection or mixing for amplifiers or loudspeakers for hearing aids H04R25/43
• CPC Class Codes: H04S 1/007 in which the audio signals are in digital form data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/305 Electronic adaptation of stereophonic audio signals to reverberation of the listening space H04S7/301 takes precedence

Abstract

Described are systems and methods performed by computer to reduce crosstalk produced by loudspeakers when rendering binaural sound that is emitted from the loudspeakers into a room. The room may have sound-reflecting surfaces that reflect some of the sound produced by the loudspeakers. To reduce crosstalk, a room model stored by the computer, is accessed. The room model models at least sound reflected by one or more of the physical surfaces. The room model is used to calculate a model of an audio channel from the loudspeakers to a listener. The model of the audio channel models sound transmission from the loudspeakers to the listener. The computer uses the model of the audio channel to cancel crosstalk from the loudspeakers when rendering the binaural sound.

Claims

1. A method performed by a computer to reduce crosstalk produced by loudspeakers when rendering binaural sound that is emitted from the loudspeakers into a room, the room comprising a physical area with a plurality of sound-reflecting physical surfaces that reflect some of the sound produced by the loudspeakers, the reflected and un-reflected sound from the loudspeakers available to be heard by a listener if present in the room, the method comprising:

receiving positions and/or orientations of the listener;

accessing a static room model stored by the computer, the room model comprising indicia of planar surfaces, the room model able to model sound from positions of the loudspeakers reflected by one or more of the physical surfaces represented in the room model to arbitrary positions of the listener;

for each received position and/or orientation, using the room model to calculate, according to the received position and/or orientation of the listener, by the computer, a model of an audio channel from the loudspeakers to the listener, the model of the audio channel comprising an estimated room impulse response comprising a first component modeling sound transmission both from the loudspeakers directly to the listener and comprising a second component modeling sound transmission from the loudspeakers indirectly by reflection from the physical surfaces of the room; and

for each computed model of the audio channel, using, by the computer, the first component of the model of the audio channel to cancel direct crosstalk from the loudspeakers and the second component of the model to cancel reflected crosstalk from the loudspeakers to render the binaural sound.

2. A method according to claim 1, wherein the modeled reflected sound corresponds to an image of a loudspeaker and a modeled surface that corresponds to a physical surface of the room.

3. A method according to claim 1, wherein the calculating the model of the audio channel from the loudspeakers to the listener is performed using information about the current position and/or orientation of the listener in the room.

4. A method according to claim 3, wherein the orientation and position of the listener is obtained by a video based head tracker.

5. A method according to claim 3, where the model of the audio channel takes into account a different HRTF (head related transfer function) in relation to each of the direct or reflected sounds, each HRTF corresponding to the distance and direction of arrival of each corresponding direct or reflected sound.

6. A method according to claim 1, wherein the model of the audio channel comprises an acoustic transfer matrix, and wherein the using the model of the audio channel to cancel crosstalk comprises computing an inverse of the acoustic transfer matrix.

7. A method according to claim 6, wherein the inverse of the acoustic transfer matrix is used to compute a crosstalk cancellation matrix, and the cancelling crosstalk is performed according to the crosstalk cancellation matrix.

8. A method according to claim 1, further comprising computing the room model by using sound data captured from a microphone array to infer geometry of the room.

9. A method according to claim 1, further comprising repeatedly determining the current position and orientation of the listener in the room and repeatedly using the different current positions and orientations that result from movement of the listener to repeatedly calculate new models of the audio channel, and repeatedly using the new models of the audio channel to perform the crosstalk cancellation.

10. One or more computer-readable storage apparatuses storing information that causes a computer to perform a process of reducing crosstalk sound produced when the computer renders binaural sound, wherein sound transmitted from a loudspeaker reaches a first ear of a listener and a second ear of the listener, the crosstalk sound comprising sound transmitted from the loudspeaker and reflected from a surface of a room to the second ear of the listener, the process comprising:

repeatedly accessing a room model stored by the computer and repeatedly accessing stance information stored by the computer to repeatedly compute estimated room impulse responses that includes a crosstalk cancellation function, the crosstalk cancellation function including a component to cancel an estimation of direct crosstalk and including a component to cancel an estimation of indirect crosstalk reflected by the room, the room model able to model reflection of sound from the loudspeaker to arbitrary positions of the listener, the room model not comprising a measured audio impulse response, the stance information comprising a current stance of the listener that is repeatedly updated as the listener moves; and

repeatedly reducing the crosstalk sound by using the estimated room impulse response to generate a binaural audio signal as the stance of the listener changes, the speakers emitting sound corresponding to the generated binaural audio signal.

11. One or more computer-readable storage apparatuses according to claim 10, wherein the room model models reflected sound.

12. One or more computer-readable storage apparatuses according to claim 11, using the accessed room model and the stance information comprises computing an acoustic transfer matrix based on the accessed room model and the stance information, the acoustic transfer matrix modeling how sound is transmitted from the loudspeaker to the listener according to the current stance of the listener and according to reflection of crosstalk sound reflected from a surface to the listener.

13. One or more computer-readable storage apparatuses according to claim 12, further comprising computing an inverse of the acoustic transfer matrix, and based on the inverse of the acoustic transfer matrix, computing a crosstalk canceller matrix that is used to reduce the crosstalk sound when generating the binaural sound signal.

14. One or more computer-readable storage apparatuses according to claim 10, further comprising performing binaural synthesis on sound data to produce synthetic binaural sound data, and then performing the reducing of the crosstalk sound by performing crosstalk cancellation on the binaural sound data.

15. One or more computer-readable storage apparatuses according to claim 14, the room model modeling a ghost image of the loudspeaker, wherein the performing crosstalk cancellation comprises applying a matrix that includes one or more elements that are computed based on the image of the loudspeaker.

16. One or more computer-readable storage apparatuses according to claim 15, wherein the image comprises a modeled position of where the listener would perceive the loudspeaker to be when hearing the loudspeaker reflected to the listener by a surface modeled by the room model.

17. A computing device that produces binaural sound, the computing device comprising a processor and storage in communication with the processor and storing instructions for the processor, the computing device including loudspeakers in a room comprised of sound-reflecting surfaces, the loudspeakers emitting synthesized binaural sound that reflects off of the surfaces and then reaches ears of a person in the room, the computing device comprising:

a binaural synthesis module that generates a binaural sound signal;

a crosstalk cancellation module that receives the binaural sound signal and modifies the binaural sound signal by;

computing an estimated room impulse response comprising an acoustic transfer matrix that corresponds to acoustic indirect transfer from the loudspeakers to the reflective surfaces to the person and acoustic direct transfer from the loudspeakers to the person, the computing using a model representing the reflective surfaces such that the model can be used to compute sound reflection from the loudspeakers to arbitrary positions,

using the acoustic transfer matrix to compute a crosstalk cancellation function that includes a component to cancel estimated direct crosstalk and includes a component to cancel estimated reflected crosstalk, and

modifying the binaural sound signal generated by the binaural synthesis module according to the crosstalk cancellation function.

18. A computing device according to claim 17, the modifying the binaural sound signal according to the acoustic transfer matrix comprising:

computing an inverse of the acoustic transfer matrix.

19. A computing device according to claim 18, further comprising computing a crosstalk cancellation matrix from the inverse of the acoustic transfer matrix, and applying the crosstalk cancellation matrix to the binaural sound signal to reduce the crosstalk sound.

20. A computing device according to claim 17, wherein the acoustic transfer matrix is computed based on stance information indicating a position and orientation of the person relative to the loudspeakers, the stance information having been obtained by a sensor in communication with the computing device.