Signal processing methods and systems for rendering audio on virtual loudspeaker arrays

• US 10,142,755 B2
• Filed: 02/07/2017
• Issued: 11/27/2018
• Est. Priority Date: 02/18/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Boland, Francis Morgan
• Primary Examiner(s): Patel, Yogeshkumar
• Application Number:
  US15/426,629
• Publication Number:
  US 20170245082A1
• Time in Patent Office: 658 Days
• Field of Search: 381 17
• CPC Class Codes: H04S 1/005 For headphones
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04S 2420/11 Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 3/02 of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

Techniques of rendering audio involve applying a balanced-realization state space model to each head-related transfer function (HRTF) to reduce the order of an effective FIR or even an infinite impulse response (IIR) filter. Along these lines, each HRTF G(z) is derived from a head-related impulse response filter (HRIR) via, e.g., a z-transform. The data of the HRIR may be used to construct a first state space representation [A, B, C, D] of the HRTF via the relation G(z)=C(zI−A)⁻¹B+D This first state space representation is not unique and so for an FIR filter, A and B may be set to simple, binary-valued arrays, while C and D contain the HRIR data. This representation leads to a simple form of a Gramian Q whose eigenvectors provide system states that maximize the system gain as measured by a Hankel norm.

Claims

1. A method of rendering sound fields in a left ear and a right ear of a human listener, the sound fields being produced by a plurality of virtual loudspeakers, the method comprising:

obtaining, by processing circuitry of a sound rendering computer configured to render the sound fields in the left ear and the right ear of the head of the human listener, a plurality of head-related impulse responses (HRIRs), each of the plurality of HRIRs being associated with a virtual loudspeaker of the plurality of virtual loudspeakers and the left or right ear of the human listener, each of the plurality of HRIRs including samples of a sound field produced at a specified sampling rate in the left or right ear produced in response to an audio impulse produced by that virtual loudspeaker;

generating a first state space representation of each of the plurality of HRIRs, the first state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the first state space representation having a first size;

performing a state space reduction operation to produce a second state space representation of each of the plurality of HRIRs, the second state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the second state space representation having a second size that is less than the first size; and

producing a plurality of head-related transfer functions (HRTFs) based on the second state space representation, each of the plurality of HRTFs corresponding to a respective HRIR of the plurality of HRIRs, an HRTF corresponding to the respective HRIR producing, upon multiplication by a frequency-domain sound field produced by the virtual loudspeaker with which the respective HRIR is associated, a component of the sound field rendered in the left or right ear of the human listener.

2. The method as in claim 1, wherein performing the state space reduction operation includes, for each HRIR of the plurality of HRIRs:

generating a respective Gramian matrix based on the first state space representation of that HRIR, the Gramian matrix having a plurality of eigenvalues arranged in descending order of magnitude; and

generating the second state space representation of that HRIR based on the Gramian matrix and the plurality of eigenvalues, wherein the second size is equal to a number of eigenvalues of the plurality of eigenvalues greater than a specified threshold.

3. The method as in claim 2, wherein generating the second state space representation of each HRIR of the plurality of HRIRs includes forming a transformation matrix that, when applied to the Gramian matrix that is based on the first state space representation of that HRIR, produces a diagonal matrix, each diagonal element of the diagonal matrix being equal to a respective eigenvalue of the plurality of eigenvalues.

4. The method as in claim 1, further comprising, for each of the plurality of HRIRs:

generating a cepstrum of that HRIR, the cepstrum having causal samples taken at positive times and non-causal samples taken at negative times;

for each of the non-causal samples of the cepstrum, performing a phase minimization operation by adding that non-causal sample taken at a negative time to a causal sample of the cepstrum taken at the opposite of that negative time; and

producing a minimum-phase HRIR by setting each of the non-causal samples of the cepstrum to zero after performing the phase minimization operation for each of the non-causal samples of the cepstrum.

5. The method as in claim 1, further comprising generating a multiple input, multiple output (MIMO) state space representation, the MIMO state space representation including a composite matrix, a column vector matrix, and a row vector matrix, the composite matrix of the MIMO state space representation including the matrix of the first state space representation of each of the plurality of HRIRs, the column vector matrix of the MIMO state space representation including the column vector of the first state space representation of each of the plurality of HRIRs, the row vector matrix of the MIMO state space representation including the row vector of the first state space representation of each of the plurality of HRIRs;

and

wherein performing the state space reduction operation includes generating a reduced composite matrix, a reduced column vector matrix, and a reduced row vector matrix, each of the reduced composite matrix, reduced column vector matrix, and reduced row vector matrix having a size that is respectively less than a size of the composite matrix, the column vector matrix, and the row vector matrix.

6. The method as in claim 5, wherein generating the MIMO state space representation includes:

forming, as the composite matrix of the MIMO state space representation, a first block matrix having a matrix of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as a diagonal element of the first block matrix, matrices of the first state space representation of HRIRs associated with the same virtual loudspeaker being in adjacent diagonal elements of the first block matrix;

forming, as the column vector matrix of the MIMO state space representation, a second block matrix having a column vector of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as a diagonal element of the second block matrix, column vectors of the first state space representation of HRIRs associated with the same virtual loudspeaker being in adjacent diagonal elements of the second block matrix; and

forming, as the row vector matrix of the MIMO state space representation, a third block matrix having a row vector of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as an element of the third block matrix, row vectors of the first state space representation of HRIRs that render sounds in the left ear being in odd-numbered elements of a first row of the third block matrix, row vectors of the first state space representation of HRIRs that render sounds in the right ear being in even-numbered elements of a second row of the third block matrix.

7. The method as in claim 5, further comprising, prior to generating the MIMO state space representation, for each HRIR of the plurality of HRIRs, performing a single input single output (SISO) state space reduction operation to produce, as the first state space representation of that HRIR, a SISO state space representation of that HRIR.

8. The method as in claim 1, wherein, for each of the plurality of virtual loudspeakers, there are a left HRIR and a right HRIR of the plurality of HRIRs associated with that virtual loudspeaker, the left HRIR producing, upon multiplication by the frequency-domain sound field produced by that virtual loudspeaker, the component of the sound field rendered in the left ear of the human listener, the right HRIR producing, upon multiplication by the frequency-domain sound field produced by that virtual loudspeaker, the component of the sound field rendered in the right ear of the human listener;

and

wherein, for each of the plurality of virtual loudspeakers, there is an interaural time delay (ITD) between the left HRIR associated with that virtual loudspeaker and the right HRIR associated with that virtual loudspeaker, the ITD being manifested in the left HRIR and the right HRIR by a difference between a number of initial samples of the sound field of the left HRIR that have zero values and a number of initial samples of the sound field of the right HRIR that have zero values.

9. The method as in claim 8, further comprising:

generating an ITD unit subsystem matrix based on the ITD between the left HRIR and right HRIR associated with each of the plurality of virtual loudspeakers; and

multiplying the plurality of HRTFs by the ITD unit subsystem matrix to produce a plurality of delayed HRTFs.

10. The method as in claim 1, wherein each of the plurality of HRTFs are represented by finite impulse filters (FIRs);

and

wherein the method further comprises performing a conversion operation on each of the plurality of HRTFs to produce another plurality of HRTFs that are each represented by infinite impulse response filters (IIRs).

11. A computer program product comprising a nontransitive storage medium, the computer program product including code that, when executed by processing circuitry of a sound rendering computer configured to render sound fields in a left ear and a right ear of a human listener, causes the processing circuitry to perform a method, the method comprising:

obtaining a plurality of head-related impulse responses (HRIRs), each of the plurality of HRIRs being associated with a virtual loudspeaker of a plurality of virtual loudspeakers and the left or right ear of the human listener, each of the plurality of HRIRs including samples of a sound field produced at a specified sampling rate in the left or right ear produced in response to an audio impulse produced by that virtual loudspeaker;

generating a first state space representation of each of the plurality of HRIRs, the first state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the first state space representation having a first size;

performing a state space reduction operation to produce a second state space representation of each of the plurality of HRIRs, the second state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the second state space representation having a second size that is less than the first size; and

producing a plurality of head-related transfer functions (HRTFs) based on the second state space representation, each of the plurality of HRTFs corresponding to a respective HRIR of the plurality of HRIRs, an HRTF corresponding to a respective HRIR producing, upon multiplication by a frequency-domain sound field produced by the virtual loudspeaker with which the respective HRIR is associated, a component of a sound field rendered in the left or right ear of the human listener.

12. The computer program product as in claim 11, wherein performing the state space reduction operation includes, for each HRIR of the plurality of HRIRs:

generating a respective Gramian matrix based on the first state space representation of that HRIR, the Gramian matrix having a plurality of eigenvalues arranged in descending order of magnitude; and

generating the second state space representation of that HRIR based on the Gramian matrix and the plurality of eigenvalues, wherein the second size is equal to a number of eigenvalues of the plurality of eigenvalues greater than a specified threshold.

13. The computer program product as in claim 12, wherein generating the second state space representation of each HRIR of the plurality of HRIRs includes forming a transformation matrix that, when applied to the Gramian matrix that is based on the first state space representation of that HRIR, produces a diagonal matrix, each diagonal element of the diagonal matrix being equal to a respective eigenvalue of the plurality of eigenvalues.

14. The computer program product as in claim 11, wherein the method further comprises, for each of the plurality of HRIRs:

generating a cepstrum of that HRIR, the cepstrum having causal samples taken at positive times and non-causal samples taken at negative times;

for each of the non-causal samples of the cepstrum, performing a phase minimization operation by adding that non-causal sample taken at a negative time to a causal sample of the cepstrum taken at the opposite of that negative time; and

producing a minimum-phase HRIR by setting each of the non-causal samples of the cepstrum to zero after performing the phase minimization operation for each of the non-causal samples of the cepstrum.

15. The computer program product as in claim 11, wherein the method further comprises generating a multiple input, multiple output (MIMO) state space representation, the MIMO state space representation including a composite matrix, a column vector matrix, and a row vector matrix, the composite matrix of the MIMO state space representation including the matrix of the first state space representation of each of the plurality of HRIRs, the column vector matrix of the MIMO state space representation including the column vector of the first state space representation of each of the plurality of HRIRs, the row vector matrix of the MIMO state space representation including the row vector of the first state space representation of each of the plurality of HRIRs;

and

wherein performing the state space reduction operation includes generating a reduced composite matrix, a reduced column vector matrix, and a reduced row vector matrix, each of the reduced composite matrix, reduced column vector matrix, and reduced row vector matrix having a size that is respectively less than a size of the composite matrix, the column vector matrix, and the row vector matrix.

16. The computer program product as in claim 15, wherein generating the MIMO state space representation includes:

forming, as the composite matrix of the MIMO state space representation, a first block matrix having a matrix of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as a diagonal element of the first block matrix, matrices of the first state space representation of HRIRs associated with the same virtual loudspeaker being in adjacent diagonal elements of the first block matrix;

forming, as the column vector matrix of the MIMO state space representation, a second block matrix having a column vector of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as a diagonal element of the second block matrix, column vectors of the first state space representation of HRIRs associated with the same virtual loudspeaker being in adjacent diagonal elements of the second block matrix; and

forming, as the row vector matrix of the MIMO state space representation, a third block matrix having a row vector of the first state space representation of an HRIR associated with a virtual loudspeaker of the plurality of virtual loudspeakers as an element of the third block matrix, row vectors of the first state space representation of HRIRs that render sounds in the left ear being in odd-numbered elements of a first row of the third block matrix, row vectors of the first state space representation of HRIRs that render sounds in the right ear being in even-numbered elements of a second row of the third block matrix.

17. The computer program product as in claim 11, wherein, for each of the plurality of virtual loudspeakers, there are a left HRIR and a right HRIR of the plurality of HRIRs associated with that virtual loudspeaker, the left HRIR producing, upon multiplication by the frequency-domain sound field produced by that virtual loudspeaker, the component of the sound field rendered in the left ear of the human listener, the right HRIR producing, upon multiplication by the frequency-domain sound field produced by that virtual loudspeaker, the component of the sound field rendered in the right ear of the human listener;

and

wherein, for each of the plurality of virtual loudspeakers, there is an interaural time delay (ITD) between the left HRIR associated with that virtual loudspeaker and the right HRIR associated with that virtual loudspeaker, the ITD being manifested in the left HRIR and the right HRIR by a difference between a number of initial samples of the sound field of the left HRIR that have zero values and a number of initial samples of the sound field of the right HRIR that have zero values.

18. The computer program product as in claim 17, wherein the method further comprises:

generating an ITD unit subsystem matrix based on the ITD between the left HRIR and right HRIR associated with each of the plurality of virtual loudspeakers; and

multiplying the plurality of HRTFs by the ITD unit subsystem matrix to produce a plurality of delayed HRTFs.

19. The computer program product as in claim 11, wherein each of the plurality of HRTFs are represented by finite impulse filters (FIRs);

and

wherein the method further comprises performing a conversion operation on each of the plurality of HRTFs to produce another plurality of HRTFs that are each represented by infinite impulse response filters (IIRs).

20. An electronic apparatus configured to render sound fields in a left ear and a right ear of a human listener, the electronic apparatus comprising:

memory; and

controlling circuitry coupled to the memory, the controlling circuitry being configured to;

obtain a plurality of head-related impulse responses (HRIRs), each of the plurality of HRIRs being associated with a virtual loudspeaker of a plurality of virtual loudspeakers and the left or right ear of the human listener, each of the plurality of HRIRs including samples of a sound field produced at a specified sampling rate in the left or right ear produced in response to an audio impulse produced by that virtual loudspeaker;

generate a first state space representation of each of the plurality of HRIRs, the first state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the first state space representation having a first size;

perform a state space reduction operation to produce a second state space representation of each of the plurality of HRIRs, the second state space representation including a matrix, a column vector, and a row vector, each of the matrix, the column vector, and the row vector of the second state space representation having a second size that is less than the first size; and

produce a plurality of head-related transfer functions (HRTFs) based on the second state space representation, each of the plurality of HRTFs corresponding to a respective HRIR of the plurality of HRIRs, an HRTF corresponding to a respective HRIR producing, upon multiplication by a frequency-domain sound field produced by the virtual loudspeaker with which the respective HRIR is associated, a component of a sound field rendered in the left or right ear of the human listener.