Methods and apparatus for source location estimation from microphone-array time-delay estimates

• US 6,243,471 B1
• Filed: 04/06/1998
• Issued: 06/05/2001
• Est. Priority Date: 03/07/1995
• Status: Expired due to Term

• Current Assignee: Brown University
• Original Assignee: Brown University
• Inventors: Silverman, Harvey F., Adcock, John E., Brandstein, Michael S.
• Primary Examiner(s): Nguyen, Duc
• Application Number:
  US09/055,556
• Time in Patent Office: 1,156 Days
• Field of Search: 348/14-15, 348/169, 381/91-92, 381/26, 381/122, 381/306, 381/56, 381/333
• US Class Current: 381/92
• CPC Class Codes: G01V 1/001 Acoustic presence detection
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Apparatus for determining the location of a signal-generating source (e.g., a conferee in a telephone conference) includes at least three sensors (e.g., microphones) arranged in a plurality of sets, each having two or more sensors. A surface-finding element responds to receipt at each sensor set of signals (e.g., speech) from the source for identifying a geometric surface (e.g., the surface of a hyperboloid or cone) representing potential locations of the source as a function of sensor locations and time difference of arrival of the signals. A location-approximating element coupled to two or more of the sets identifies a line that further defines potential source locations at the intersection of the surfaces. A location signal representing those potential locations is generated in accord with parameters of that line. Further functionality generates generating the location signal as a function of closest intersections the plural ones of the aforementioned lines.

Claims

1. A camera steering apparatus for use in a video teleconferencing system, the apparatus being responsive to at least three microphones arrangeable in a plurality of sets, each set including two or more microphones, the apparatus comprising:

a data processor, coupled with the microphones, for responding to receipt of a source signal by the microphones, the data processor including

i) means for identifying a bearing line, each bearing line representing a locus of potential source locations, and each bearing line being defined as a function of the locations of microphones in the plurality of sets and a time difference of arrival signal of the source signal at those microphones, and

ii) intersection-finding means for generating a location signal representing a location of the source as a function of the points of closest intersection of a plurality of bearing lines, and

at least one motor for steering a camera towards the source as a function of the location signal.

2. The camera steering apparatus according to claim 1, wherein the means for identifying a bearing line further includes surface-finding means for identifying a geometric surface representing a locus of potential source locations as a function of locations of the microphones in a set and a time difference of arrival of the source signal at those microphones, and wherein the bearing line is defined as a function of an intersection of geometric surfaces identified by two or more microphone sets.

3. The camera steering apparatus according to claim 1, further including microphones designed for picking up speech such that the camera tracks the location of an active speaker.

4. The camera steering apparatus according to claim 1, further including an analog-to-digital converter coupled with the microphone for converting analog electrical signals output by the microphone to digital electrical signals.

5. The camera steering apparatus according to claim 4, further including windowing elements for selecting discrete portions of the digital electrical signals.

6. The apparatus according to claim 1, wherein

A. each microphone set includes two microphones, and

B. at least one pair of microphone sets are substantially centered and orthogonal to one another, each such pair of substantially centered and orthogonal sets being arranged such that

(i) a linear segment defined by microphone locations of one of the sets in the pair is substantially orthogonal to a linear segment defined by microphone locations of the other of the sets in the pair,

(ii) centerpoints of the linear segments defined by microphone locations of each of the sets are substantially aligned with one another.

7. The apparatus according to claim 2, wherein the surface-finding means includes means for identifying, in connection with receipt of a source signal by each microphone set, a substantially conical surface representing a locus of potential source locations, the substantially conical surface having a direction angle θ

, with respect to a linear segment defined by m₁ and m₂, in accord with the expression $\mathrm{\theta <br><br>}={\cos }^{-1}<br><br>\left(\frac{c·<br><br>\mathrm{\tau <br><br>}}{<br><br>m_1-m_2<br><br>}\right)$

where c is a speed of propagation of the source signal,

τ

is a measured time difference of arrival of the source signal at the microphones of the microphone set,

m₁ is a location of a first microphone in the microphone set, and

m₂ is a location of a second microphone in the microphone set.

8. The apparatus according to claim 7, wherein the means for identifying a bearing line includes means for identifying a line defined as a function of an intersection of the substantially conical surfaces, the line having direction angles α

, β and

γ

from the substantially aligned centerpoints, the direction angles being defined in accord with the expression

9. The apparatus according to claim 1, wherein the intersection-finding means includes means for identifying points of closest intersection, s_jk and s_kj, of two such lines, l_j and l_k, in accord with the expression

10. The apparatus according to claim 9, wherein the intersection-finding means includes means for generating the location signal to as a weighted function of the closest points of closest intersection of a plurality of the lines.

11. The apparatus according to claim 10, wherein the intersection-finding means includes means for generating the location signal as a function of a point ŝ

defined in accord with the expression $\hat{s}=\frac{\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br>\underset{k=1,k\ne <br><br>j}{\overset{M}{\sum <br><br>}}<br><br>W_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}<br><br>s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}}{\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br>\underset{k=1,k\ne <br><br>j}{\overset{M}{\sum <br><br>}}<br><br>W_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}}$

where W_jk is a weight associated with each potential source location in accord with the expression $\begin{array}{c}{W}_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}=\text{\hspace{1em}<br><br>}<br><br>\underset{i=1}{\overset{M}{\prod <br><br>}}<br><br>P<br><br>\left(T<br><br>\left(\left\{m_{\mathrm{l1}},m_{\mathrm{l2}}\right\},s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}\right),{\mathrm{\tau <br><br>}}_{\mathrm{l12}},{\mathrm{\sigma <br><br>}}_{\mathrm{l12}}^2\right)·<br><br>\\ \text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(T<br><br>\left(\left\{m_{\mathrm{l3}},m_{\mathrm{l4}}\right\},s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}\right),{\mathrm{\tau <br><br>}}_{\mathrm{l34}},{\mathrm{\sigma <br><br>}}_{\mathrm{l34}}^2\right)\end{array}$

M is a number of pairs of microphone sets,

m_l1 is the location of the first microphone in the first microphone set of the lth microphone set pair,

m_l2 is the location of the second microphone in the first microphone set of the lth microphone set pair,

m_l3 is the location of the first microphone in the second microphone set of the lth microphone set pair,

m_l4 is the location of the second microphone in the second microphone set of the lth microphone set pair,

τ

_l12 is the measured time difference of arrival of the source signal at the microphones of a first microphone set of the lth pair of microphone sets,

τ

_l34 is the measured time difference of arrival of the source signal at the microphones of a second microphone set of the lth pair of microphone sets,

σ

₁₂² is a variance of a measured time difference of arrival of the source signal at the microphones of a first microphone set of the lth pair of microphone sets,

σ

₃₄² is a variance of a measured time difference of arrival of the source signal at the microphones of a second microphone set of the lth pair of microphone sets,

P(x,m,σ

²)is the value a Gaussian distribution with mean m and variance σ

² in accord with the expression $P<br><br>\left(x,m,{\mathrm{\sigma <br><br>}}^2\right)=\frac{1}{\mathrm{\sigma <br><br>}<br><br>\sqrt{2<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\left(\frac{-{\left(x-m\right)}^2}{2<br><br>{\mathrm{\sigma <br><br>}}^2}\right)$

T({m_l1, m_l2},s) is a true time-difference of a source signal originating at position s at the microphones of an lth microphone set pair in accord with the expression $T<br><br>\left(\left\{m_1,m_2\right\},s\right)=\frac{<br><br>s-m_1<br><br>-<br><br>s-m_2<br><br>}{c}$

m₁ and m₂ are locations of the first and second microphones in a microphone set.

12. A steerable camera for use in a video teleconferencing system, the steerable camera being responsive to at least three microphones arranged in a plurality of sets, each set including two or more microphones, the steerable camera comprising:

a moveable camera,

a data processor, coupled with the microphones, for responding to receipt of a source signal by the microphones, the data processor including

i) means for identifying a bearing line, each bearing line representing a locus of potential source locations, and each bearing line being defined as a function of the locations of microphones in the plurality of sets and a time difference of arrival signal of the source signal at those microphones, and

ii) intersection-finding means for generating a location signal representing a location of the source as a function of the points of closest intersection of a plurality of bearing lines, and

a motor coupled with the moveable camera for steering the camera towards the source as a function of the location signal.

13. Method for steering a camera towards a location of a source that generates source signals, the method comprising the steps of

A. receiving the source signal with at least three microphones arranged in a plurality of sets, each set including two or more microphones,

B. identifying a bearing line, each bearing line representing a locus of potential source locations, and each bearing line being defined as a function of the locations of microphones in the plurality of sets and a time difference of arrival signal of the source signal at those microphones,

C. generating a location signal representing a location of the source as a function of the points of closest intersection of a plurality of bearing lines, and

D. steering the camera towards the location of the source as a function of the location signal.

14. The method according to claim 13, wherein the step of identifying a bearing line includes finding a geometric surface representing a locus of potential source locations as a function of locations of the microphones in a set and a time difference of arrival of the source signal at those microphones, and wherein the bearing line is determined by identifying an intersection of the geometric surfaces identified by two or more microphone sets.

15. The method according to claim 14, wherein the surface-finding step includes the step of identifying, in connection with receipt of a source signal by each microphone set, a substantially conical surface representing a locus of potential source locations, the substantially conical surface having a direction angle θ

, with respect to a linear segment defined by m₁ and m₂, in accord with the expression $\mathrm{\theta <br><br>}={\cos }^{-1}<br><br>\left(\frac{c·<br><br>\mathrm{\tau <br><br>}}{<br><br>m_1-m_2<br><br>}\right)$

where c is a speed of propagation of the source signal,

τ

is a measured time difference of arrival of the source signal at the microphones of the microphone set,

m₁ is a location of a first microphone in the microphone set, and

m₂ is a location of a second microphone in the microphone set.

16. The method according to claim 15, wherein the step of identifying a bearing line includes means identifying a line defined as a function of an intersection of the substantially conical surfaces, the line having direction angles α

, β and

γ

from the substantially aligned centerpoints, the direction angles being defined in accord with the expression

17. The method according to claim 13, further including the steps of receiving source signals with microphones designed for picking up speech and tracking the location of an active speaker with the camera.

18. The method according to claim 13, further including the step of converting the analog electrical signals generating by the microphones into digital electrical signals.

19. The method according to claim 18, further including the step of windowing selected discrete portions of the digital electrical signals.

20. The method according to claim 13, further including the steps of

A. providing each microphone set with two microphones, and

B. arranging at least one pair of microphone sets to be substantially centered and orthogonal to one another, each such pair of substantially centered and orthogonal sets being arranged such that

(i) a linear segment defined by microphone locations of one of the sets in the pair is substantially orthogonal to a linear segment defined by microphone locations of the other of the sets in the pair,

(ii) centerpoints of the linear segments defined by microphone locations of each of the sets are substantially aligned with one another.

21. The method according to claim 13, wherein the step of generating a location signal includes the step of identifying points of closest intersection, s_jk and s_kj, of two such bearing lines, l_j and l_k, in accord with the expression

22. The method according to claim 21, wherein the step of generating a location signal includes the step of generating the location signal as a weighted function of the closest points of closest intersection of a plurality of the bearing lines.

23. The method according to claim 22, wherein the step of generating a location signal includes the step of generating the location signal as a function of a point ŝ

defined in accord with the expression $\hat{s}=\frac{\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br>\underset{k=1,k\ne <br><br>j}{\overset{M}{\sum <br><br>}}<br><br>W_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}<br><br>s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}}{\underset{j=1}{\overset{M}{\sum <br><br>}}<br><br>\underset{k=1,k\ne <br><br>j}{\overset{M}{\sum <br><br>}}<br><br>W_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}}$

where W_jk is a weight associated with each point of closest intersection in accord with the expression $\begin{array}{c}{W}_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}=\text{\hspace{1em}<br><br>}<br><br>\underset{i=1}{\overset{M}{\prod <br><br>}}<br><br>P<br><br>\left(T<br><br>\left(\left\{m_{\mathrm{l1}},m_{\mathrm{l2}}\right\},s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}\right),{\mathrm{\tau <br><br>}}_{\mathrm{l12}},{\mathrm{\sigma <br><br>}}_{\mathrm{l12}}^2\right)·<br><br>\\ \text{\hspace{1em}<br><br>}<br><br>P<br><br>\left(T<br><br>\left(\left\{m_{\mathrm{l3}},m_{\mathrm{l4}}\right\},s_{j<br><br>\text{\hspace{1em}<br><br>}<br><br>k}\right),{\mathrm{\tau <br><br>}}_{\mathrm{l34}},{\mathrm{\sigma <br><br>}}_{\mathrm{l34}}^2\right)\end{array}$

M is a number of pairs of microphone sets,

m_l1 is the location of the first microphone in the first microphone set of the lth microphone set pair,

m_l2 is the location of the second microphone in the first microphone set of the lth microphone set pair,

m_l3 is the location of the first microphone in the second microphone set of the lth microphone set pair,

m_l4 is the location of the second microphone in the second microphone set of the lth microphone set pair,

τ

_l12 is the measured time difference of arrival of the source signal at the microphones of a first microphone set of the lth pair of microphone sets,

τ

_l34 is the measured time difference of arrival of the source signal at the microphones of a second microphone set of the lth pair of microphone sets,

σ

₁₂² is a variance of a measured time difference of arrival of the source signal at the microphones of a first microphone set of the lth pair of microphone sets,

σ

₃₄² is a variance of a measured time difference of arrival of the source signal at the microphones of a second microphone set of the lth pair of microphone sets,

P(x,m,σ

²) is the value a Gaussian distribution with mean m and variance σ

² in accord with the expression $P<br><br>\left(x,m,{\mathrm{\sigma <br><br>}}^2\right)=\frac{1}{\mathrm{\sigma <br><br>}<br><br>\sqrt{2<br><br>\mathrm{\pi <br><br>}}}<br><br>\exp <br><br>\left(\frac{-{\left(x-m\right)}^2}{2<br><br>{\mathrm{\sigma <br><br>}}^2}\right)$

T({m_l1, m_l2},s) is a true time-difference of arrival of a source signal originating a position s at the microphones of an lth microphone set pair in accord with the expression $T<br><br>\left(\left\{m_1,m_2\right\},s\right)=\frac{<br><br>s-m_1<br><br>-<br><br>s-m_2<br><br>}{c}$

m₁ and m₂ are locations of the first and second microphones in a microphone set.