System and method for steering directional microphones

• US 5,940,118 A
• Filed: 12/22/1997
• Issued: 08/17/1999
• Est. Priority Date: 12/22/1997
• Status: Expired due to Term

• Current Assignee: RPX Clearinghouse LLC (RPX Corporation)
• Original Assignee: Nortel Networks Corporation
• Inventors: Van Schyndel, Andre J.
• Primary Examiner(s): Loomis, Paul
• Assistant Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US08/996,135
• Time in Patent Office: 603 Days
• Field of Search: 348/14, 348/15, 348/169
• US Class Current: 348/14.05
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A system that selects and/or steers a directional steerable microphone system based on input from an optical transducer is described. An optical transducer, such as a video camera, provides video input to a processor that controls a steerable directional microphone (such as a microphone array) in the direction of audience members that exhibit physical cues commonly expressed by persons who are speaking or are about to speak.

Claims

1. An audio system for a conference room containing audience members comprising:

optical transducer means directed at the audience members;

a processor receiving information from the optical transducer means and determining from that information which audience members are speaking or are about to speak; and

a steerable directional microphone connected to the processor, the processor causing activation of the microphone which is directed towards audience members who are speaking or are about to speak.

2. An audio system of claim 1 wherein said optical transducer means comprises at least one video camera.

3. An audio system of claim 1 wherein said processor includes a frame grabber memory and processing device for acquiring, storing and processing video images received from said optical transducer means.

4. An audio system of claim 1 wherein said steerable directional microphone is a steerable microphone array.

5. An audio system of claim 4 wherein said processor causes activation of a microphone of the array which microphone is directed towards an audience member who is speaking or is about to speak.

6. An audio system of claim 1 wherein said processor is connected to memory for storing a database of a reference set of image signatures corresponding to predetermined facial features of an average audience member.

7. An audio system of claim 1 further including at least one loudspeaker for broadcasting the sounds received by said steerable directional microphone throughout the conference room.

8. An audio system of claim 1 wherein said processor determines which audience members are about to speak by calculating a sum of factors based on the identification of one or more of the following physical cues:

i. an audience member looking carefully at the current talker;

ii. an audience member taking a quick, audible breath;

iii. an audience member nodding in agreement or disagreement;

iv. an audience member using arm motions;

v. an audience member straightening of body posture; and

vi. an audience member opening his/her mouth.

9. An audio system of claim 1 wherein said processor calculates the three-dimensional coordinates of audience members that are speaking or are about to speak.

10. An audio system of claim 1 wherein said processor calculates the three-dimensional coordinates of audience members that are speaking or are about to speak and wherein said optical transducer means comprises two video cameras, said video cameras being offset from each other by a known distance, and wherein said three-dimensional coordinates are calculated by means of a parallax calculation based on the known distance between said video cameras.

11. An audio system of claim 1 wherein said steerable directional microphone has multiple beam capability to pick up the speech of two or more simultaneous talkers.

12. An audio system of claim 1 wherein:

said optical transducer means comprises at least one video camera;

said processor includes a frame grabber memory and processing device for acquiring, storing and processing video images received from said video camera, and is connected to memory for storing a database of a reference set of image signatures corresponding to predetermined facial features of an average audience member; and

said steerable directional microphone has multiple beam capability to pick up the speech of two or more simultaneous talkers.

13. An audio system of claim 1 wherein:

said optical transducer means comprises two video cameras being offset from each other by a known distance;

said processor including a frame grabber memory and processing device for acquiring, storing and processing video images received from said video cameras, and said processor calculates the three-dimensional coordinates of audience members that are speaking or are about to speak by means of a parallax calculation based on the known distance between said video cameras; and

said steerable directional microphone has multiple beam capability to pick up the speech of two or more simultaneous talkers.

14. An audio system of claim 1 wherein:

said optical transducer means comprises two video cameras being offset from each other by a known distance; and

said processor;

including a frame grabber memory and processing device for acquiring, storing and processing video images received from said video cameras;

determining which audience members are about to speak by calculating a sum of factors based on the identification of one or more of the following physical cues;

i. an audience member looking carefully at the current talker;

ii. an audience member taking a quick, audible breath;

iii. an audience member nodding in agreement or disagreement;

iv. an audience member using arm motions;

v. an audience member straightening of body posture; and

vi. an audience member opening his/her mouth; and

calculating the three-dimensional coordinates of audience members that are speaking or are about to speak by means of a parallax calculation based on the known distance between said video cameras.

15. An audio system of claim 1 further including a remote station connected to said conference room by a transmission system, wherein said remote station contains a video display device for displaying video signals received from said optical transducer means, and a loudspeaker for reproducing sounds received by said steerable directional microphone.

16. A telephone comprising optical transducer means directed at a talker, a processor receiving information from the optical transducer means and determining from that information the physical location of the talker;

and a steerable directional microphone connected to the processor, the processor causing activation of the microphone which is directed towards the talker.

17. A telephone of claim 16 wherein when said telephone is in hands-free mode, said steerable directional microphone has multiple beam capability to pick up the speech of two or more simultaneous talkers.

18. A telephone of claim 16 wherein said optical transducer means comprises an infrared detector for detecting heat given off by the face of a talker.

19. In a conference room containing audience members, optical transducer means directed at the audience members, a processor, and a steerable directional microphone, a method for broadcasting audio signals throughout said conference room comprising the steps of:

a. receiving, at said optical transducer means, an image of said audience members;

b. transferring said image to said processor;

c. determining from said image which audience members are speaking or are about to speak;

d. activating said steerable directional microphone towards audience members who are speaking or are about to speak; and

e. broadcasting audio signals received by said steerable directional microphone throughout the conference room.

20. The method of claim 19 wherein said determining step includes the step of performing a calculation of a sum of factors based on the identification of one or more of the following physical cues:

i. an audience member looking carefully at the current talker;

ii. an audience member taking a quick, audible breath;

iii. an audience member nodding in agreement or disagreement;

iv. an audience member using arm motions;

v. an audience member straightening of body posture;

vi. an audience member opening his/her mouth.

21. The method of claim 19 wherein said determining step includes the step of calculating the three-dimensional coordinates of audience members that are speaking or are about to speak.

22. The method of claim 19 wherein said steerable directional microphone has multiple beam capability to pick up the speech of two or more simultaneous talkers and wherein said method further includes the step of shutting off the beam of an audience member who has stopped talking.

23. The method of claim 19 wherein said step of determining from said image which audience members are speaking includes the steps of:

a. identify the coordinates of the mouths of the audience members;

b. track the mouths of the audience members;

c. identify which mouths of which audience members are moving.

24. The method of claim 19 wherein said step of determining from said image which audience members are about to speak includes the steps of:

a. identify the coordinates of the audience members;

b. track the audience members;

c. perform a calculation of a sum of factors based on the identification of one or more of the following physical cues;

i. an audience member looking carefully at the current talker;

ii. an audience member taking a quick, audible breath;

iii. an audience member nodding in agreement or disagreement;

iv. an audience member using arm motions;

v. an audience member straightening of body posture; and

vi. an audience member opening his/her mouth.

25. The method of claim 19 further including the step of steering the steerable directional microphone in the direction of greatest sound intensity when a determination cannot be made from said image of which audience members are speaking or are about to speak.

26. The method of claim 19 further including the step of deactivating the steerable directional microphone from audience members who have ceased speaking.

27. The method of claim 19 further including the step of deactivating the steerable directional microphone from audience members who are determined to be about to speak, but who do not commence speaking within a pre-determined time interval.

28. The method of claim 19 further including the steps of:

deactivating the steerable directional microphone from audience members who have ceased speaking; and

deactivating the steerable directional microphone from audience members who are determined to be about to speak, but who do not commence speaking within a pre-determined time interval.