Video and audio conferencing system with spatial audio

• US 7,667,728 B2
• Filed: 10/17/2005
• Issued: 02/23/2010
• Est. Priority Date: 10/15/2004
• Status: Active Grant

• Current Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Original Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Inventors: Kenoyer, Michael L., Oxford, William V.
• Primary Examiner(s): Ramakrishnaiah; Melur
• Application Number:
  US11/252,188
• Publication Number:
  US 20060104458A1
• Time in Patent Office: 1,590 Days
• Field of Search: 348 1401- 1416, 379/202.01, 379/406.01, 381/92, 381/122
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/14 Systems for two-way working H04N7/12, H04N7/173 take precedence
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation

Abstract

In some embodiments, spatially realistic audio may be provided for a conference call. Voices from participants on the left side of a display, in a conference call, may be directed through audio on the left side of the display at the other conferencing system in the conference call (similarly for voices from the center and right side of the display). In some embodiments, two speakers may be used in the system to create synthesized stereo sound at a location specified by directional information received as side information along with the existing audio channel. The location may be determined by using beamforming with integrated microphones on a camera or speakerphone. In some embodiments, the audio signal and directional information may be sent in the form of a left audio channel and a right audio channel.

Claims

1. A system, comprising:

a first plurality of microphones;

a second plurality of microphones for producing an audio signal of a participant; and

a computing system coupled to the first and second plurality of microphones, wherein the computing system is configured to;

beamform a response of the first plurality of microphones to determine directional information for the audio signal relative to the first plurality of microphones;

beamform a response of the second plurality of microphones to determine directional information for the audio signal relative to the second plurality of microphones;

cross correlate the beamformed response of the first plurality of microphones and the second plurality of microphones to determine final directional information for the audio signal, wherein the final directional information indicates a left side or a right side of the second plurality of microphones; and

transmit the audio signal from the second plurality of microphones based on the final directional information, wherein the computing system is configured to transmit the audio signal by transmitting a left audio channel and a right audio channel, wherein the left audio channel comprises audio from at least one beam on the left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on the right side of the second plurality of microphones.

2. The system of claim 1, wherein the first plurality of microphones are located in a camera, and wherein the second plurality of microphones are located in a speakerphone.

3. The system of claim 1, wherein the computing system is further configured to implement an echo canceller based on the determined active participant'"'"'s location.

4. The system of claim 1, further comprising a plurality of independent echo cancellers, each configured based on a potential participant location.

5. The system of claim 1, further comprising:

a multiple channel echo canceller;

wherein the multiple channel echo canceller is configured for multiple channels.

6. The system of claim 1, further comprising a remote conference system with speakers in at least two locations relative to the remote conference system, wherein the audio from the participant is reproduced at speakers in at least one of the two locations relative to the remote conference system based on the location of the participant relative to the system that transmitted the audio.

7. The system of claim 1, wherein a remote site uses the left and right audio channels to place the transmitted audio signal on speakers on a respective side of the remote site indicated by the left and right audio channels.

8. A method, comprising:

beamforming audio using a first plurality of microphones to determine directional information for a conference participant relative to a conference system;

beamforming the audio using a second plurality of microphones to determine directional information for the conference participant relative to the conference system;

cross correlating the beamformed response from the first plurality of microphones with the beamformed response of the second plurality of microphones to determine final directional information for the audio, wherein the final directional information indicates a left side or a right side of the second plurality of microphones;

capturing an audio signal of the conference participant using a second plurality of microphones;

transmitting the audio signal to a remote conference system based on the final directional information, wherein said transmitting the audio signal comprises transmitting a left audio channel and a right audio channel, wherein the left audio channel comprises audio from at least one beam on the left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on the right side of the second plurality of microphones;

wherein the audio signal is used to reproduce audio from the participant at the remote conference system.

9. The method of claim 8, wherein the first plurality of microphones are located in a camera, and wherein the second plurality of microphones are located in a speakerphone.

10. The method of claim 8, further comprising:

implementing an echo canceller based on the approximate location of the participant.

11. The method of claim 10, wherein the echo canceller is an independent echo canceller picked from a plurality of independent echo cancellers and wherein the specific independent echo canceller used is based on the approximate location of the participant.

12. The method of claim 8, wherein a remote site uses the left and right audio channels to place the transmitted audio signal on speakers on a respective side of the remote site indicated by the left and right audio channels.

13. A system, comprising:

a first plurality of microphones;

a second plurality of microphones for producing an audio signal of a participant; and

a computing system coupled to the first and second plurality of microphones, wherein the computing system is configured to;

beamform a response of the first plurality of microphones to determine directional information for the audio signal relative to the first plurality of microphones; and

transmit the audio signal from the second plurality of microphones and directional information, wherein the computing system is configured to transmit the audio and directional information by transmitting a left audio channel and a right audio channel, wherein the left audio channel comprises audio from at least one beam on a left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on a right side of the second plurality of microphones, and wherein the computing system is configured to determine the left side and the right side of the second plurality of microphones by cross correlating the beamformed response from the first plurality of microphones with a beamformed response of the second plurality of microphones.

14. A method, comprising:

beamforming the audio using a first plurality of microphones to determine directional information for a conference participant relative to a conference system;

capturing an audio signal of the conference participant using a second plurality of microphones;

transmitting the audio signal and directional information to a remote conference system, wherein said transmitting the audio signal and directional information comprises transmitting a left audio channel and a right audio channel, wherein the left audio channel comprises audio from at least one beam on a left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on a right side of the second plurality of microphones, and wherein the left side and the right side of the second plurality of microphones are determined by cross correlating the beamformed response from the first plurality of microphones with a beamformed response of the second plurality of microphones.