Video and audio conferencing system with spatial audio

• US 20060104458A1
• Filed: 10/17/2005
• Published: 05/18/2006
• Est. Priority Date: 10/15/2004
• Status: Active Grant

• Current Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Original Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Inventors: Kenoyer, Michael L., Oxford, William V.
• Granted Patent: US 7,667,728 B2
• Time in Patent Office: Days
• US Class Current: 381/92
• CPC Class Codes: H04N 7/14 Systems for two-way working H04N7/12, H04N7/173 take precedence
• CPC Class Codes: H04R 2201/401 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation

Abstract

In some embodiments, spatially realistic audio may be provided for a conference call. Voices from participants on the left side of a display, in a conference call, may be directed through audio on the left side of the display at the other conferencing system in the conference call (similarly for voices from the center and right side of the display). In some embodiments, two speakers may be used in the system to create synthesized stereo sound at a location specified by directional information received as side information along with the existing audio channel. The location may be determined by using beamforming with integrated microphones on a camera or speakerphone. In some embodiments, the audio signal and directional information may be sent in the form of a left audio channel and a right audio channel.

Claims

1. A system, comprising:

a first plurality of microphones;

a second plurality of microphones for producing an audio signal of a participant; and

a computing system coupled to the first and second plurality of microphones, wherein the computing system is configured to;

beamform a response of the first plurality of microphones to determine directional information for the audio signal relative to the first plurality of microphones; and

transmit the audio signal from the second plurality of microphones and directional information.

2. The system of claim 1, wherein a response from the second plurality of microphones is beamformed, and the beamformed response from the first plurality of microphones is cross-correlated with the beams from the second plurality of microphones.

3. The system of claim 2, wherein the first plurality of microphones are located in a camera, and wherein the second plurality of microphones are located in a speakerphone.

4. The system of claim 1, wherein the computing system is further configured to implement an echo canceller based on the determined active participant'"'"'s location.

5. The system of claim 1, further comprising a plurality of independent echo cancellers, each configured based on a potential participant location.

6. The system of claim 1, further comprising:

a multiple channel echo canceller;

wherein the multiple channel echo canceller is configured for multiple channels.

7. The system of claim 1, further comprising a remote conference system with speakers in at least two locations relative to the remote conference system, wherein the audio from the participant is reproduced at speakers in at least one of the two locations relative to the remote conference system based on the location of the participant relative to the system that transmitted the audio and the location information.

8. The system of claim 1, wherein the system is configured to transmit the side directional information on the location of the audio with respect to the first plurality of microphones, and wherein a remote site uses the side directional information to place the transmitted audio signal on speakers on a respective side of the remote site indicated by the side directional information.

9. The system of claim 1,

wherein the computing system is configured to transmit the audio and directional information by transmitting a left audio channel and a right audio channel;

wherein the left audio channel comprises audio from at least one beam on a left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on a right side of the second plurality of microphones; and

wherein the computing system is configured to determine the left side and the right side of the second plurality of microphones by cross correlating the beamformed response from the first plurality of microphones with a beamformed response of the second plurality of microphones.

10. The system of claim 1, wherein the first plurality of microphones and the second plurality of microphones are the same.

11. A method, comprising:

beamforming the audio using a first plurality of microphones to determine directional information for a conference participant relative to a conference system;

capturing an audio signal of the conference participant using a second plurality of microphones;

transmitting the audio signal and directional information to a remote conference system;

wherein the directional information and the audio signal are used to reproduce audio from the participant at the remote conference system.

12. The method of claim 11, further comprising:

beamforming a response from the second plurality of microphones; and

cross correlating beams from the first plurality of microphones with beams from the second plurality of microphones.

13. The method of claim 12, wherein the first plurality of microphones are located in a camera, and wherein the second plurality of microphones are located in a speakerphone.

14. The method of claim 11, further comprising:

implementing an echo canceller based on the approximate location of the participant.

15. The method of claim 14, wherein the echo canceller is an independent echo canceller picked from a plurality of independent echo cancellers and wherein the specific independent echo canceller used is based on the approximate location of the participant.

16. The method of claim 11, wherein transmitting the directional information comprises transmitting side directional information on the location of the audio with respect to the first plurality of microphones, and wherein a remote site uses the side directional information to place the transmitted audio signal on speakers on a respective side of the remote site indicated by the side information.

17. The method of claim 11,

wherein transmitting the audio signal and directional information comprises transmitting a left audio channel and a right audio channel, wherein the left audio channel comprises audio from at least one beam on a left side of the second plurality of microphones and the right audio channel comprises audio from at least one beam on a right side of the second plurality of microphones;

and

wherein the left side and the right side of the second plurality of microphones are determined by cross correlating the beamformed response from the first plurality of microphones with a beamformed response of the second plurality of microphones.

18. The system of claim 11, wherein the first plurality of microphones and the second plurality of microphones are the same.

19. An integrated video conferencing system, comprising:

a plurality of microphones; and

a computing system coupled to the plurality of microphones, wherein the computing system is configured to;

beamform a response of the plurality of microphones to determine an directional information for a participant relative to the plurality of microphones; and

transmit audio from the plurality of microphones and directional information.

20. The integrated video conferencing system of claim 19, wherein the video conferencing system includes the computing system, the plurality of microphones, a camera, and a display integrated in a single housing.