Video-assisted audio signal processing system and method

• US 6,483,532 B1
• Filed: 07/13/1998
• Issued: 11/19/2002
• Est. Priority Date: 07/13/1998
• Status: Expired

• Current Assignee: 8X8, Inc., VLSI Technology, Inc. (Koninklijke Philips N.V.)
• Original Assignee: Netergy Microelectronics, Inc. (8X8, Inc.)
• Inventors: Girod, Bernd
• Primary Examiner(s): Nguyen, Duc
• Assistant Examiner(s): LAO, LUNSEE
• Application Number:
  US09/114,668
• Time in Patent Office: 1,590 Days
• Field of Search: 348/14.12, 379/406-1-4, 379/100.17, 381/71.1, 381/71.11, 381/71.12, 381/71.13, 381/71.8
• US Class Current: 348/14.12
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

A circuit arrangement for controlling audio signal transmissions for a communications system that includes a microphone and a video camera. The arrangement comprises a video processor configured and arranged to receive a video signal from the video camera, detect movement of an object in the video signal, and provide a motion-indicating signal indicating movement relative to the object. An audio processor is coupled to the video processor and is configured and arranged to modify the audio signal to be transmitted responsive to the motion-indicating signal. In another embodiment, a video signal processor is configured and arranged to receive a video signal from the video camera, detect mouth movement of a person and provide a mouth-movement signal indicative of movement of the person'"'"'s mouth. An echo-cancellation circuit is coupled to the video signal processor and configured and arranged to filter from an audio signal provided by the microphone sound energy output by the speaker responsive to the mouth-movement signal.

Claims

1. A circuit arrangement for controlling audio signal transmissions for a communications system that includes a microphone and a video camera, comprising:

a video processor configured and arranged to receive a video signal from the video camera, detect movement of an object in the video signal, provide a motion-indicating signal indicating movement relative to the object; and

an audio processor coupled to the video processor and configured and arranged to modify and mute the audio signal to be transmitted responsive to the motion-indicating signal.

2. The circuit arrangement of claim 1, wherein the object is a person.

3. The circuit arrangement of claim 1, wherein the object is a person'"'"'s face.

4. The circuit arrangement of claim 1, wherein the object is a person'"'"'s mouth.

5. An echo-cancellation arrangement for a video communication system that includes a microphone, a speaker, and a video camera for use by a video conference participant at a first location, comprising:

a video signal processor configured and arranged to receive a video signal from the video camera, detect mouth movement of the participant and provide a mouth-movement signal indicative of movement of the participant'"'"'s mouth;

an echo-cancellation circuit coupled to the video signal processor and configured and arranged to filter from an audio signal provided by the microphone sound energy output by the speaker responsive to the mouth-movement signal.

6. The arrangement of claim 5, wherein the video signal processor includes:

a background detector configured and arranged to distinguish a foreground portion of an image from a background portion of the image;

a face detector coupled to the background detector and configured and arranged to detect an image of the participant'"'"'s face in the foreground portion and detect movement of the participant'"'"'s face; and

a mouth-movement detector coupled to the face detector and configured and arranged to detect mouth movement in the image of the face and provide the mouth-movement signal.

7. The arrangement of claim 5, wherein the echo-cancellation circuit includes:

a double-talk detector configured and arranged to detect and generate a double-talk signal in response to a received audio signal and a transmit audio signal;

a coefficient adapter coupled to the double-talk detector and to the video signal processor and configured and arranged to generate filter coefficients responsive to the double-talk and mouth-movement signals; and

a filter coupled to the adaptive processor.

8. A video communication arrangement with video-assisted echo-cancellation, the arrangement for use by a video conference participant at a first location, comprising:

a microphone;

a speaker;

a video camera arranged to provide a video signal;

a video signal processor coupled to the video camera and configured and arranged to detect mouth movement of the participant in the video signal and provide a mouth-movement signal indicative of the participant speaking;

an echo-cancellation circuit coupled to the microphone, speaker, and video signal processor and configured and arranged to filter, responsive to the mouth-movement signal, from an audio signal provided by the microphone sound energy output by the speaker;

a video display device;

a channel interface;

a multiplexer coupled to the channel interface, the echo-cancellation circuit, and the video signal processor, and configured and arranged to provide audio and video signals as output to the channel interface; and

a demultiplexer coupled to the channel interface, the echo-cancellation circuit, the video display device, and the speaker, and configured and arranged to provide audio and video signals.

9. The arrangement of claim 8, wherein the video signal processor includes:

a background detector configured and arranged to distinguish a foreground portion of an image from a background portion of the image;

a face detector coupled to the background detector and configured and arranged to detect an image of the participant'"'"'s face in the foreground portion and detect movement of the participant'"'"'s face; and

a mouth-movement detector coupled to the face detector and configured and arranged to detect mouth movement in the image of the face and provide the mouth-movement signal.

10. The arrangement of claim 9, wherein the echo-cancellation circuit includes:

a double-talk detector configured and arranged to detect and generate a double-talk signal in response to a received audio signal and a transmit audio signal;

a coefficient adapter coupled to the double-talk detector and to the video signal processor and configured and arranged to generate filter coefficients responsive to the double-talk and mouth-movement signals; and

a filter coupled to the adaptive processor.

11. The arrangement of claim 8, wherein the echo-cancellation circuit includes:

a double-talk detector configured and arranged to detect and generate a double-talk signal in response to a received audio signal and a transmit audio signal;

a coefficient adapter coupled to the double-talk detector and to the video signal processor and configured and arranged to generate filter coefficients responsive to the double-talk and mouth-movement signals; and

a filter coupled to the adaptive processor.

12. A method for audio signal and video signal processing, comprising:

receiving a video signal from a video camera;

receiving an audio signal from a microphone;

detecting movement of an object in the video signal;

providing a motion-indicating signal to an audio signal processor when movement of the object is detected;

modifying the audio signal in response to the motion-indicating signal; and

providing a muted audio signal when no motion is detected.

13. The method of claim 12, wherein the object is a person.

14. The method of claim 12, wherein the object is a person'"'"'s face.

15. The method of claim 12, wherein the object is a person'"'"'s mouth.

16. A method for audio signal and video signal processing, comprising:

receiving a video signal from a video camera;

receiving an audio signal from a microphone;

detecting movement of a person'"'"'s mouth in the video signal;

providing a motion-indicating signal to an echo-cancellation circuit when movement is detected; and

modifying filter coefficients in response to the motion-indicating signal.

17. The method of claim 16, further comprising:

detecting a foreground portion of an image in the video signal;

detecting a face in the foreground portion of the image; and

detecting a mouth on the face.

18. An apparatus for audio signal and video signal processing, comprising:

means for receiving a video signal from a video camera;

means for receiving an audio signal from a microphone;

means for detecting movement of a person'"'"'s mouth in the video signal;

means for providing a motion-indicating signal to an echo-cancellation circuit when movement is detected; and

means for modifying filter coefficients in response to the motion-indicating signal.