Acoustic echo cancellation using visual cues

• US 10,242,695 B1
• Filed: 09/14/2017
• Issued: 03/26/2019
• Est. Priority Date: 06/27/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Velusamy, Kavitha, Chu, Wai C., Gopalan, Ramya, Chhetri, Amit S.
• Primary Examiner(s): Beasley, Deirdre L
• Application Number:
  US15/704,285
• Time in Patent Office: 558 Days
• Field of Search: 348143
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2021/02082 the noise being echo, reverberation of the speech
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04B 3/23 using a replica of transmitted signal in the time domain, e.g. echo cancellers
• CPC Class Codes: H04B 3/231 Echo cancellers using readout of a memory to provide the echo replica
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

Techniques for enhancing an acoustic echo canceller based on visual cues are described herein. The techniques include changing adaptation of a filter of the acoustic echo canceller, calibrating the filter, or reducing background noise from an audio signal processed by the acoustic echo canceller. The changing, calibrating, and reducing are responsive to visual cues that describe acoustic characteristics of a location of a device that includes the acoustic echo canceller. Such visual cues may indicate that no human being is present at the location, that some subject(s) are engaged in speaking or sound generating activities, or that motion associated with an echo path change has occurred at the location.

Claims

1. A device, comprising:

one or more processors;

a camera;

a loudspeaker;

a microphone; and

an acoustic echo canceller unit, wherein the device is configured to;

capture, using the camera, an image of an environment in which the device is located,

determine, using the one or more processors and based at least in part on the image, an absence of a human within the environment,

output a first audio signal, via the loudspeaker, based at least in part on determining the absence of the human,

determine one or more echo paths associated with the environment based at least in part on a second audio signal detected by the microphone, and

calibrate a filter of the acoustic echo canceller unit based at least in part on the one or more echo paths.

2. The device of claim 1, wherein the environment comprises an enclosed acoustic space, and the one or more echo paths extend from the microphone to the loudspeaker.

3. The device of claim 1, wherein the second audio signal is indicative of the absence of the human, and is based at least partly on the first audio signal.

4. The device of claim 1, wherein the device is further configured to:

determine a change in position of an object located within the environment,

determine an echo path change based at least in part on the change in position of the object, and

calibrate the filter of the acoustic echo canceller unit based at least in part on the echo path change.

5. The device of claim 4, wherein the device is further configured to:

determine a magnitude of the change in position,

determine that the magnitude is above a predetermined minimum magnitude, and

calibrate the filter of the acoustic echo canceller unit based at least in part on determining that the magnitude is above the predetermined minimum magnitude.

6. The device of claim 4, wherein the device is further configured to calibrate the filter based at least in part on determining that

the object has a size exceeding a minimum size threshold, and

the object is within a specified distance from the device.

7. The device of claim 1, wherein the first audio signal comprises an initial audio signal output via the loudspeaker.

8. The device of claim 1, wherein the device is further configured to:

capture, using the camera, an additional image of the environment,

detect, using the microphone, that a person within the environment is speaking, and

generate, using the additional image, a confidence score indicating an accuracy of detecting that the person within the environment is speaking.

9. The device of claim 8, wherein the device is further configured to change an adaptation of the filter based at least in part on the confidence score.

10. The device of claim 1, wherein the device is further configured to

determine background noise characteristics using the second audio signal,

detect, using the microphone, a third audio signal associated with a person within the environment speaking, and

reduce background noise in the third audio signal based at least in part on the background noise characteristics.

11. A method, comprising:

capturing, with a camera, an image of an environment,

determining, with an electronic device and using the image, an absence of a human within the environment,

outputting a first audio signal within the environment, using a loudspeaker, based at least in part on determining the absence of the human,

detecting a second audio signal within the environment using a microphone,

determining, with the electronic device, one or more echo paths associated with the environment based at least in part on the second audio signal, and

changing an adaptation of an acoustic filter of the electronic device based at least in part on the one or more echo paths.

12. The method of claim 11, further comprising:

detecting a third audio signal within the environment using the microphone,

determining, using a voice activity detector of the electronic device, that the third audio signal includes a human voice audio signal, and

reducing background noise in the third audio signal based at least in part on determining that the third audio signal includes the human voice audio signal.

13. The method of claim 12, further comprising:

determining background noise characteristics using the second audio signal, and

reducing background noise in the third audio signal based at least in part on the background noise characteristics.

14. The method of claim 11, further comprising

capturing, using the camera, an additional image of the environment,

detecting, using the microphone, that a person within the environment is speaking, and

generating, using the additional image, a confidence score indicating an accuracy of detecting that the person within the environment is speaking.

15. The method of claim 14, further comprising:

capturing a further image of the environment,

modifying the confidence score based at least in part on the further image, and

changing the adaptation of the acoustic filter based at least in part on the confidence score.

16. A method, comprising:

determining, with an electronic device, an absence of a human within an environment;

outputting a first audio signal, within the environment, based at least in part on determining the absence of the human;

capturing a second audio signal, within the environment, during the absence, the second audio signal being indicative of the absence of the human;

determining, with the electronic device, an echo path associated with the environment based at least in part on the second audio signal;

changing an adaptation of an acoustic filter of the electronic device based at least in part on the echo path;

capturing a third audio signal from the environment;

determining, with the electronic device, a confidence score indicating a likelihood that the third audio signal is associated with a person speaking within the environment;

determining that the confidence score exceeds a threshold; and

removing, at least in part, background noise from the third audio signal with the acoustic filter based at least in part on the confidence score exceeding the threshold, wherein an amount of the background noise removed from the third audio signal is based at least in part on the echo path.

17. The method of claim 16, further comprising:

capturing an image of the environment using a camera of the electronic device;

modifying the confidence score based at least in part on the image; and

changing the adaptation of the acoustic filter of the electronic device based at least in part on the confidence score.

18. The method of claim 16, further comprising:

determining that an item at the location has changed position;

determining that the change in position of the item is associated with a corresponding change in the echo path; and

accelerating the adaptation of the acoustic filter of the electronic device based at least in part on determining that the change in position of the item is associated with the corresponding change in the echo path.

19. The method of claim 16, further comprising removing, at least in part, an acoustic echo from the third audio signal, wherein an amount of the acoustic echo removed from the third audio signal is based at least in part on the echo path.

20. The method of claim 16, further comprising:

determining that one or more images of the environment show movement of lips of the person in a specified time period; and

modifying the confidence score based at least in part on determining that the one or more images of the environment show movement of the lips of the person in the specified time period.