Exploiting visual information for enhancing audio signals via source separation and beamforming

• US 9,904,851 B2
• Filed: 06/11/2014
• Issued: 02/27/2018
• Est. Priority Date: 06/11/2014
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Dimitriadis, Dimitrios, Bowen, Donald J., Ji, Lusheng, Schroeter, Horst J.
• Primary Examiner(s): Tsang, Fan
• Assistant Examiner(s): Siegel, David
• Application Number:
  US14/302,110
• Publication Number:
  US 20150365759A1
• Time in Patent Office: 1,357 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S

Abstract

A system for exploiting visual information for enhancing audio signals via source separation and beamforming is disclosed. The system may obtain visual content associated with an environment of a user, and may extract, from the visual content, metadata associated with the environment. The system may determine a location of the user based on the extracted metadata. Additionally, the system may load, based on the location, an audio profile corresponding to the location of the user. The system may also load a user profile of the user that includes audio data associated with the user. Furthermore, the system may cancel, based on the audio profile and user profile, noise from the environment of the user. Moreover, the system may include adjusting, based on the audio profile and user profile, an audio signal generated by the user so as to enhance the audio signal during a communications session of the user.

Claims

1. A system, comprising:

a memory that stores instructions;

a processor that executes the instructions to perform operations, the operations comprising;

obtaining visual content associated with an environment of a user;

extracting, from the visual content, first metadata associated the environment of the user, wherein the first metadata indicates location information, information that identifies an interferer generating noise in the environment, information that identifies a location of the interferer, and an action being performed by the user;

determining a location of the user based on the first metadata;

loading, based on the location, an audio profile corresponding to the location of the user;

loading a user profile of the user, wherein the user profile comprises audio data associated with the user;

cancelling, based on the audio profile, the first metadata, and the user profile, the noise from the environment of the user; and

adjusting, based on the audio profile and the user profile, an audio signal generated by the user so as to enhance the audio signal during a communications session of the user.

2. The system of claim 1, wherein the operations further comprise obtaining audio content associated with the environment of the user.

3. The system of claim 2, wherein the operations further comprise extracting, from the audio content, second metadata associated with the environment of the user.

4. The system of claim 3, wherein the operations further comprise determining the location of the user based on the second metadata.

5. The system of claim 1, wherein the operations further comprise transmitting a null in a direction of a source generating the noise from the environment of the user.

6. The system of claim 1, wherein the communications session comprises a communications session selected from a group consisting of a telephone conversation, a speech recognition session, a video conference session, and any combination thereof.

7. The system of claim 1, wherein the operations further comprise separating, by utilizing the audio profile, each source of a plurality of sources contributing to the noise, and wherein the operations further comprise separating each source of the plurality of sources from the user.

8. The system of claim 1, wherein the operations further comprise obtaining the visual content from the environment using a device selected from a group consisting of a smartphone, a camera, a computer, a laptop, a visual sensor, and any combination thereof.

9. The system of claim 1, wherein the operations further comprise modifying the user profile of the user based on the visual content obtained.

10. A method, comprising:

capturing visual content associated with an environment of a user;

extracting, from the visual content, first metadata associated the environment of the user, wherein the first metadata indicates location information, information that identifies an interferer generating noise in the environment, information that identifies a location of the interferer, and an action being performed by the user;

determining, by utilizing instructions from memory that are executed by a processor, a location of the user based on the first metadata;

loading, based on the location, an audio profile corresponding to the location of the user;

loading a user profile of the user, wherein the user profile comprises audio data associated with the user;

cancelling, based on the audio profile, the first metadata, and the user profile, noise from the environment of the user; and

adjusting, based on the audio profile and the user profile, an audio signal generated by the user so as to enhance the audio signal during a communications session of the user.

11. The method of claim 10, further comprising obtaining audio content from the environment of the user.

12. The method of claim 11, further comprising extracting, from the audio content, second metadata associated with the environment of the user.

13. The method of claim 12, further comprising determining the location of the user based on the second metadata.

14. The method of claim 10, further comprising transmitting a null in a direction of a source generating the noise from the environment of the user.

15. The method of claim 10, wherein the communications session comprises a communications session selected from a group consisting of a telephone conversation, a speech recognition session, a video conference session, and any combination thereof.

16. The method of claim 10, further comprising separating, by utilizing the audio profile, each source of a plurality of sources contributing to the noise from the environment.

17. The method of claim 10, further comprising obtaining the visual content from the environment using a device selected from a group consisting of a smartphone, a camera, a computer, a laptop, a visual sensor, and any combination thereof.

18. The method of claim 10, further comprising modifying the user profile of the user based on the visual content obtained.

19. A non-transitory computer-readable device comprising instructions, which when executed by a processor, cause the processor to perform operations comprising:

obtaining visual content associated with an environment of a user;

extracting, from the visual content, first metadata associated the environment of the user, wherein the first metadata indicates location information, information that identifies an interferer generating noise in the environment, information that identifies a location of the interferer, and an action being performed by the user;

determining a location of the user based on the first metadata;

loading, based on the location, an audio profile corresponding to the location of the user;

loading a user profile of the user, wherein the user profile comprises audio data associated with the user;

cancelling, based on the audio profile, the first metadata, and the user profile, noise from the environment of the user; and

adjusting, based on the audio profile and the user profile, an audio signal generated by the user so as to enhance the audio signal during a communications session of the user.

20. The non-transitory computer-readable device of claim 19, wherein the operations further comprise separating, by utilizing the audio profile, each source of a plurality of sources contributing to the noise from the environment.