Method and apparatus for performing microphone beamforming

• US 9,330,673 B2
• Filed: 09/13/2011
• Issued: 05/03/2016
• Est. Priority Date: 09/13/2010
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Cho, Sung-Jae, Kim, Hyun-Soo
• Primary Examiner(s): He, Jialong
• Application Number:
  US13/231,705
• Publication Number:
  US 20120065973A1
• Time in Patent Office: 1,694 Days
• US Class Current: 1/1
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 29/005 Microphone arrays

Abstract

A method and apparatus for performing microphone beamforming. The method includes recognizing a speech of a speaker, searching for a previously stored image associated with the speaker, searching for the speaker through a camera based on the image, recognizing a position of the speaker, and performing microphone beamforming according to the position of the speaker.

Claims

1. A method of performing microphone beamforming, the method comprising:

distinguishing at least a portion of received sound as speech by a speaker from noise;

identifying the speaker by comparing the received speech with previously stored speech in a data storage to obtain an identification of the speaker;

based on the identification of the speaker, searching images stored in the data storage for a stored image associated with the identified speaker;

searching for a position of the identified speaker through a camera by comparing the stored image of the identified speaker with image data received through the camera;

recognizing the position of the identified speaker; and

performing microphone beamforming according to the position of the identified speaker.

2. The method of claim 1, further comprising:

receiving a speech signal corresponding to the speech by the speaker;

extracting a section including a predetermined speech signal from the received speech signal; and

identifying the speaker based on the section.

3. The method of claim 2, wherein identifying the speaker comprises identifying the speaker by comparing the predetermined speech signal with speech signals that are previously acquired from one or more speakers and stored.

4. The method of claim 1, wherein searching for a position of the identified speaker comprises extracting an object corresponding to the identified speaker from the image data received through the camera by comparing the image data received through the camera with previously stored image data associated with one or more speakers.

5. The method of claim 4, wherein recognizing the position of the identified speaker comprises determining whether the position of the identified speaker is changed to a changed position based on information about the object, and wherein performing microphone beamforming comprises adaptively performing the microphone beamforming according to the changed position if the position of the identified speaker is changed to the changed position.

6. The method of claim 1, wherein performing microphone beamforming comprises amplifying a signal corresponding to sound generated from the position of the identified speaker and attenuating signals corresponding to sound generated from other positions.

7. An apparatus for performing microphone beamforming, the apparatus comprising:

data storage configured to store images associated with speakers;

a microphone array configured to receive sound;

a camera configured to acquire image data;

a processor configured to

distinguish at least a portion of the received sound as speech by a speaker from noise,

identify the speaker by comparing the received speech with previously stored speech in the data storage to obtain an identification of the speaker,

based on the identification of the speaker, search the images stored in the data storage for a stored image associated with the identified speaker,

search for a position of the identified speaker through the camera by comparing the stored image of the identified speaker with image data received through the camera, and

recognize the position of the identified speaker; and

a beamforming controller configured to perform microphone beamforming according to the position of the identified speaker.

8. The apparatus of claim 7, wherein the processor is configured to:

receive a speech signal corresponding to the speech by the speaker,

extract a section including a predetermined speech signal from the speech signal, and

identify the speaker based on the section.

9. The apparatus of claim 8, wherein the processor is configured to identify the speaker by comparing the predetermined speech signal with speech signals that are previously acquired from one or more speakers and stored.

10. The apparatus of claim 7, wherein the processor is configured to extract an object corresponding to the identified speaker from the image data received through the camera by comparing the image data received through the camera with previously stored image data associated with one or more speakers.

11. The apparatus of claim 10, wherein the processor is configured to determine whether the position of the identified speaker is changed to a changed position based on the image acquired through the camera, and wherein the beamforming controller is configured to adaptively perform microphone beamforming according to the changed position.

12. The apparatus of claim 7, wherein the beamforming controller is configured to amplify a signal corresponding to sound generated from the position of the identified speaker and to attenuate signals corresponding to sound generated from other positions.

13. The apparatus of claim 7, wherein the processor is configured to search for the stored image of the identified speaker in a database in the apparatus.

14. The apparatus of claim 7, wherein the processor is configured to search for the stored image of the identified speaker using a search engine on a network.

15. A non-transitory computer-readable recording medium encoded with computer-executable instructions that when executed cause a data processing system to:

distinguish at least a portion of received sound as speech by a speaker from noise;

identify the speaker by comparing the received speech with previously stored speech in a data storage to obtain an identification of the speaker;

based on the identification of the speaker, search images stored in a data storage for a stored image associated with the identified speaker;

search for a position of the identified speaker through a camera by comparing the stored image of the identified speaker with image data received through the camera;

recognize the position of the identified speaker; and

perform microphone beamforming according to the position of the identified speaker.

16. The computer-readable recording medium of claim 15, wherein the computer-executable instructions that when executed cause the data processing system to:

receive a speech signal corresponding to the speech by the speaker;

extract a section including a predetermined speech signal from the received speech signal; and

identify the speaker based on the section.

17. The computer-readable recording medium of claim 16, wherein the computer-executable instructions that when executed cause the data processing system to identify the speaker comprise computer-executable instructions that when executed cause the data processing system to compare the predetermined speech signal with speech signals that are previously acquired from one or more speakers and stored.

18. The computer-readable recording medium of claim 15, wherein the computer-executable instructions that when executed cause the data processing system to search for the position of the identified speaker comprise computer-executable instructions that when executed cause the data processing system to extract an object corresponding to the identified speaker from the image data received through the camera by comparing the image data received through the camera with stored image data associated with one or more speakers.

19. The computer-readable recording medium of claim 18, wherein the computer-executable instructions that when executed cause the data processing system to recognize the position comprise computer-executable instructions that when executed cause the data processing system to determine whether the position of the identified speaker is changed to a changed position based on information about the object, and wherein the computer-executable instructions that when executed cause the data processing system to perform microphone beamforming comprise computer-executable instructions that when executed cause the data processing system to adaptively perform the microphone beamforming according to the changed position if the position of the identified speaker is changed to the changed position.

20. The computer-readable recording medium of claim 15, wherein the computer-executable instructions that when executed cause the data processing system to perform microphone beamforming comprise computer-executable instructions that when executed cause the data processing system to amplify a signal corresponding to sound generated from the position of the identified speaker and to attenuate signals corresponding to sound generated from other positions.