Microphone array steering with image-based source location

• US 8,761,412 B2
• Filed: 12/16/2010
• Issued: 06/24/2014
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Hernandez-Abrego, Gustavo, Larsen, Eric
• Primary Examiner(s): Paul, Disler
• Application Number:
  US12/969,896
• Publication Number:
  US 20120155703A1
• Time in Patent Office: 1,286 Days
• Field of Search: 381/92, 381/122, 381 56- 58, 382/103, 348/231.4, 348169-170
• US Class Current: 381/92
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6072 of an input signal, e.g. pitch and rhythm extraction, voice recognition
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: G10L 2021/02087 the noise being separate speech, e.g. cocktail party
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/0216 characterised by the method used for estimating noise
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

Methods and systems for beam forming an audio signal based on a location of an object relative to the listening device, the location being determined from positional data deduced from an optical image including the object. In an embodiment, an object'"'"'s position is tracked based on video images of the object and the audio signal received from a microphone array located at a fixed position is filtered based on the tracked object position. Beam forming techniques may be applied to emphasize portions of an audio signal associated with sources near the object.

Claims

1. A method of focusing a microphone array, comprising:

receiving an audio signal from the microphone array, the microphone array disposed a distance away from a first component of an audio beam forming system;

receiving a visual image from a plurality of images forming a video segment from an optical image sensor having a known position relative to the microphone array, the visual image including the first component within a field of view (FOV);

determining positional information for the first component, relative to the microphone array by analyzing the visual image, and by determining a distance between the first component and the image sensor based on a predetermined size of the first component;

tracking a change in the position of the first component relative to the image sensor based on changes in the visual image within the video segment; and

filtering the audio signal to emphasize or de-emphasize audio sources proximate to the first component based on a beam forming algorithm that is a function of the positional information and on the change in the position of the first component.

2. The method of claim 1, wherein filtering the audio signal further comprises steering the audio signal from a first position to the position of the first component.

3. The method of claim 1, wherein filtering the audio signal further comprises removing portions of the audio signal originating from outside of a listening zone centered about the first component.

4. The method of claim 1, wherein analyzing the visual image further comprises:

identifying the first component with a pattern recognition algorithm; and

calculating a position of the first component relative to the optical image sensor.

5. The method of claim 4, wherein calculating a position of the first component relative to the optical image sensor further comprises determining a three dimensional vector that points from the optical axis to a center of the first component.

6. The method of claim 4, further comprising receiving a change in orientation of the first component, and wherein filtering the audio signal based on the determined position of the first component further comprises determining whether to offset an audio steering angle from the determined position of the first component based on the orientation change.

7. The method of claim 1, wherein filtering the audio signal based on the determined position of the first component further comprises:

transforming the received audio signal into the frequency domain;

separating the audio signal into discrete frequency bands; and

applying at least one component of the positional information to filter each of the frequency bands.

8. The method of claim 1, further comprising:

analyzing the visual image to determine the position of a second component of the audio beam forming system relative to the array; and

steering the audio signal from the first component toward the second component based on the determined position of the second component.

9. The method of claim 8, wherein the steering toward the second component further comprises removing portions of the audio signal originating from near the first component based on the determined position of the first component.

10. A non-transistory computer-readable medium having stored thereon a set of instructions, which when executed by a processing system, cause the processing system to perform the method of claim 1.

11. A system, comprising:

a computing platform;

a microphone array coupled to the computing platform to receive an audio signal; and

an optical image sensor coupled to the computing platform, wherein the computing platform further comprises;

an object tracking module to determine positional information for a first component and for a second component of an audio beam steering system relative to the microphone array by analyzing an image signal from the optical image sensor, the image signal including the first component and the second component within a field of view (FOV) of the image sensor; and

an audio signal processor to emphasize or de-emphasize audio sources proximate to the first component based on a beam forming algorithm that is a function of the positional information for the first component, and to steer the received audio signal from the first component to the second component based on the positional information for the second component.

12. The system as in claim 11, wherein the microphone array is integrated with the optical image sensor into a video camera disposed a distance away from the first component.

13. The system as in claim 12, wherein the object tracking module is to track a change in the positional information for the first component based on changes in the image signal within a video segment;

and wherein the audio signal processor is to steer the received audio signal with filter coefficients generated based on the change in the positional information.

14. The system as in claim 11, wherein the audio signal processor is to:

transform the received audio signal into frequency domain;

separate the audio signal into discrete frequency bands; and

apply at least one component of the positional information to filter each of the frequency bands.

15. The system as in claim 11, wherein the system further comprises the first component, and wherein the first component is a first controller of the computing platform, includes one or more inertial sensors, and is to gather and transmit inertial data to the computing platform.

16. The system as in claim 15, wherein the audio signal processor is to filter the received audio signal with a steering angle offset from a determined position of the first component based on the inertial data.

17. The system as in claim 11, wherein the audio signal processor is to remove portions of the received audio signal originating from near the first component based on the positional information for the first component.