Electronic device directional audio capture

• US 9,445,193 B2
• Filed: 07/31/2008
• Issued: 09/13/2016
• Est. Priority Date: 07/31/2008
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Tico, Marius, Hamalainen, Matti S., Vehvilainen, Markku K.
• Primary Examiner(s): HUYNH, NAM TRUNG
• Application Number:
  US12/221,395
• Publication Number:
  US 20100026780A1
• Time in Patent Office: 2,966 Days
• Field of Search: 348/14.02, 348/14.16, 348 1408- 141
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G10L 21/0208 Noise filtering
• CPC Class Codes: H04M 1/6041 Portable telephones adapted for handsfree use
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

Disclosed herein is an apparatus. The apparatus includes a housing, electronic circuitry, and an audio-visual source tracking system. The electronic circuitry is in the housing. The audio-visual source tracking system includes a first video camera and an array of microphones. The first video camera and the array of microphones are attached to the housing. The audio-visual source tracking system is configured to receive video information from the first video camera. The audio-visual source tracking system is configured to capture audio information from the array of microphones at least partially in response to the video information. The audio-visual source tracking system might include a second video camera that is attached to the housing, wherein the first and second video cameras together estimate the beam orientation of the array of microphones.

Claims

1. An apparatus comprising:

a housing; and

an audio-visual source tracking system comprising a first video camera and an array of microphones, wherein the first video camera and the array of microphones are attached to the housing, wherein the array of microphones are proximate the first video camera, wherein the array of microphones is configured to have an adjustable beam, wherein the first video camera and the array of microphones are proximate an exterior portion of the housing of the apparatus, wherein the audio-visual source tracking system is configured to receive video information from the first video camera, wherein the audio-visual source tracking system is configured to capture audio information from the array of microphones at least partially in response to the video information, wherein the audio-visual source tracking system is configured to adjust and direct a sensitivity of the adjustable beam of the array of microphones while a user is speaking at least partially in response to the video information which tracks a head position of the user relative to the audio-visual source tracking system and is configured to update a location of the user, wherein the audio-visual source tracking system is further configured to adjust and direct the sensitivity of the adjustable beam of the array of microphones based at least partially upon a detected face of the user at a time after an initial face detection of the user, wherein a direction of the array of microphones is determined based, at least partially, on a ratio of the camera focal length and a reference point of the user, and wherein the apparatus is a portable electronic device.

2. The apparatus of claim 1 wherein the array of microphones comprises least three microphones.

3. The apparatus of claim 1 wherein the audio-visual source tracking system is configured to receive video information corresponding to a user of the apparatus from the first video camera.

4. The apparatus of claim 1 wherein the apparatus comprises a mobile handset.

5. The apparatus of claim 1 wherein the audio-visual source tracking system is configured to capture a voice of a user of the apparatus at least partially in response to an orientation of the user.

6. The apparatus of claim 1 wherein the first video camera is configured to be oriented towards a user of the apparatus, wherein fixed positions of the first camera and the array of microphones within the portable electronic device allows for a known orientation of the first video camera relative to the orientation of the array of microphones.

7. The apparatus of claim 1 wherein the array of microphones surrounds the first video camera.

8. The apparatus of claim 1 further comprising a second video camera, wherein the first and the second video cameras are configured to estimate a beam orientation for the array of microphones.

9. The apparatus of claim 1 further comprising a second video camera, wherein the first and the second video cameras are configured to estimate a beam orientation for the array of microphones, wherein the audio-visual source tracking system is configured to adjust and direct the sensitivity of the array of microphones at least partially in response to video information captured by the first video camera and the second video camera, and wherein the apparatus comprises a mobile handset.

10. The apparatus of claim 1 wherein each microphone of the array of microphones is configured for microphone array beam steering in two dimensions or in three dimensions.

11. The apparatus of claim 1 wherein the first video camera and the array of microphones are mounted to the housing proximate an exterior face of the housing.

12. The apparatus of claim 1 wherein a direction of the array of microphones is determined based, at least partially, on a first angle and a second angle, wherein the first angle corresponds to a ratio of the camera focal length and horizontal position of a reference point of the user, and wherein the second angle corresponds to a ratio of the camera focal length and a vertical position of a reference point of the user.

13. A method comprising:

providing a housing; and

connecting a tracking system to the housing, wherein the tracking system comprises a first video camera and an array of microphones, wherein the first video camera is attached to the housing, wherein the array of microphones is proximate the first video camera, wherein the array of microphones is configured to have an adjustable beam, wherein the first video camera and the array of microphones are proximate an exterior portion of the housing, wherein the tracking system is configured to receive a video signal from the first video camera, wherein the tracking system is configured to capture an audio signal from the array of microphones at least partially in response to the video signal, wherein the audio-visual source tracking system is configured to adjust and direct a sensitivity of the adjustable beam of the array of microphones while a user is speaking at least partially in response to the video information which tracks a head position of the user relative to the audio-visual source tracking system and is configured to update a location of the user, wherein the audio-visual source tracking system is further configured to adjust and direct the sensitivity of the adjustable beam of the array of microphones based at least partially upon a detected face of the user at a time after an initial face detection of the user, wherein a direction of the array of microphones is determined based, at least partially, on a ratio of the camera focal length and a reference point of the user, and wherein the housing is a portable electronic device housing of a portable electronic device.

14. The method of claim 13 wherein the array of microphones comprises at least three microphones.

15. The method of claim 14 wherein the providing of the tracking system further comprises providing a face detection system.

16. The method of claim 14 wherein the providing of the tracking system further comprises attaching the array of microphones around the first video camera.

17. The method of claim 13 wherein the housing forms a portion of an apparatus, and wherein the providing of the tracking system further comprises orienting the first video camera towards a user of the apparatus.

18. The method of claim 13 wherein the providing of the tracking system further comprises providing a second video camera, wherein the first and the second video cameras are configured to estimate a beam orientation for the array of microphones.

19. A method comprising:

capturing a first image with a camera of an apparatus;

determining an orientation of a portion of the first image;

controlling a predetermined characteristic of an array of microphones of the apparatus based at least partially on the orientation of the portion of the first image, wherein the array of microphones is configured to have an adjustable beam, wherein the camera and the array of microphones are proximate an exterior portion of the apparatus, wherein the controlling of the predetermined characteristic of the array of microphones further comprises controlling and directing a sensitivity of the adjustable beam of the array of microphones while a user is speaking in response to a tracked head position of the user relative to the audio-visual source tracking system, wherein the audio-visual source tracking system is further configured to adjust and direct the sensitivity of the adjustable beam of the array of microphones based at least partially upon a detected face of the user at a time after an initial face detection of the user, wherein a direction of the array of microphones is determined based, at least partially, on a ratio of the camera focal length and a reference point of the user, and wherein the apparatus is a portable electronic device; and

updating a location of the user.

20. The method of claim 19 wherein the capturing of the first image further comprises capturing an image of a user of the apparatus, and wherein the determining of the orientation of the portion of the first image, further comprises at least one of:

determining an orientation of a head of the user; and

detecting a face of a user of the apparatus in the first image.

21. The method of claim 20 wherein the controlling of the predetermined characteristic of the array of microphones further comprises controlling a predetermined characteristic of the array of microphones based at least partially on the determined head orientation.

22. The method of claim 21 wherein the controlling of the predetermined characteristic of the array of microphones further comprises adjusting and directing a sensitivity of the array of microphones based at least partially on the determined head orientation.

23. The method of claim 19 further comprising:

capturing a second image with the camera of the apparatus; and

determining an orientation of a portion of the another image based at least partially on the determined orientation of the portion of the first image.

24. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine for performing operations to sense an audio signal, the operations comprising:

analyzing a portion of an image with an audio-visual source tracking system;

calculating a direction corresponding to the portion of the image, wherein the direction is relative to an apparatus, and wherein the image is captured by a first and second cameras of the apparatus; and

directing an array of microphones from the apparatus based at least partially upon the calculated direction, wherein the first and second cameras and the array of microphones are proximate an exterior portion of the apparatus, wherein the array of microphones is configured to have an adjustable beam, wherein the audio-visual source tracking system is configured to adjust and direct a sensitivity of the adjustable beam of the array of microphones while a user is speaking at least partially in response to the calculated direction which tracks a head position of a user relative to the audio-visual source tracking system and is configured to update a location of the user, wherein the audio-visual source tracking system is further configured to adjust and direct the sensitivity of the adjustable beam of the array of microphones based at least partially upon a detected face of the user at a time after an initial face detection of the user, wherein a direction of the array of microphones is determined based, at least partially, on a ratio of the camera focal length and a reference point of the user, and wherein the apparatus is a portable electronic device.

25. The program storage device of claim 24 wherein the calculating of the direction corresponding to the portion of the image further comprises calculating a position of a mouth of a user of the apparatus captured in the image.