Audio lens

• US 9,716,943 B2
• Filed: 12/13/2012
• Issued: 07/25/2017
• Est. Priority Date: 12/21/2011
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Patwardhan, Pushkar Prasad, Vasilache, Adriana, Hagqvist, Jari Mathias, Laaksonen, Lasse Juhani, Ugur, Kemal
• Primary Examiner(s): Elahee, Md S
• Assistant Examiner(s): Diaz, Sabrina
• Application Number:
  US14/364,419
• Publication Number:
  US 20140348342A1
• Time in Patent Office: 1,685 Days
• Field of Search: 381 92
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2430/25 Array processing for suppression of unwanted side-lobes in directivity characteristics, e.g. a blocking matrix
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/40 Visual indication of stereophonic sound image

Abstract

An apparatus configured to: determine a viewing angle associated with at least one apparatus camera; determine from at least two audio signals at least one audio source orientation relative to an apparatus; and generate at least one spatial filter including at least a first orientation range associated with the viewing angle and a second orientation range relative to the apparatus.

Claims

1. An apparatus comprising at least one camera, at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to:

determine a viewing angle associated with the at least one camera;

determine from at least two audio signals at least one audio source orientation relative to the apparatus; and

generate at least one spatial filter defining a first orientation range associated with the viewing angle and a second orientation range associated with outside of the viewing angle so as to control a level of the at least one audio source dependent on whether the audio source is located within the viewing angle;

wherein the apparatus is caused to display a representation of the viewing angle on a user interface so as to receive a user interface input that comprises an indication of at least one of;

a spatial filtering region; and

a spatial filtering region gain value.

2. The apparatus as claimed in claim 1, wherein the determined at least one audio source orientation comprises a direction of the at least one audio source relative to the apparatus.

3. The apparatus as claimed in claim 1, wherein the apparatus further generates an audio signal associated with the at least one audio source from the at least two audio signals to be outputted after the at least one spatial filter filters the audio signal.

4. The apparatus as claimed in claim 3, wherein the spatial filter spatially filters the audio signal associated with the at least one audio source orientation.

5. The apparatus as claimed in claim 3, wherein the audio signal associated with the at least one audio source comprises one of:

a mono audio signal;

a binaural audio signal;

a stereo channel audio signal; and

a multi-channel audio signal.

6. The apparatus as claimed in claim 3, wherein the apparatus further comprises an audio output configured to output the audio signal to one of:

a headset;

a multichannel speaker system;

a memory; and

a transmitter.

7. The apparatus as claimed in claim 1, wherein the at least one spatial filter is generated dependent on the at least one audio source orientation relative to the apparatus.

8. The apparatus as claimed in claim 1, wherein the at least one spatial filter comprises a pass filter for the first orientation range, wherein the orientation of the at least one audio source is included.

9. The apparatus as claimed in claim 1, wherein the at least one spatial filter comprises a stop filter for the second orientation range, wherein the orientation of the at least one audio source is not included.

10. The apparatus as claimed in claim 1, wherein the determined viewing angle causes the apparatus to least one of:

determine a horizontal field of view of the at least one camera;

determine a vertical field of view of the at least one camera;

determine a focal length of the at least one camera;

determine a sensor size of the at least one camera;

determine an aspect ratio of the at least one camera;

determine a digital zoom of the at least one camera; and

determine a viewing angle from a user interface input.

11. The apparatus as claimed in claim 1, wherein the apparatus is caused to determine spatial filtering characteristics to be applied to an audio source outside the viewing angle, said characteristics including one or more of the following:

a frequency of roll-off; and

a degree of damping.

12. The apparatus as claimed in claim 1, wherein the at least two audio signals are provided by at least two microphones of the apparatus.

13. The apparatus as claimed in claim 1, wherein the apparatus further comprises a display configured to receive output from the at least one camera and to generate a video output comprising data of the at least one camera;

and

an audio signal generator configured to generate an audio signal associated with the at least one audio source from at least two audio signals.

14. The apparatus as claimed in claim 1 wherein the at least one memory and the computer code are further configured to with the at least one processor cause the apparatus to determine a spatial filter for the second orientation range configured not to include audio sources outside the viewing angle.

15. The apparatus as claimed in claim 1 wherein the at least one memory and the computer code are further configured to with the at least one processor cause the apparatus to use the viewing angle and the at least one audio source orientation to classify sounds into those that are located within the viewing angle and those that are located outside the viewing angle.

16. The apparatus as claimed in claim 1 wherein the level of audio sources is separately amplified or muted based, at least partially, on video and audio information.

17. The apparatus as claimed in claim 1 wherein the at least one memory and the computer code are further configured to with the at least one processor cause the apparatus to determine spatial filtering characteristics to be applied to audio signals outside of the viewing angle.

18. The apparatus as claimed in claim 1 wherein the at least one memory and the computer code are further configured to with the at least one processor cause the apparatus to spatially filter playback audio using a separate audio profile based on the at least two audio signals.

19. The apparatus as claimed in claim 1 wherein more than one region outside of the viewing angle comprises different spatial audio filter gain profiles for each region.

20. The apparatus as claimed in claim 1 wherein the at least one memory and the computer code are further configured to with the at least one processor cause the apparatus to focus on a center region within the viewing angle with a lower audio gain value than the sources outside of the viewing angle.

21. A method for audio signal adjustment comprising:

determining a viewing angle associated with at least one camera;

determining from at least two audio signals at least one audio source orientation relative to the apparatus; and

generating at least one spatial filter defining at least a first orientation range associated with the viewing angle and a second orientation range associated with outside of the viewing angle relative to the apparatus for controlling a level of the at least one audio source dependent on whether the audio source is located within the viewing angle;

such that displaying a representation of the viewing angle on a user interface so as to receive a user interface input that comprises an indication of at least one of;

a spatial filtering region; and

a spatial filtering region gain value.

22. The method for audio signal adjustment as claimed in claim 21, further comprising:

generating an audio signal associated with the at least one audio source from the at least two audio signals;

filtering the audio signal associated with the at least one audio source dependent on the at least one spatial filter; and

outputting the audio signal after filtering.

23. The method for audio signal adjustment as claimed in claim 21, wherein determining the at least one audio source orientation comprises a direction of the at least one audio source relative to the apparatus.

24. The method for audio signal adjustment as claimed in claim 22, wherein the spatial filter spatially filters the audio signal associated with the at least one audio source orientation.

25. The method for audio signal adjustment as claimed in claim 21, further comprising generating the at least one spatial filter dependent on the at least one audio source orientation relative to the apparatus.

26. The method for audio signal adjustment as claimed in claim 21, wherein the filter comprises at least one of:

a pass filter for the first range to determine the orientation of the at least one audio source relative to the apparatus; and

a stop filter for the second range wherein the orientation of the at least one audio source is not included.

27. The method for audio signal adjustment as claimed in claim 21, further comprising determining spatial filtering characteristics to be applied to an audio source outside the viewing angle, said characteristics including one or more of the following:

a frequency of roll-off; and

a degree of damping.

28. An apparatus comprising at least one camera, at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to:

determine a viewing angle associated with the at least one camera;

determine from at least two audio signals at least one audio source orientation relative to the apparatus; and

generate at least one spatial filter defining a first orientation range associated with the viewing angle and a second orientation range associated with outside of the viewing angle so as to control the level of the at least one audio source dependent on whether the at least one audio source is located within or outside the viewing angle;

wherein the apparatus is caused to display a representation of the viewing angle on a touch screen, wherein the representation of the viewing angle comprises an indication of a respective gain value for at least one of the first and second orientation ranges, and to receive, via a portion of the touch screen caused to display said representation of the viewing angle, a user interface input that comprises an indication of a change in said respective gain value that controls the level of the at least one audio source dependent on whether the at least one audio source is within or outside the viewing angle.