SPATIALLY BIASED SOUND PICKUP FOR BINAURAL VIDEO RECORDING

• US 20200137489A1
• Filed: 09/18/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Sheaffer, Jonathan D., Atkins, Joshua D., Raffensperger, Peter A., Delikaris Manias, Symeon
• Granted Patent: US 11,012,774 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 19/08 Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence

Abstract

A method for producing a target directivity function that includes a set of spatially biased HRTFs. A set of left ear and right ear head related transfer functions (HRTFs) are selected. The left ear and right ear head HRTFs are multiplied with an on-camera emphasis function (OCE), to produce the spatially biased HRTFs. The OCE may be designed to shape the sound profile of the HRTFs to provide emphasis in a desired location or direction that is a function of the specific orientation of the device as it is being used to make a video recording. Other aspects are also described and claimed.

Claims

1. A method for producing a spatially biased sound pickup beamforming function, the method comprising:

generating a target directivity function that includes a set of spatially biased head related transfer functions;

generating a left ear set of beamforming coefficients and a right ear set of beamforming coefficients by determining a fit for the target directivity function based on a device steering matrix; and

outputting the left ear set of beamforming coefficients and the right ear set of beamforming coefficients.

2. The method of claim 1, wherein the device steering matrix includes a plurality of transfer functions of a plurality of microphones, wherein each of the transfer functions describes a response by a respective one of the microphones to a single sound source direction.

3. The method of claim 2, wherein the fit for the target directivity coefficients is determined by utilizing a least squares method, wherein the least squares method comprises inputting the target directivity function and the device steering matrix into a least squares beamformer design algorithm.

4. The method of claim 3, wherein the least squares beamformer design algorithm includes a determined white-noise gain constraint while determining a fit for the target directivity function based on a device steering matrix for a first ear.

5. The method of claim 4, wherein the least squares method produces regularizer values due to the white-noise gain constraint while generating the set of beamforming coefficients for the first ear, and the regularizer values produced for the first ear are used in the least squares method for generating the set of beamforming coefficients for a second ear.

6. The method of claim 5, wherein the first ear is the left ear if the left side of the device has a lower density of microphones than the right side of the device, or the first ear is the right ear if the right side of the device has a lower density of microphones than the left side of the device.

7. The method of claim 1, further comprising:

selecting a set of head related transfer functions (HRTFs); and

selecting an on-camera emphasis function (OCE) in response to detecting an orientation of a video recording device that has a plurality of possible orientations for capturing audio-video, wherein the OCE includes a plurality of spatial weights,

wherein generating the target directivity function comprises producing the set of spatially biased HRTF by multiplying in frequency domain the set of HRTFs with a first set of spatial weights from the OCE that emphasize sound from a first desired direction.

8. The method of claim 1, further comprising processing the left ear beamforming coefficients and the right ear beamforming coefficients with an asymmetric equalizer.

9. The method of claim 1, wherein the left ear beamforming coefficients and the right ear beamforming coefficients are associated with an orientation of a multimedia recording device while the device is recording audio and video, wherein said device can record in a plurality of orientations.

10. The method of claim 9, further comprising generating a beamforming coefficients library that includes a plurality of beamforming coefficients wherein each of a set of left ear beamforming coefficients and right ear beamforming coefficients are associated with a respective orientation of the device.

11. The method of claim 1 further comprising:

applying the left ear set of beamforming coefficients and the right ear set of beamforming coefficients to a multi-channel audio recording of an audio-video recording made by a multimedia recording device, during playback of the audio-video recording.

12. A method for producing a target directivity function, the method comprising:

selecting a set of head related transfer functions (HRTFs);

selecting an on-camera emphasis function (OCE) that is specific to an orientation of a video recording device that can record audio and video in a plurality of orientations, wherein the OCE includes a plurality of spatial weights; and

generating a set of spatially biased HRTFs, wherein the set of spatially biased HRTFs are generated by multiplying in frequency domain the set of HRTFs with a first set of spatial weights from the OCE that emphasize sound from a first desired direction.

13. The method of claim 12 wherein selecting the OCE is in response to detecting that the video recording device is zooming in, and wherein the selected OCE when zooming in has a narrower sound pickup beam width or higher directivity than another OCE that is selected when the video recording device is not zooming in.

14. The method of claim 12 further comprising detecting that the recording device is zooming out, in response to which a default OCE is selected.

15. The method of claim 12 further comprising detecting that the recording device is zooming out, and in response providing the selected set of HRTFs directly to a spatial sound renderer for binaural rendering without any spatial bias that would be present due to application of the OCE.

16. The method of claim 12, wherein the OCE is selected from a plurality of OCEs, wherein each OCE is specific to an orientation of the device and the selected OCE is associated with an orientation that matches the orientation of the device while the device is being used to record the audio and video.

17. The method of claim 12 wherein the OCE is designed to produce a desired sound profile that emphasizes spatial focus in a determined direction and reduces sound level at undesired directions, wherein the determined direction matches a direction at which a camera of the device is aimed to record video.

18. A system for producing a sound pickup beamforming function to be applied to a multi-channel audio recording made by a video recording device, comprising

a processor;

and

memory having stored therein instructions that when executed by the processor

generate a target directivity function that includes a set of spatially biased head related transfer functions,

generate a left ear set of beamforming coefficients and a right ear set of beamforming coefficients by determining a fit for the target directivity function based on a device steering matrix that describes beamforming capability of a microphone array in the video recording device;

output the left ear set of beamforming coefficients and the right ear set of beamforming coefficients

19. The system of claim 18, wherein the device steering matrix includes transfer functions of a plurality of microphones that constitute the microphone array.

20. A method for asymmetric equalization, comprising:

a) receiving a set of beamforming coefficients for a first ear;

b) calculating a diffuse power average across a plurality of beamforming coefficients from the received set of beamforming coefficients; and

c) applying a correction filter to the received set of beamforming coefficients such that the diffuse field power average of the plurality of beamforming coefficients equals the diffuse field power average of a single microphone of a microphone array.

21. The method of claim 20 further comprising:

receiving a further set of beamforming coefficients for a second ear;

calculating a diffuse power average across a further plurality of beamforming coefficients from the received further set of beamforming coefficients; and

applying a correction filter to the received further set of beamforming coefficients such that the diffuse field power average of the further plurality of beamforming coefficients equals the diffuse field power average of a single microphone of the microphone array.