Associating Audio with Three-Dimensional Objects in Videos

• US 20170366896A1
• Filed: 06/20/2016
• Published: 12/21/2017
• Est. Priority Date: 06/20/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Macmillan, Timothy
• Granted Patent: US 10,045,120 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 2205/01 specially adapted for specific applications
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G01S 5/30 Determining absolute distances from a plurality of spaced points of known location
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02

Abstract

Disclosed is a system and method for generating a model of the geometric relationships between various audio sources recorded by a multi-camera system. The spatial audio scene module associates source signals, extracted from recorded audio, of audio sources to visual objects identified in videos recorded by one or more cameras. This association may be based on estimated positions of the audio sources based on relative signal gains and delays of the source signal received at each microphone. The estimated positions of audio sources are tracked indirectly by tracking the associated visual objects with computer vision. A virtual microphone module may receive a position for a virtual microphone and synthesize a signal corresponding to the virtual microphone position based on the estimated positions of the audio sources.

Claims

1. A method for locating and tracking one or more audio sources recorded by a plurality of microphones, the method comprising:

receiving positions and orientations for each of at least one camera;

receiving positions for each of the plurality of microphones.

receiving at least one video recorded by a camera;

receiving a plurality of audio signals, each audio signal recorded by a microphone of the plurality of microphones;

applying source separation to the plurality of audio signals to generate one or more audio source signals, each audio source signal having originated from a respective audio source of the one or more audio sources;

estimating, for each audio source, a position associated with the audio source;

estimating a position with computer vision for each of one or more visual objects based on a visual analysis of the at least one video and the at least one position of the at least one camera;

matching each of the one or more audio sources to a corresponding visual object of the one or more visual objects based on the estimated position of the audio source and the estimated position of the visual object;

tracking movement of the one or more visual objects to generate visual object position data associated with movement of the one or more visual objects; and

storing audio source position data for each of the one or more audio sources based on the visual object position data associated with the visual object to which the audio source was matched.

2. The method of claim 1, wherein applying source separation to the plurality of audio signals to generate one or more audio source signals comprises:

determining a first audio component, the first component a component of a first audio signal of the plurality of audio signals; and

determining a second audio component, the second audio component a component of a second audio signal of the plurality of audio signals, the second audio signal different than the first audio signal, wherein the second audio component and the first audio component both correspond to an audio source of the one or more audio sources.

3. The method of claim 2, wherein estimating a position for an audio source of the one or more audio sources comprises:

determining a relative signal power of the first audio component compared to the second audio signal;

determining a relative time delay of the first audio component compared to the second audio signal; and

estimating a position for the audio source that the first and second audio component correspond to based on the relative signal power and the relative time delay.

4. The method of claim 1, wherein each of the audio source signals comprises two or more signal components, each signal component associated with a respective audio signal of the plurality of audio signals.

5. The method of claim 1, wherein estimating a position with computer vision for each of the one or more visual objects in the first time period based on the visual analysis of the at least one video comprises:

identifying a visual object of the one or more visual objects in two or more of the at least one video; and

estimating the position for the visual object based on triangulation.

6. The method of claim 1, wherein estimating, for each audio source, the position associated with the audio source, comprises:

determining, for each audio source, an estimated signal gain, the estimated signal gain associated with a first audio signal of the plurality of audio signals and the audio source signal that originated from the audio source, the signal gain indicating a magnitude of a component of the first audio signal that corresponds to the audio source signal.

7. The method of claim 6, wherein estimating, for each audio source, the position associated with the audio source is based on:

the estimated signal gain associated with the first audio signal and the audio source signal that originated from the audio source;

the orientation of the microphone of the plurality of microphones that recorded the first audio signal; and

a model of a directional gain for the microphone.

8. The method of claim 1, wherein estimating, for each audio source, the position associated with the audio source, comprises:

determining, for each audio source, an estimated delay, the estimated delay associated with a first audio signal of the plurality of audio signals and the audio source signal that originated from the audio source, the delay indicating a time offset of a component of the first audio signal that corresponds to the audio source signal.

9. The method of claim 1, wherein the microphones are directional microphones.

10. The method of claim 1, wherein each microphone of the plurality of microphones is an integrated microphone in a respective camera of the at least one cameras.

11. A non-transitory computer readable medium storing instructions for locating and tracking one or more audio sources recorded by a plurality of microphones, wherein the instructions when executed by one or more computer processors cause the one or more processors to perform steps comprising:

receiving at least one position for each of at least one camera;

receiving positions for each of the plurality of microphones.

receiving at least one video recorded by a camera;

receiving a plurality of audio signals, each audio signal recorded by a microphone of the plurality of microphones;

applying source separation to the plurality of audio signals to generate one or more audio source signals, each audio source signal having originated from a respective audio source of the one or more audio sources;

estimating, for each audio source, a position associated with the audio source;

estimating a position for each of one or more visual objects based on the at least one video;

matching each of the one or more audio sources to a corresponding visual object of the one or more visual objects based on the estimated position of the audio source and the estimated position of the visual object;

tracking movement of each of the one or more visual objects to generate visual object position data associated with movement of the one or more visual objects; and

storing audio source position data for each of the one or more audio sources based on the visual object position data associated with the visual object to which the audio source was matched.

12. The non-transitory computer readable medium of claim 11, wherein applying source separation to the plurality of audio signals to generate one or more audio source signals comprises:

determining a first audio signal, the first audio signal a component of a first audio signal of the plurality of audio signals; and

determining a second audio signal, the second audio signal a component of a second audio signal of the plurality of audio signals, the second audio signal different than the first audio signal, wherein the second audio signal and the first audio signal both correspond to an audio source of the one or more audio sources.

13. The non-transitory computer readable medium of claim 12, wherein estimating a position for an audio source of the one or more audio sources comprises:

determining a relative signal power of the first audio signal compared to the second audio signal;

determining a relative time delay of the first audio signal compared to the second audio signal; and

estimating a position for the audio source based on the relative signal power and the relative time delay.

14. The non-transitory computer readable medium of claim 11, wherein estimating a position with for each of the one or more visual objects in the first time period based on the at least one video comprises:

identifying a visual object of the one or more visual objects in two or more of the at least one video; and

estimating the position for the visual object based on triangulation.

15. The non-transitory computer readable medium of claim 11, wherein estimating, for each audio source, the position associated with the audio source, comprises:

determining, for each audio source, an estimated signal gain, the estimated signal gain associated with a first audio signal of the plurality of audio signals and the audio source signal that originated from the audio source, the signal gain indicating a magnitude of a component of the first audio signal that corresponds to the audio source signal.

16. The non-transitory computer readable medium of claim 15, wherein estimating, for each audio source, the position associated with the audio source is based on:

the estimated signal gain associated with the first audio signal and the audio source signal that originated from the audio source; and

a model of a directional gain for the microphone of the plurality of microphones that recorded the first audio signal.

17. The non-transitory computer readable medium of claim 11, wherein estimating, for each audio source, the position associated with the audio source, comprises:

determining, for each audio source, an estimated delay, the estimated delay associated with a first audio signal of the plurality of audio signals and the audio source signal that originated from the audio source, the delay indicating a time offset of a component of the first audio signal that corresponds to the audio source signal.

18. The non-transitory computer readable medium of claim 11, wherein each microphone of the plurality of microphones is an integrated microphone in a respective camera of the at least one cameras.

19. A method for locating and tracking one or more audio sources recorded by a plurality of microphones, the method comprising:

receiving at least one position for each of at least one camera;

receiving positions for each of the plurality of microphones.

receiving at least one video recorded by a camera;

receiving a plurality of audio signals, each audio signal recorded by a microphone of the plurality of microphones;

estimating a position for each of one or more visual objects based on the at least one video corresponding to a first time in the at least one video;

applying source separation to the plurality of audio signals to generate one or more audio source signals, each audio source signal having originated from a respective audio source of the one or more audio sources;

estimating, for each audio source, a position associated with the audio source corresponding to the first time;

matching each of the one or more audio sources to a corresponding visual object of the one or more visual objects based on the estimated position of the audio source and the estimated position of the visual object;

tracking the position of each of the one or more audio sources by tracking the visual object to which the audio source is matched.

20. The method of claim 19, wherein estimating, for each audio source, the position associated with the audio source, comprises:

determining, for each audio source, an estimated signal gain, the estimated signal gain associated with a first audio signal of the plurality of audio signals and the audio source signal that originated from the audio source, the signal gain indicating a magnitude of a component of the first audio signal that corresponds to the audio source signal; and

estimating the position of each audio source based on the signal gain associated with the audio source.