Associating audio with three-dimensional objects in videos

• US 10,045,120 B2
• Filed: 06/20/2016
• Issued: 08/07/2018
• Est. Priority Date: 06/20/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Macmillan, Timothy
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): YANG, NIEN
• Application Number:
  US15/187,695
• Publication Number:
  US 20170366896A1
• Time in Patent Office: 778 Days
• Field of Search: 386239
• CPC Class Codes: G01S 2205/01 specially adapted for specific applications
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G01S 5/30 Determining absolute distances from a plurality of spaced points of known location
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02

Abstract

Disclosed is a system and method for generating a model of the geometric relationships between various audio sources recorded by a multi-camera system. The spatial audio scene module associates source signals, extracted from recorded audio, of audio sources to visual objects identified in videos recorded by one or more cameras. This association may be based on estimated positions of the audio sources based on relative signal gains and delays of the source signal received at each microphone. The estimated positions of audio sources are tracked indirectly by tracking the associated visual objects with computer vision. A virtual microphone module may receive a position for a virtual microphone and synthesize a signal corresponding to the virtual microphone position based on the estimated positions of the audio sources.

Claims

1. A method for locating and tracking one or more audio sources recorded by a set of microphones, the set of microphones including a first microphone and a second microphone, the method comprising:

receiving position information for a camera;

receiving position information for individual ones of the microphones in the set of microphones;

receiving a video recorded by the camera, the video including a first visual object and a second visual object, the first visual object being a first audio source and the second visual object being a second audio source;

receiving audio signals recorded by the microphones, the audio signals including a first audio signal recorded by the first microphone and a second audio signal recorded by the second microphone, individual ones of the audio signals including sounds generated by the first audio source and the second audio source, wherein the first audio signal includes a first audio component corresponding to the sounds generated by the first audio source and a second audio component corresponding to the sounds generated by the second audio source, and the second audio signal includes a third audio component corresponding to the sounds generated by the first audio source and a fourth audio component corresponding to the sounds generated by the second audio source;

applying source separation to the audio signals to generate individual audio source signals for the sounds generated by individual audio sources, the audio source signals including a first audio source signal and a second audio source signal, the first audio source signal including the sounds generated by the first audio source and the second audio source signal including the sounds generated by the second audio source, wherein the first audio source signal is generated by combining the first audio component and the third audio component; and

the second audio source signal is generated by combining the second audio component and the fourth audio component;

estimating positions of the first audio source and the second audio source based on the position information for the microphones;

estimating positions of the first visual object and the second visual object based on a visual analysis of the video and the position information for the camera;

matching the individual audio sources to corresponding visual objects based on the estimated positions of the audio sources and the estimated positions of the visual objects such that the first audio source is matched to the first visual object and the second audio source is matched to the second visual object;

tracking movement of the visual objects to generate visual object position data associated with movement of the visual objects; and

storing audio source position data for the individual audio source signals based on the visual object position data associated with the visual object to which the respective audio source was matched.

2. The method of claim 1, wherein estimating the position of the first audio source comprises:

determining an estimated signal gain of the first audio signal, the signal gain indicating a magnitude of a component of the first audio signal.

3. The method of claim 2, wherein estimating the position of the first audio source is based on:

the estimated signal gain of the first audio signal;

the orientation of the first microphone that recorded the first audio signal; and

a model of a directional gain for the first microphone.

4. The method of claim 1, wherein estimating the position of the first audio source comprises:

determining an estimated delay of the first audio signal, the delay indicating a time offset of a component of the first audio signal.

5. The method of claim 1, wherein the microphones are directional microphones.

6. The method of claim 1, wherein the individual ones of the microphones are integrated in a respective camera.

14. The method of claim 1, wherein:

the first audio signal includes a first audio component corresponding to the sounds generated by the first audio source and a second audio component corresponding to the sounds generated by the second audio source; and

the first audio source signal is generated by suppressing the second audio component of the first audio signal.

15. The method of claim 1, applying source separation to the audio signals to generate the individual audio source signals for the sounds generated by the individual audio sources includes:

identifying a given class of the first visual object, the given class associated with particular sounds or particular forms of sounds; and

separating the audio signals based on the given class of the first visual object.

16. The method of claim 15, wherein the given class of the first visual object is identified or the audio signals are separated based on the given class of the first visual object using a machine learning model, the machine learning model trained with audio training data that is cataloged or annotated based on types of audio sources.

17. The method of claim 1, wherein the source separation is applied based on source information about the audio sources, the source information identifying a number of expected audio source signals.

18. The method of claim 17, wherein the source information further identifies audio characteristics of the sounds generated by the audio sources.

19. The method of claim 18, wherein the source information is selected based on visual cues in the video associated with the audio sources.

7. A non-transitory computer readable medium storing instructions for locating and tracking one or more audio sources recorded by a set of microphones, the set of microphones including a first microphone and a second microphone, wherein the instructions when executed by one or more computer processors cause the one or more processors to perform steps comprising:

receiving position information for a camera;

receiving position information for individual ones of the microphones in the set of microphones;

receiving a video recorded by the camera, the video including a first visual object and a second visual object, the first visual object being a first audio source and the second visual object being a second audio source;

receiving audio signals recorded by the microphones, the audio signals including a first audio signal recorded by the first microphone and a second audio signal recorded by the second microphone, individual ones of the audio signals including sounds generated by the first audio source and the second audio source, wherein the first audio signal includes a first audio component corresponding to the sounds generated by the first audio source and a second audio component corresponding to the sounds generated by the second audio source, and the second audio signal includes a third audio component corresponding to the sounds generated by the first audio source and a fourth audio component corresponding to the sounds generated by the second audio source;

applying source separation to the signals to generate individual audio source signals for the sounds generated by individual audio sources, the audio source signals including a first audio source signal and a second audio source signal, the first audio source signal including the sounds generated by the first audio source and the second audio source signal including the sounds generated by the second audio source, wherein the first audio source signal is generated by combining the first audio component and the third audio component; and

the second audio source signal is generated by combining the second audio component and the fourth audio component;

estimating positions of the first audio source and the second audio source based on the position information for the microphones;

estimating positions of the first visual object and the second visual object based on a visual analysis of the video and the position information for the camera;

matching the individual audio sources to corresponding visual objects based on the estimated positions of the audio sources and the estimated positions of the visual objects such that the first audio source is matched to the first visual object and the second audio source is matched to the second visual object;

tracking movement of each of the visual objects to generate visual object position data associated with movement of the visual objects; and

storing audio source position data for the individual audio source signals based on the visual object position data associated with the visual object to which the respective audio source was matched.

8. The non-transitory computer readable medium of claim 7, wherein estimating the position of the first audio source comprises:

determining an estimated signal gain of the first audio signal, the signal gain indicating a magnitude of a component of the first audio signal.

9. The non-transitory computer readable medium of claim 8, wherein estimating the position of the first audio source is based on:

the estimated signal gain of the first audio signal; and

a model of a directional gain for the first microphone that recorded the first audio signal.

10. The non-transitory computer readable medium of claim 7, wherein estimating the position of the first audio source comprises:

determining an estimated delay of the first audio signal, the delay indicating a time offset of a component of the first audio signal.

11. The non-transitory computer readable medium of claim 7, wherein the individual ones of the microphones are integrated in a respective camera.

12. A method for locating and tracking one or more audio sources recorded by a set of microphones, the set of microphones including a first microphone and a second microphone, the method comprising:

receiving position information for a camera;

receiving position information for individual ones of the microphones in the set of microphones;

receiving a video recorded by the camera, the video including a first visual object and a second visual object, the first visual object being a first audio source and the second visual object being a second audio source;

receiving audio signals recorded by the microphones, the audio signals including a first audio signal recorded by the first microphone and a second audio signal recorded by the second microphone, individual ones of the audio signals including sounds generated by the first audio source and the second audio source, wherein the first audio signal includes a first audio component corresponding to the sounds generated by the first audio source and a second audio component corresponding to the sounds generated by the second audio source, and the second audio signal includes a third audio component corresponding to the sounds generated by the first audio source and a fourth audio component corresponding to the sounds generated by the second audio source;

estimating positions of the first visual object and the second visual object based on a visual analysis of the video corresponding to a first time in the video and the position information for the camera;

applying source separation to the audio signals to generate individual audio source signals for the sounds generated by individual audio sources, the audio source signals including a first audio source signal and a second audio source signal, the first audio source signal including the sounds generated by the first audio source and the second audio source signal including the sounds generated by the second audio source, wherein the first audio source signal is generated by combining the first audio component and the third audio component; and

the second audio source signal is generated by combining the second audio component and the fourth audio component;

estimating positions of the first audio source and the second audio source corresponding to the first time based on the position information for the microphones;

matching the individual audio sources to corresponding visual objects based on the estimated positions of the audio sources and the estimated positions of the visual objects such that the first audio source is matched to the first visual object and the second audio source is matched to the second visual object; and

tracking the positions of the individual audio sources by tracking the visual objects to which the audio sources are matched.

13. The method of claim 12, wherein estimating the position of the first audio source comprises:

determining an estimated signal gain of the first audio signal, the signal gain indicating a magnitude of a component of the first audio signal; and

estimating the position of the first audio source based on the signal gain of the first audio signal.