Automatic audio ducking with real time feedback based on fast integration of signal levels

• US 10,642,571 B2
• Filed: 11/06/2017
• Issued: 05/05/2020
• Est. Priority Date: 11/06/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Becherer, Nico, Duwenhorst, Sven
• Primary Examiner(s): Nguyen, Cao H
• Application Number:
  US15/804,526
• Publication Number:
  US 20190138262A1
• Time in Patent Office: 911 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/27 characterised by the analysis technique
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Various embodiments describe audio signal processing. In an example, a computer system generates metrics, such as RMS levels, for audio slices from a foreground audio signal. A summed-area table is generated from the metrics. An observation window is used to determine whether to add a key frame or not. The observation window includes a set of audio slices. A total metrics, such as an average RMS level, is computed for the audio slices in the observation window. Based on the total metric, the computer system adds a key frame. The key frame references audio ducking parameters applicable to a background audio signal.

Claims

1. A computer-implemented method for audio signal processing, the method comprising:

accessing, by a multimedia editing application hosted on a computing device, a first audio signal associated with a foreground label, wherein the foreground label indicates that the first audio signal is a foreground audio signal;

analyzing, by the multimedia editing application, the first audio signal to determine metrics corresponding to audio slices of the first audio signal,

wherein each metric corresponds to a respective audio slice, indicates a value for an audio property in the respective audio slice, and is generated by the multimedia editing application based on an audio signal of the respective audio slice;

computing, by the multimedia editing application, a total metric for an audio slice as a function of a set of the metrics corresponding to a set of the audio slices, wherein the set of the audio slices includes the audio slice; and

adding, by the multimedia editing application, a key frame to a track based on the total metric, wherein the track organizes a presentation of the first audio signal and of a second audio signal having a background label, wherein a location of the key frame corresponds to a location of the audio slice on the track, and wherein the key frame indicates a change to the audio property of the second audio signal at the location on the track,

wherein the key frame added to the track based on the total metric is utilizable for audio ducking, the audio ducking comprising adapting a property of the second audio signal based on a state of the first audio signal.

2. The computer-implemented method of claim 1, wherein the first audio signal has a time length, wherein each of the audio slices has a first predefined time length smaller than the time length of the first audio signal, and wherein the set of the audio slices forms an observation window having a second predefined time length larger than the first predefined time length and smaller than the time length of the first audio signal.

3. The computer-implemented method of claim 2, wherein the first predefined time length of each audio slice is in a range of 2 ms to 20 ms, and wherein the second predefined time length is in a range of 30 ms to 200 ms.

4. The computer-implemented method of claim 1, further comprising:

generating, by the multimedia editing application, a summed-area table data structure by computing entries of the summed-area table data structure based on the metrics of the audio slices.

5. The computer-implemented method of claim 4, wherein the metrics are root mean squares (RMS) levels, and wherein the total metric is an average RMS level computed based on the summed-area table data structure.

6. The computer-implemented method of claim 5, wherein the set of the audio slices is organized in a sequence that starts with the audio slice for which the total metric is computed and that ends with another audio slice, and wherein the average RMS level is computed from an entry of the summed-area table data structure corresponding to the audio slice and from another entry of the summed-area table data structure corresponding to the other audio slice.

7. The computer-implemented method of claim 5, further comprising:

generating, by the multimedia editing application, a leap table based on changes to the metrics of the audio slices, wherein an entry of the leap table indicates that a change to the audio property between two audio slices of the first audio signal is within a predefined range.

8. The computer-implemented method of claim 7, further comprising:

determining, by the multimedia editing application, that a particular key frame between locations of the two audio slices on the track is unneeded based on the entry of the leap table; and

skipping, by the multimedia editing application, an addition of the particular key frame between the locations of the two audio slices on the track.

9. The computer-implemented method of claim 1, wherein the key frame is added to the track based on a comparison of the total metric to a threshold, and wherein the threshold is defined based on parameters associated with ducking the second audio signal against the first audio signal.

10. The computer-implemented method of claim 1, wherein the key frame is added to the track based on a distance between the first audio signal and the second audio signal, wherein the distance is computed based on the total metric generated for the audio slice of the first audio signal and based on a second total metric generated for a second audio slice of the second audio signal, and wherein the audio slice and the second audio slice have a common location on the track.

11. The computer-implemented method of claim 1, further comprising:

presenting, by the multimedia editing application, the track on a user interface; and

receiving, by the multimedia editing application, user input specifying the foreground label of the first audio signal, the background label of the second audio signal, a ducking of the second audio signal against the first audio signal, and parameters associated with the ducking, and wherein the change to the audio property of the first audio signal is based on the parameters associated with the ducking.

12. The computer-implemented method of claim 1, further comprising:

generating, by the multimedia editing application, a summed-area table data structure comprising entries as a function of the metrics of the audio slices;

receiving, by the multimedia editing application, user input specifying a movement of the first audio signal on the track;

updating, by the multimedia editing application, the track based on the movement of the first audio signal;

computing, by the multimedia editing application, a second total metric for the audio slice based on the summed-area table data structure, wherein the summed-area table data structure remains unchanged based on the movement of the first audio signal; and

adding, by the multimedia editing application, a second key frame to the track based on the second total metric.

13. The computer-implemented method of claim 1, further comprising:

generating, by the multimedia editing application, a summed-area table data structure comprising entries as a function of the metrics of the audio slices;

receiving, by the multimedia editing application, user input specifying a gain of the first audio signal;

updating, by the multimedia editing application, the entries of the summed-area table data structure based on the gain without re-generating the metrics; and

adding, by the multimedia editing application, a second key frame to the track based on the updated entries of the summed-area table.

14. A non-transitory computer-readable storage medium comprising instructions that, upon execution on a computer system, cause the computer system to perform operations comprising:

accessing a foreground audio signal;

analyzing the foreground audio signal to determine metrics corresponding to audio slices of the foreground audio signal,

wherein each metric corresponds to a respective audio slice, indicates a value for an audio property in the respective audio slice, and is generated automatically based on an audio signal of the respective audio slice;

computing a total metric for an audio slice as a function of a set of the metrics corresponding to a set of the audio slices, wherein the set of the audio slices includes the audio slice; and

adding a key frame to a track based on the total metric, wherein the track organizes a presentation of the foreground audio signal and of a background audio signal, wherein a location of the key frame corresponds to a location of the audio slice on the track, and wherein the key frame indicates a change to the audio property of the background audio signal at the location on the track,

wherein the key frame added to the track based on the total metric is utilizable for audio ducking, the audio ducking comprising adapting a property of the background audio signal based on a state of the foreground audio signal.

15. The non-transitory computer-readable storage medium of claim 14, wherein the foreground audio signal has a label indicating that an audio signal is foreground, and wherein accessing the foreground audio signal comprising accessing the label from metadata of the audio signal.

16. The non-transitory computer-readable storage medium of claim 14, wherein adding the key frame to the track comprises:

determining that the background audio signal is un-ducked and that the total metric indicates that the background audio signal should be ducked; and

setting the key frame based on parameters associated with ducking the background audio signal.

17. The non-transitory computer-readable storage medium of claim 14, wherein adding the key frame to the track comprises:

determining that the background audio signal is ducked and that the total metric indicates that the background audio signal should be un-ducked; and

setting the key frame based on parameters associated with un-ducking the background audio signal.

18. A computer system comprising:

means for accessing a foreground audio signal;

means for analyzing the foreground audio signal to determine metrics corresponding to audio slices of the foreground audio signal,

wherein each metric corresponds to a respective audio slice, indicates a value for an audio property in the respective audio slice, and is generated automatically based on an audio signal of the respective audio slice;

means for computing a total metric for an audio slice as a function of a set of the metrics corresponding to a set of the audio slices, wherein the set of the audio slices includes the audio slice; and

means for adding a key frame to a track based on the total metric, wherein the track organizes a presentation of the foreground audio signal and of a background audio signal, wherein a location of the key frame corresponds to a location of the audio slice on the track, and wherein the key frame indicates a change to the audio property of the background audio signal at the location on the track,

wherein the key frame added to the track based on the total metric is utilizable for audio ducking, the audio ducking comprising adapting a property of the background audio signal based on a state of the foreground audio signal.

19. The computer system of claim 18, further comprising:

means for presenting a user interface associated with ducking the background audio signal,

means for receiving, based on the user interface, user input labeling the foreground audio signal as foreground, specifying the ducking of the background audio signal against the foreground audio signal, and identifying parameters associated with the ducking, wherein the change to the audio property of the background audio signal is based on the parameters associated with the ducking.

20. The computer system of claim 19, wherein the user input specifies a threshold, a gain reduction, and a fade timing as the parameters, and the computer system further comprising:

means for determining a look-ahead time, a hold time, and a fade length based on the fade timing, wherein the key frame is added to the track based on the threshold, the gain reduction, the look-ahead time, the hold time, and the fade length.