Dynamic audio ducking

• US 8,428,758 B2
• Filed: 02/16/2009
• Issued: 04/23/2013
• Est. Priority Date: 02/16/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Naik, Devang Kalidas, Silverman, Kim Ernest Alexander, Paquier, Baptiste Pierre, Zhang, ShawShin, Rottler, Benjamin Andrew
• Primary Examiner(s): ELBIN, JESSE A
• Application Number:
  US12/371,861
• Publication Number:
  US 20100211199A1
• Time in Patent Office: 1,527 Days
• Field of Search: 381/56, 381/57, 381/73.1, 381/94.5, 381/107, 381/108, 700/94, 715/716, 715/727
• US Class Current: 700/94
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence

Abstract

Various dynamic audio ducking techniques are provided that may be applied where multiple audio streams, such as a primary audio stream and a secondary audio stream, are being played back simultaneously. For example, a secondary audio stream may include a voice announcement of one or more pieces of information pertaining to the primary audio stream, such as the name of the track or the name of the artist. In one embodiment, the primary audio data and the voice feedback data are initially analyzed to determine a loudness value. Based on their respective loudness values, the primary audio stream may be ducked during the period of simultaneous playback such that a relative loudness difference is generally maintained with respect to the loudness of the primary and secondary audio streams. Accordingly, the amount of ducking applied may be customized for each piece of audio data depending on its loudness characteristics.

Claims

1. A method, comprising:

selecting a primary media item for playback on an electronic device;

selecting a secondary media item for playback on the electronic device; and

ducking the primary media item by a ducking value while the second media item is played based upon a desired relative loudness difference, such that the relative loudness difference is substantially maintained and such that the primary media item is played at a ducked loudness level during an interval of concurrent playback in which the primary and secondary media items are both played back simultaneously on the electronic device, wherein the primary media item is associated with a plurality of loudness values corresponding to a plurality of respective discrete time samples of the primary media item, and wherein the time at which the concurrent playback interval begins is determined based on a time sample corresponding to the selection of an optimal loudness value from the plurality of loudness values.

2. The method of claim 1, wherein the ducking value is determined based at least partially upon the desired relative loudness difference, a loudness value associated with the primary media item, and a loudness value associated with the secondary media item.

3. The method of claim 2, wherein the loudness values associated with the primary and secondary media items are read from metadata information associated with the primary and secondary media item, respectively.

4. The method of claim 2, wherein the loudness values associated with the primary or the secondary media items are determined using RMS analysis, spectral analysis, cepstral analysis, linear prediction, analysis of dynamic range compression coefficients, an auditory model, or some combination thereof, prior to playback on the electronic device.

5. The method of claim 1, wherein selecting the optimal loudness value comprises:

analyzing a portion of the plurality of discrete time samples based on a defined future interval; and

selecting a loudness value within the future interval that minimizes the ducking value, wherein the time sample corresponding to the selected loudness value is used to determine the time at which the concurrent playback interval begins.

6. The method of claim 1, wherein ducking the primary media item comprises:

ducking in the primary media item prior to the concurrent playback interval; and

ducking out the primary media item following the concurrent playback interval.

7. The method of claim 6, wherein ducking in the primary media item comprises either fading out the primary media item to the ducked loudness level if the primary media item is currently in the process of being played back on the electronic device, or fading in the primary media item to the ducked loudness level if playback of the primary media item has not begun playback.

8. The method of claim 6, wherein ducking in and ducking out the primary media item is performed non-linearly.

9. The method of claim 6, wherein the rate at which the primary media item is ducked in and ducked out is variable depending on one or more characteristics of the primary media item.

10. The method of claim 1, wherein the secondary media item is a voice feedback announcement associated with the primary media item, and wherein the primary and secondary media item collectively comprise an enhanced media item.

11. The method of claim 1, wherein secondary media item is a system feedback announcement that is not associated with a particular media item, and wherein the interval of concurrent playback is initiated in response to the occurrence of a system event.

12. The method of claim 1, wherein ducking the primary media item comprises:

determining the genre of the primary media item; and

if the genre of the primary media file is substantially music data, ducking the primary media item based upon a first relative loudness difference, such that the first relative loudness difference is substantially maintained during an interval of concurrent playback, or else, if the genre of the primary media item is substantially speech data, ducking the primary media item based upon a second relative loudness difference, such that the second relative loudness difference is substantially maintained during the interval of concurrent playback, wherein the second relative loudness difference is greater than the first relative loudness difference.

13. The method of claim 12, wherein determining the genre of the primary media item comprises reading the genre information from metadata associated with the primary media item.

14. The method of claim 12, wherein determining the genre of the primary media item comprises using frequency analysis to determine the frequencies at which the audio data of the primary media item is most concentrated.

15. The method of claim 14, wherein the genre of the primary media item is determined to be substantially speech data if the audio data is generally concentrated within a frequency range of 1000-6000 hertz.

16. The method of claim 14, wherein determining the frequency analysis comprises spectral or cepstral analysis, or some combination thereof.

17. One or more tangible, non-transitory computer-readable storage media having instructions encoded thereon for execution by a processor, the instructions comprising:

a routine for selecting a primary media item for playback on an electronic device, the primary media item having an associated loudness value;

a routine for selecting a secondary media item for playback on the electronic device;

a routine for comparing the loudness value of the primary media item to a ducking threshold value; and

a routine for ducking one of the primary and secondary media items based upon the comparison, such that a desired relative loudness difference is substantially maintained during an interval of concurrent playback, wherein ducking one of the primary and secondary media items comprises ducking the primary media item if the loudness value is greater than the ducking threshold value, or else ducking the secondary media item if the loudness value is less than the ducking threshold value.

18. The one or more tangible, non-transitory computer-readable storage media of claim 17, wherein ducking the secondary media item comprises reducing the loudness level of the secondary media item while the primary media item is played at its associated loudness level during the concurrent playback interval.

19. An electronic device, comprising:

a processor;

a storage device configured to store a plurality of media items and their associated loudness values;

a memory device communicatively coupled to the processor and configured to store a media player application executable by the processor, wherein the media player application is configured to provide for the playback of one or more of the plurality of media items;

an audio processing circuit comprising;

a mixer configured to mix a plurality of audio input streams during an interval of concurrent playback to produce a composite mixed audio output stream, wherein the plurality of audio input streams includes a primary audio stream corresponding to a primary media item and a secondary audio stream corresponding to a secondary media item; and

audio ducking logic configured to duck the primary audio stream by a determined ducking value while the second media item played based upon a desired relative loudness difference, such that the relative loudness difference is substantially maintained during the concurrent playback interval, wherein the primary media item is associated with a plurality of loudness values corresponding to a plurality of respective discrete time samples of the primary media item, and wherein the audio ducking logic is configured to select an optimal time at which the concurrent playback interval begins by selecting an optimal loudness value from the plurality of loudness values; and

an audio output device configured to output the composite audio stream.

20. The electronic device of claim 19, wherein the ducking value is determined based at least partially upon the desired relative loudness difference, a loudness value associated with the primary media item, and a loudness value associated with the secondary media item.

21. The electronic device of claim 19, wherein the audio ducking logic is configured to read the loudness values from metadata associated with the primary and secondary media items.

22. The electronic device of claim 20, wherein the loudness values associated with the primary or the secondary media items are determined using RMS analysis, spectral analysis, cepstral analysis, linear prediction, analysis of dynamic range compression coefficients, an auditory model, or some combination thereof, prior to playback on the electronic device, and to associate the determined loudness values with the respective primary or secondary media item.

23. The electronic device of claim 22, comprising a network interface or a data interface, wherein the loudness are determined on an external device and received by the electronic device using either the network or data interface.

24. The electronic device of claim 20, wherein the audio ducking logic is configured to select the desired relative loudness difference is selected from first and second relative loudness difference values, and wherein the selection of the first or second relative loudness difference value is based at least partially upon genre information corresponding to the primary media item.

25. The electronic device of claim 24, wherein the audio ducking logic is configured to duck the primary media item based upon the first relative loudness difference if the genre of the primary media file is substantially music data, such that the first relative loudness difference is substantially maintained during the interval of concurrent playback, and to duck the primary media item based upon the second relative loudness difference if the genre of the primary media item is substantially speech data, such that the second relative loudness difference is substantially maintained during the interval of concurrent playback, and wherein the second relative loudness difference is greater than the first relative loudness difference.

26. The electronic device of claim 24, wherein the genre of the primary media item is determined by reading genre information from metadata associated with the primary media item.

27. The electronic device of claim 24, wherein in determining the genre of the primary media item, the audio processing circuit is configured to perform frequency analysis to determine the frequencies at which the audio data of the primary media item is most concentrated.

28. The electronic device of claim 27, wherein the genre of the primary media item is determined to be speech data if the audio data is generally concentrated within a frequency range from 1000-6000 hertz.

29. The electronic device of claim 19, wherein the audio ducking logic, in selecting the optimal loudness value, is configured to analyze a portion of the plurality of discrete time samples based on a defined future interval and to select a loudness value within the future interval that minimizes the ducking value, wherein the time sample corresponding to the selected loudness value is used by the audio ducking logic to determine the optimal time.

30. The electronic device of claim 19, wherein the primary media item comprises a music file, an audiobook, or a podcast, or some combination thereof, and wherein the secondary media item comprises a voice feedback announcement or a system feedback announcement.

31. The electronic device of claim 19, comprising a display device configured to display a graphical user interface associated with the media player application.

32. The electronic device of claim 31, wherein the user interface provides a user of the electronic device access to a plurality of configurable secondary media playback options.

33. The electronic device of claim 32, wherein the configurable secondary media playback options comprise enabling or disabling the playback of one or more types of secondary media items or adjusting the speed at which secondary media items are played back, or a combination thereof.

34. The electronic device of claim 19, wherein the electronic device is a portable digital media player.

35. A method, comprising:

selecting a primary media item for playback on an electronic device;

selecting a secondary media item for playback on the electronic device; and

ducking the primary media item by a ducking value while the second media item is played based upon a desired relative loudness difference, such that the relative loudness difference is substantially maintained and such that the primary media item is played at a ducked loudness level during an interval of concurrent playback in which the primary and secondary media items are both played back simultaneously on the electronic device, wherein ducking the primary media item comprises;

ducking in the primary media item prior to the concurrent playback interval; and

ducking out the primary media item following the concurrent playback interval, wherein the rate at which the primary media item is ducked in and ducked out is variable depending on one or more characteristics of the primary media item.