Ranking representative segments in media data

• US 9,313,593 B2
• Filed: 12/15/2011
• Issued: 04/12/2016
• Est. Priority Date: 12/30/2010
• Status: Expired due to Fees

• Current Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Resch, Barbara, Radhakrishnan, Regunathan, Biswas, Arijit, Engdegard, Jonas
• Primary Examiner(s): KUNTZ, CURTIS A
• Assistant Examiner(s): Zhu, Qin
• Application Number:
  US13/997,866
• Publication Number:
  US 20130289756A1
• Time in Patent Office: 1,580 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/061 for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
• CPC Class Codes: G10H 2240/151 Thumbnail, i.e. retrieving, playing or managing a short and musically relevant song preview from a library, e.g. the chorus thumbnail extraction, analysis phrases G10H2210/061
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308

Abstract

Techniques for ranking representative segments in media data are provided. Media features of many different types may be extracted from the media data. A plurality of ranking scores may be assigned to a plurality of candidate representative segments. Each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene in one or more statistical patterns in media features of the media data based on one or more types of features extractable from the media data. Each individual ranking score in the plurality of ranking scores may be assigned to an individual candidate representative segment in the plurality of candidate representative segments. A representative segment to be played to an end user may be selected from the candidate representative segments, based on the plurality of ranking scores.

Claims

1. A method for ranking candidate representative segments within media data, comprising:

creating one or more media fingerprints each of which comprises a plurality of hash bits generated from the media data;

extracting features from the media data;

detecting a plurality of scenes within the media data based at least in part on the one or more media fingerprints and a distance analysis for the features extracted from the media data;

assigning a plurality of ranking scores to a plurality of candidate representative segments in the media data, each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene of the plurality of scenes in the media data, each individual ranking score in the plurality of ranking scores being assigned to an individual candidate representative segment in the plurality of candidate representative segments;

selecting from the plurality of candidate representative segments, based on the plurality of ranking scores, a representative segment;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein each individual ranking score in the plurality of ranking scores comprises one or more component scores, at least one component score in the one or more component scores relates to at least one of structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources of the media data.

3. The method of claim 2, further comprising setting one or more weights for the one or more component ranking scores using one or more rules.

4. The method of claim 1, wherein each individual ranking score in the plurality of ranking scores comprises at least one component score based on one or more of:

duration, a measure for overlapping between different candidate representative segments, time-wise positions of candidate representative segments in the media data, chroma distance, MFCC, spectral contrast, spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, presence of singing voice, absence of singing voice, one or more rhythm patterns, energy, one or more stereo parameters, perceptual entropy, co-modulation, or dynamics.

5. The method of claim 1, wherein an individual ranking score comprises one or more component scores, each being assigned a weight factor in a plurality of weight factors.

6. The method of claim 1, wherein an individual ranking score is obtained by using a probabilistic learning framework.

7. The method of claim 5, further comprising setting the plurality of weight factors using a probabilistic learning framework.

8. The method of claim 5, further comprising setting the plurality of weight factors using at least one statistical model.

9. The method of claim 5, further comprising setting the plurality of weight factors using one or more rules.

10. The method of claim 5, further comprising setting the plurality of weight factors using a boost algorithm.

11. The method of claim 1, wherein the representative segment corresponds to a candidate representative segment with a ranking score that is large in relation to other ranking scores assigned to other candidate representative segments in the plurality of candidate representative segments.

12. The method of claim 1, wherein the features extracted from the media data are used to provide one or more digital representations of the media data based on one or more of:

chroma, chroma difference, fingerprints, Mel-Frequency Cepstral Coefficient (MFCC), chroma-based fingerprints, rhythm pattern, energy, or other variants.

13. The method of claim 1, wherein the features extracted from the media data are used to provide one or more digital representations relates to one or more of:

fast Fourier transforms (FFTs), digital Fourier transforms (DFTs), short time Fourier transforms (STFTs), Modified Discrete Cosine Transforms (MDCTs), Modified Discrete Sine Transforms (MDSTs), Quadrature Mirror Filters (QMFs), Complex QMFs (CQMFs), discrete wavelet transforms (DWTs), or wavelet coefficients.

14. The method of claim 1, further comprising deriving the media data from one or more of:

audio files, media database records, network streaming applications, media applets, media applications, media data bitstreams, media data containers, over-the-air broadcast media signals, storage media, cable signals, or satellite signals.

15. The method of claim 1, wherein the features extracted from the media data relate to a tuning system of 12 equal temperaments.

16. A non-transitory computer readable storage medium, comprising a set of instructions, which when executed by a processing or computing device cause, control or program the device to execute or perform a process, wherein the process comprises the steps of:

creating one or more media fingerprints each of which comprises a plurality of hash bits generated from media data;

extracting features from the media data;

detecting a plurality of scenes within the media data based at least in part on the one or more media fingerprints and a distance analysis for the features extracted from the media data;

assigning a plurality of ranking scores to a plurality of candidate representative segments in the media data, each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene of the plurality of scenes in the media data, each individual ranking score in the plurality of ranking scores being assigned to an individual candidate representative segment in the plurality of candidate representative segments;

selecting from the plurality of candidate representative segments, based on the plurality of ranking scores, a representative segment.

17. The non-transitory computer readable storage medium of claim 16, wherein each individual ranking score in the plurality of ranking scores comprises one or more component scores, at least one component score in the one or more component scores relates to at least one of structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources of the media data.

18. The non-transitory computer readable storage medium of claim 17, wherein the process further comprises setting one or more weights for the one or more component ranking scores using one or more rules.

19. The non-transitory computer readable storage medium of claim 16, wherein each individual ranking score in the plurality of ranking scores comprises at least one component score based on one or more of:

duration, a measure for overlapping between different candidate representative segments, time-wise positions of candidate representative segments in the media data, chroma distance, MFCC, spectral contrast, spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, presence of singing voice, absence of singing voice, one or more rhythm patterns, energy, one or more stereo parameters, perceptual entropy, co-modulation, or dynamics.

20. The non-transitory computer readable storage medium of claim 16, wherein an individual ranking score comprises one or more component scores, each being assigned a weight factor in a plurality of weight factors.

21. The non-transitory computer readable storage medium of claim 16, wherein an individual ranking score is obtained by using a probabilistic learning framework.

22. The non-transitory computer readable storage medium of claim 20, wherein the process further comprises setting the plurality of weight factors using a probabilistic learning framework.

23. The non-transitory computer readable storage medium of claim 20, wherein the process further comprises setting the plurality of weight factors using at least one statistical model.

24. The non-transitory computer readable storage medium of claim 20, wherein the process further comprises setting the plurality of weight factors using one or more rules.

25. The non-transitory computer readable storage medium of claim 20, wherein the process further comprises setting the plurality of weight factors using a boost algorithm.

26. The non-transitory computer readable storage medium of claim 16, wherein the representative segment corresponds to a candidate representative segment with a ranking score that is large in relation to other ranking scores assigned to other candidate representative segments in the plurality of candidate representative segments.

27. The non-transitory computer readable storage medium of claim 16, wherein the features extracted from the media data are used to provide one or more digital representations of the media data based on one or more of:

chroma, chroma difference, fingerprints, Mel-Frequency Cepstral Coefficient (MFCC), chroma-based fingerprints, rhythm pattern, energy, or other variants.

28. The non-transitory computer readable storage medium of claim 16, wherein the features extracted from the media data are used to provide one or more digital representations relates to one or more of:

fast Fourier transforms (FFTs), digital Fourier transforms (DFTs), short time Fourier transforms (STFTs), Modified Discrete Cosine Transforms (MDCTs), Modified Discrete Sine Transforms (MDSTs), Quadrature Mirror Filters (QMFs), Complex QMFs (CQMFs), discrete wavelet transforms (DWTs), or wavelet coefficients.

29. The non-transitory computer readable storage medium of claim 16, wherein the process further comprises deriving the media data from one or more of:

audio files, media database records, network streaming applications, media applets, media applications, media data bitstreams, media data containers, over-the-air broadcast media signals, storage media, cable signals, or satellite signals.

30. The non-transitory computer readable storage medium of claim 16, wherein the features extracted from the media data relate to a tuning system of 12 equal temperaments.