Ranking Representative Segments in Media Data

• US 20130289756A1
• Filed: 12/15/2011
• Published: 10/31/2013
• Est. Priority Date: 12/30/2010
• Status: Active Grant

• Current Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Resch, Barbara, Radhakrishnan, Regunathan, Biswas, Arijit, Engdegard, Jonas
• Granted Patent: US 9,313,593 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/061 for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
• CPC Class Codes: G10H 2240/151 Thumbnail, i.e. retrieving, playing or managing a short and musically relevant song preview from a library, e.g. the chorus
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308

Abstract

Techniques for ranking representative segments in media data are provided. Media features of many different types may be extracted from the media data. A plurality of ranking scores may be assigned to a plurality of candidate representative segments. Each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene in one or more statistical patterns in media features of the media data based on one or more types of features extractable from the media data. Each individual ranking score in the plurality of ranking scores may be assigned to an individual candidate representative segment in the plurality of candidate representative segments. A representative segment to be played to an end user may be selected from the candidate representative segments, based on the plurality of ranking scores.

Claims

1-35. -35. (canceled)

36. A method for ranking candidate representative segments within media data, comprising:

assigning a plurality of ranking scores to a plurality of candidate representative segments, each individual candidate representative segment in the plurality of candidate representative segments comprises at least one scene in one or more statistical patterns in media features of the media data based on one or more types of features extractable from the media data, each individual ranking score in the plurality of ranking scores being assigned to an individual candidate representative segment in the plurality of candidate representative segments;

selecting from the candidate representative segments, based on the plurality of ranking scores, a representative segment;

wherein the method is performed by one or more computing devices.

37. The method of claim 36, wherein each individual ranking score in the plurality of ranking scores comprises one or more component scores, at least one component score in the one or more component scores relates to at least one of structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources of the media data.

38. The method of claim 37, further comprising setting one or more weights for the one or more component ranking scores using one or more rules.

39. The method of claim 36, wherein each individual ranking score in the plurality of ranking scores comprises at least one component score based on one or more of:

duration, a measure for overlapping between different candidate representative segments, time-wise positions of candidate representative segments in the media data, chroma distance, MFCC, spectral contrast, spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, presence of singing voice, absence of singing voice, one or more rhythm patterns, energy, one or more stereo parameters, perceptual entropy, co-modulation, dynamics.

40. The method of claim 36, wherein an individual ranking score comprises one or more component scores, each being assigned one in a plurality of weight factors.

41. The method of claim 36, wherein an individual ranking score is obtained by using a probabilistic learning framework.

42. The method of claim 36, further comprising setting the plurality of weight factors using a probabilistic learning framework.

43. The method of claim 36, further comprising setting the plurality of weight factors using at least one statistical model.

44. The method of claim 36, further comprising setting the plurality of weight factors using one or more rules.

45. The method of claim 36, further comprising setting the plurality of weight factors using a boost algorithm.

46. The method of claim 36, wherein the representative segment corresponds to a candidate representative segment with a ranking score that is large in relation to other ranking scores assigned to other candidate representative segments in the plurality of candidate representative segments.

47. The method of claim 36, wherein at least one of the one or more types of features comprises a type of features that captures structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources as related to the media data.

48. The method of claim 36, wherein the features extractable from the media data are used to provide one or more digital representations of the media data based on one or more of:

chroma, chroma difference, fingerprints, Mel-Frequency Cepstral Coefficient (MFCC), chroma-based fingerprints, rhythm pattern, energy, or other variants.

49. The method of claim 36, wherein the features extractable from the media data are used to provide one or more digital representations relates to one or more of:

fast Fourier transforms (FFTs), digital Fourier transforms (DFTs), short time Fourier transforms (STFTs), Modified Discrete Cosine Transforms (MDCTs), Modified Discrete Sine Transforms (MDSTs), Quadrature Mirror Filters (QMFs), Complex QMFs (CQMFs), discrete wavelet transforms (DWTs), or wavelet coefficients.

50. The method of claim 36, further comprising deriving the media data from one or more of:

audio files, media database records, network streaming applications, media applets, media applications, media data bitstreams, media data containers, over-the-air broadcast media signals, storage media, cable signals, or satellite signals.

51. The method of claim 36, wherein the features extractable from the media data relate to a tuning system of 12 equal temperaments.

52. A method for determining a strong classifier based on features extracted from training media segments, comprising:

creating a set of feature vectors by extracting a plurality of features from a training set of media segments, each individual feature vector in the set of feature vectors comprises a plurality of features extracted from an individual segment in the plurality of media segment;

associating a set of reference truth labels with the training set of media segments, each individual segment in the training set of media segments is associated with an individual reference truth label in the set of reference truth labels;

calculating a plurality of sets of weak classifiers, each individual set of weak classifier in the plurality of sets of weak classifiers maps the set of features to a set of predicted truth labels;

determining a plurality of weights for the plurality of sets of weak classifiers, each individual set of weak classifiers in the plurality of sets of weak classifiers is given an individual weight in the plurality of weights;

creating, based on the plurality of sets of weak classifiers and the plurality of weights, a strong classifier that is to be applied to media data to rank and select a representative segment in a plurality of candidate representative segments;

wherein the method is performed by one or more computing devices.

53. The method of claim 52, wherein at least one feature in the plurality of features relates to at least one of:

structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources of a segment in the training set.

54. The method of claim 52, wherein at least one weak classifier in a set of weak classifiers in the plurality of sets of weak classifiers relates to one or more of:

duration, a measure for overlapping between different candidate representative segments, time-wise positions of candidate representative segments in the media data, chroma distance, MFCC, spectral contrast, spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, presence of singing voice, absence of singing voice, one or more rhythm patterns, energy, one or more stereo parameters, or perceptual entropy.

55. The method of claim 52, further comprising setting the plurality of weights using a probabilistic learning framework.

56. The method of claim 52, further comprising setting the plurality of weights using a boost algorithm.