Scene change detection around a set of seed points in media data

• US 9,317,561 B2
• Filed: 12/15/2011
• Issued: 04/19/2016
• Est. Priority Date: 12/30/2010
• Status: Expired due to Fees

• Current Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby International AB (Dolby Laboratories Incorporated), Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Resch, Barbara, Radhakrishnan, Regunathan, Biswas, Arijit, Engdegard, Jonas
• Primary Examiner(s): Flanders, Andrew C
• Application Number:
  US13/997,860
• Publication Number:
  US 20130287214A1
• Time in Patent Office: 1,587 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 2210/061 for extraction of musical phrases, isolation of musically relevant segments, e.g. musical thumbnail generation, or for temporal structure analysis of a musical piece, e.g. determination of the movement sequence of a musical work
• CPC Class Codes: G10H 2240/151 Thumbnail, i.e. retrieving, playing or managing a short and musically relevant song preview from a library, e.g. the chorus thumbnail extraction, analysis phrases G10H2210/061
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308

Abstract

Techniques for scene change detection around seed points in media data are provided. Media features of many different types may be extracted from the media data. One or more statistical patterns of media features in a plurality of time-wise intervals around a plurality of seed time points of the media data may be determined using one or more types of features extractable from the media data. At least one of the one or more types of features comprises a type of features that captures structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources as related to the media data. A plurality of beginning scene change points and a plurality of ending scene change points in the media data may be detected, based on the one or more statistical patterns, for the plurality of seed time points in the media data.

Claims

1. A method for scene change detection in media data, comprising:

deriving a set of filtered values from the media data;

identifying a plurality of seed time points among time points at which the set of filtered values derived from the media data reach extremum values;

determining one or more statistical patterns of media features in a plurality of time-wise intervals around the plurality of seed time points of the media data using one or more types of features extractable from the media data, at least one of the one or more types of features comprising a type of features that captures structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources as related to the media data;

detecting, based on the one or more statistical patterns, a plurality of beginning scene change points and a plurality of ending scene change points in the media data for the plurality of seed time points in the media data;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the plurality of seed time points comprises at least one seed time point located inside a repeating section of the media data.

3. The method of claim 1, wherein the plurality of seed time points comprises at least one seed time point located inside a unique section of the media data.

4. The method of claim 1, wherein the plurality of seed time points comprises at least one seed time point located inside the loudest section of the media data.

5. The method of claim 1, wherein the plurality of seed time points comprises at least one seed time point located inside a section of the media data with the most of number of sound sources.

6. The method of claim 1, wherein the features extractable from the media data comprise one or more features of:

chroma, chroma difference, fingerprints, Mel-Frequency Cepstral Coefficient (MFCC), chroma-based fingerprints, rhythm pattern, energy, or other variants.

7. The method of claim 1, further comprising:

determining, based on one or more pairs of scene change points each individual pair in the one or more pairs being formed by an individual beginning scene change point in the plurality of beginning scene change points and an individual ending scene change point in the plurality of ending scene change points, a plurality of candidate representative segments in the media data.

8. The method of claim 7, wherein the plurality of candidate representative segments comprises a candidate representative segment defined or delimited with a pair in the individual pairs of scene change points.

9. The method of claim 7, wherein the plurality of candidate representative segments comprises a certain section of the media data that comprises one of chorus, verse, solo section, bridge section, intro.

10. The method of claim 7, wherein the plurality of candidate representative segments comprises at least one of:

highly repetitive segments, frequently repeated segments, intermediately repeated segments, or infrequently repeated segments.

11. The method of claim 7, wherein the plurality of candidate representative segments comprises a unique segment.

12. The method of claim 1, further comprising applying a model selection criterion to the one or more statistical patterns to detect scene changes in the media data.

13. The method of claim 1, wherein the one or more statistical patterns comprises at least a statistical pattern of media features based on one or more of:

chroma distance, distance of differential chroma features, loudness dynamics, MFCC, spectral contrast, spectral centroid, spectral bandwidth, spectral roll-off, spectral flatness, one or more rhythm patterns, energy, or one or more stereo parameters.

14. The method of claim 1, wherein at least one of the one or more types of features is used to form in part a digital representation of the media data.

15. The method of claim 1, wherein the features extractable from the media data are used to provide one or more digital representations of the media data based on one or more of:

chroma, chroma difference, fingerprints, Mel-Frequency Cepstral Coefficient (MFCC), chroma-based fingerprints, rhythm pattern, energy, or other variants.

16. The method of claim 1, wherein the features extractable from the media data are used to provide one or more digital representations relates to one or more of:

fast Fourier transforms (FFTs), digital Fourier transforms (DFTs), short time Fourier transforms (STFTs), Modified Discrete Cosine Transforms (MDCTs), Modified Discrete Sine Transforms (MDSTs), Quadrature Minor Filters (QMFs), Complex QMFs (CQMFs), discrete wavelet transforms (DWTs), or wavelet coefficients.

17. The method of claim 1, wherein the media data comprises one or more of:

songs, music compositions, scores, recordings, poems, audiovisual works, movies, or multimedia presentations.

18. The method of claim 1, further comprising deriving the media data from one or more of:

audio files, media database records, network streaming applications, media applets, media applications, media data bitstreams, media data containers, over-the-air broadcast media signals, storage media, cable signals, or satellite signals.

19. The method of claim 18, wherein the media data bitstreams comprise one or more of:

Advanced Audio Coding (AAC) bitstreams, High-Efficiency AAC bitstreams, MPEG-1/2 Audio Layer 3 (MP3) bitstreams, Dolby Digital (AC3) bitstreams, Dolby Digital Plus bitstreams, Dolby Pulse bitstreams, or Dolby TrueHD bitstreams.

20. A non-transitory computer readable storage medium, comprising a set of instructions, which when executed by a processing or computing device cause, control or program the device to execute or perform a process, wherein the process comprises the steps of:

deriving a set of filtered values from media data;

identifying a plurality of seed time points among time points at which the set of filtered values derived from the media data reach extremum values;

determining one or more statistical patterns of media features in a plurality of time-wise intervals around the plurality of seed time points of the media data using one or more types of features extractable from the media data, at least one of the one or more types of features comprising a type of features that captures structural properties, tonality including harmony and melody, timbre, rhythm, loudness, stereo mix, or a quantity of sound sources as related to the media data;

detecting, based on the one or more statistical patterns, a plurality of beginning scene change points and a plurality of ending scene change points in the media data for the plurality of seed time points in the media data.