Audio-based chapter detection in multimedia stream

• US 8,422,859 B2
• Filed: 03/23/2010
• Issued: 04/16/2013
• Est. Priority Date: 03/23/2010
• Status: Active Grant

• Current Assignee: Pixelworks Incorporated
• Original Assignee: VIXS Systems Incorporated (Pixelworks Incorporated)
• Inventors: Zeng, Hong
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US12/729,378
• Publication Number:
  US 20110235993A1
• Time in Patent Office: 1,120 Days
• Field of Search: 386/241, 386/249, 386/250, 386/251, 386/285
• US Class Current: 386/241
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence

Abstract

A multimedia processing system identifies chapter breaks in a program represented by multimedia data through an analysis of audio content of a portion of the multimedia data so as to identify an interval that potentially represents a chapter break. This audio analysis can include an analysis to identify changes in high frequency edges in the audio content, an analysis to identify changes in the total energy in a central frequency band of the audio content, an analysis to identify patterns of sequentially repeating values in the audio content, an analysis to identify changes in bitrate, or some combination thereof. One or more variances in the audio information at (e.g., before) the start of the interval and the audio information at (e.g., after) the end of the interval then may be used to identify or otherwise characterize the interval as representing an actual chapter break. Further, a chapter represented by two consecutive chapter breaks can be identified as an advertisement based on the duration between the two consecutive chapter breaks and thus the multimedia processing device can implement a “commercial skip function” by omitting playback of the portion of the multimedia data representing the chapter responsive to the chapter being identified as an advertisement.

Claims

1. In a multimedia processing device, a method comprising:

identifying an interval in a multimedia stream as potentially representing a chapter break in a program represented by the multimedia stream; and

characterizing the interval as representing an actual chapter break responsive to at least one variance in audio data of the multimedia stream between a point at a start of the interval and a point at an end of the interval, the at least one variance comprising at least one of;

a variance in average bitrate of the audio data; and

a variance in average highest frequency of the audio data.

2. The method of claim 1, wherein identifying the interval comprises:

identifying the start of the interval as a first time corresponding to a point at which a total energy of a central frequency band of the audio data falls below a first threshold, the central frequency band being associated with a human speech frequency range; and

identifying the end of the interval as a second time corresponding to a point at which the total energy of the central frequency band of the audio data subsequently rises above a second threshold, the second threshold greater than the first threshold.

3. The method of claim 1, wherein identifying the interval comprises:

identifying the start of the interval as a first time corresponding to a point at which a difference between a high frequency edge of the audio data and a corresponding average high frequency edge of the audio data exceeds a first threshold; and

identifying the end of the interval as a second time corresponding to a point at which a difference between a subsequent high frequency edge of the audio data and a subsequent corresponding average high frequency edge of the audio data falls below a second threshold.

4. The method of claim 1, wherein identifying the interval comprises:

determining a sequence of audio frames greater than a threshold number of audio frames, each audio frame of the sequence comprising a byte value that is sequentially repeated in the audio frame at least a threshold number of times;

identifying the start of the interval as a first time corresponding to an initial audio frame of the sequence; and

identifying the end of the interval as a second time corresponding to a final audio frame of the sequence.

5. The method of claim 1, wherein characterizing the interval as representing an actual chapter break comprises characterizing the interval as representing an advertisement chapter break further responsive to a duration between the interval and another chapter break that precedes or follows the interval relative to a playback sequence of the multimedia stream.

6. The method of claim 5, further comprising:

responsive to characterizing the interval as representing an advertisement chapter break, marking an initial intra-coded frame of the multimedia stream that falls within the interval as a start of an advertisement and marking as an end of the advertisement a final intra-coded frame of the multimedia stream that falls within a following interval that is characterized as representing an advertisement chapter break.

7. The method of claim 1, wherein characterizing the interval as representing an actual chapter break comprises characterizing the interval as representing an actual chapter break responsive to the variance in average bitrate and the variance in average high frequency edge.

8. The method of claim 1, wherein characterizing the interval as representing an actual chapter break comprises characterizing the interval as representing an actual chapter break further responsive to a number of sequentially repeating values in a portion of the audio data corresponding to the interval.

9. The method of claim 1, wherein characterizing the interval as representing an actual chapter break comprises characterizing the interval as representing an actual chapter break responsive to a number of sequentially repeating values in a portion of the audio data corresponding to the interval, responsive to the variance in average bitrate, and responsive to the variance in average high frequency edge.

10. The method of claim 1, wherein characterizing the interval as representing an actual chapter break characterizing the interval as representing an actual chapter break further responsive to a duration of the interval.

11. The method of claim 1, wherein characterizing the interval as representing an actual chapter break comprises characterizing the interval as representing an actual chapter break based on a weighted sum of a value representing the variance in average bitrate, a value representing the variance in average highest frequency, a value representing a duration of the interval, and value representing a proportion of a number of repeating values in a portion of the audio data corresponding to the interval to a total number of values in the portion of the audio data.

12. A multimedia processing device comprising:

a chapter detection module to identify an interval in a multimedia stream as potentially representing a chapter break in a program represented by the multimedia stream and to characterize the interval as representing an actual chapter break responsive to at least one variance in audio data of the multimedia stream between a point at a start of the interval and a point at the end of the interval, the at least one variance comprising at least one of;

a variance in average bitrate of the audio data; and

a variance in average highest frequency of the audio data.

13. The multimedia processing device of claim 12, wherein the multimedia processing device is to identify the interval based on at least one of:

an analysis of total energies of in a central frequency band of the audio data, the central frequency band being associated with a human speech frequency range;

an analysis of highest frequency edges of the audio data; and

an analysis of sequentially repeating values in the audio data.

14. The multimedia processing device of claim 12, wherein the multimedia processing device is to characterize the interval as representing an actual chapter break further responsive to a number of sequentially repeating values in a portion of the audio data corresponding to the interval.

15. The multimedia processing device of claim 12, wherein the chapter detection module further is to characterize the interval as representing an advertisement chapter break responsive to a duration between the interval and another chapter break that precedes or follows the interval relative to a playback sequence of the multimedia stream.

16. The multimedia processing device of claim 15, further comprising:

a storage device to store the multimedia stream; and

wherein, in response to characterizing the interval as representing an advertisement chapter break, the chapter detection module is to store chapter information to the storage device in association with the multimedia stream, the chapter information identifying a first portion of the multimedia stream corresponding to the interval as an advertisement chapter break.

17. The multimedia processing device of claim 16, further comprising:

a display processing module to process the multimedia stream for playback, wherein the display processing module is to omit playback of multimedia content represented by a second portion of the multimedia stream responsive to the chapter information, the second portion of the multimedia stream following the first portion of the multimedia stream relative to the playback sequence.

18. In a multimedia processing device, a method comprising:

identifying a first interval of a multimedia stream as potentially representing a first chapter break based on an analysis of audio data of the multimedia stream;

identifying a second interval of the multimedia stream as potentially representing a second chapter break based on the analysis of audio data, the second interval following the first interval in a playback sequence of the multimedia stream;

characterizing the first interval as a first advertisement chapter break based on a duration of the first interval, a variance in an average bit rate of audio data of the multimedia stream at a start of the first interval and an average bit rate of audio data of the multimedia stream at an end of the first interval, a variance in an average high frequency edge of audio data at the start of the first interval and an average high frequency edge of audio data at the end of the first interval, and a number of sequentially repeating values in the audio data for the first interval; and

characterizing the second interval as a second advertisement chapter break based on a duration of the second interval, a variance in an average bit rate of audio data at a start of the second interval and an average bit rate of audio data at an end of the second interval, a variance in an average high frequency edge of audio data at the start of the second interval and an average high frequency edge of audio data at the end of the second interval, and a number of sequentially repeating values in the audio data for the second interval.

19. The method of claim 18, further comprising:

omitting playback of a portion of the multimedia stream that is between the first interval and the second interval in the playback sequence responsive to characterizing the first interval as the first advertisement chapter break and characterizing the second interval as the second advertisement chapter break.

20. The method of claim 18, wherein the first and second intervals are identified as potentially representing chapter breaks based on at least one of:

an analysis of a total energies of in a central frequency band of the audio data, the central frequency band being associated with a human speech frequency range;

an analysis of highest frequency edges of the audio data; and

an analysis of percentages of sequentially repeating values in the audio data.