Anchor person detection for television news segmentation based on audiovisual features

• US 20060288291A1
• Filed: 05/27/2005
• Published: 12/21/2006
• Est. Priority Date: 05/27/2005
• Status: Active Grant

• Current Assignee: MAVS Lab Incorporated
• Original Assignee: MAVS Lab Incorporated
• Inventors: Kuo, Chung-Chieh, Yeh, Chia-Hung, Shih, Hsuan-Huei, Lee, Shih-Hung
• Granted Patent: US 7,305,128 B2
• Time in Patent Office: Days
• US Class Current: 715/723
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A video segmentation method for segmenting video clips according to content of the video clips is disclosed. The method comprises scanning pixels of video frames with a first horizontal scan line to determine if colors of the pixels fall within a predetermined color range; creating a color map utilizing pixels located on the first horizontal scan line from a plurality of successive video frames; labeling the current video segment as a candidate video segment if the color map indicates the presence of a stable region of pixels falling within the predetermined color range for a predetermined number of successive video frames; and performing histogram color comparisons on the stable regions for detecting shot transitions. Audio signals of the video clips may also be analyzed to further verify the candidate video segments.

Claims

1. A video segmentation method for segmenting video clips according to content of the video clips, the method comprising:

receiving a video signal containing a plurality of video frames;

applying a first horizontal scan line to video frames of the video signal, the first horizontal scan line selecting at least one row of pixels for analysis;

analyzing the pixels of the video frame located on the first horizontal scan line to determine if colors of the pixels fall within a predetermined color range;

indicating regions of adjacent pixels in the video frame that fall within the predetermined color range;

creating a color map utilizing pixels located on the first horizontal scan line from a plurality of successive video frames;

labeling the current video segment as a candidate video segment if the color map indicates the presence of a stable region of pixels falling within the predetermined color range for a predetermined number of successive video frames;

for each candidate video segment, selecting one video frame out of every N video frames and creating a histogram of the stable region of each of the selected video frames;

performing a first histogram comparison between the histograms corresponding to pairs of successively selected video frames;

when a first histogram difference resulting from the first histogram comparison is greater than a first threshold value, performing a second histogram comparison on the stable regions of pairs of consecutive video frames located between the pair of successively selected video frames which resulted in the first histogram difference being greater than the first threshold value; and

indicating a shot change in the candidate video segment when the second histogram comparison yields a second histogram difference greater than a second threshold value.

2. The method of claim 1, wherein selecting one video frame out of every N video frames comprises selecting every Nth video frame.

3. The method of claim 1, wherein the first horizontal scan line is positioned at a location on the video frames approximately one-third of the distance down from the top of the video frames.

4. The method of claim 1, further comprising performing a RGB to Lab color transformation before analyzing the pixels of the video frame located on the first horizontal scan line to determine if colors of the pixels fall within the predetermined color range.

5. The method of claim 1, wherein the predetermined number of successive video frames is equivalent to three seconds of video.

6. The method of claim 1, further comprising:

applying a second horizontal scan line to the video frames of the video signal, wherein the first and second horizontal scan lines select the same number of rows for analysis;

assigning pixels of the video frame located on each of the first and second horizontal scan lines a logical value of “

1”

if the colors of the pixels fall within the predetermined color range;

performing a logical OR operation using corresponding pixels located on the first and second horizontal scan lines to create combined pixel data; and

utilizing the combined pixel data to indicate regions of adjacent pixels in the video frame that fall within the predetermined color range and to create the color map from the plurality of successive video frames.

7. The method of claim 6, wherein the first and second horizontal scan lines are positioned near a location on the video frames one-third of the distance down from the top of the video frames.

8. The method of claim 1, further comprising removing candidate video segment whose length is less than a predetermined period of time.

9. The method of claim 1, further comprising:

receiving an audio signal associated with the received video signal; and

analyzing the audio signal for filtering out candidate video segments, wherein the audio signal is processed in audio frames of a predetermined size.

10. The method of claim 9, further comprising transforming audio samples into the frequency domain for analyzing the frequency response of the audio frames and calculating a total sound energy level of the audio frames.

11. The method of claim 10, further comprising:

calculating a background sound energy level of the audio frames;

comparing the background sound energy level to the total sound energy level; and

eliminating the candidate video segment if a ratio of the background sound energy level to the total sound energy level is not within a first specified range.

12. The method of claim 11, further comprising:

calculating a ratio of the number of audio frames with a sound energy level below the background sound energy level to the total number of audio frames; and

eliminating the candidate video segment if the ratio of the number of audio frames with the sound energy level below the background sound energy level to the total number of audio frames is not within a second specified range.

13. The method of claim 10, further comprising:

calculating the average sound energy of audio frames having a frequency of 8-13 kHz;

calculating the ratio of the average sound energy of audio frames having a frequency of 8-13 kHz to the total sound energy level; and

eliminating the candidate video segment if the ratio of the average sound energy of audio frames having a frequency of 8-13 kHz to the total sound energy level is not within a specified range.

14. The method of claim 10, further comprising:

calculating the variance of frequency centroids for the current candidate segment; and

eliminating the candidate video segment if the variance of frequency centroids is not within a specified range.

15. A video segmentation method for segmenting video clips according to content of the video clips, the method comprising:

receiving a video signal containing a plurality of video frames;

receiving an audio signal associated with the received video signal;

applying first and second horizontal scan lines to video frames of the video signal, the first and second horizontal scan lines each selecting at least one row of pixels for analysis;

assigning pixels of the video frame located on each of the first and second horizontal scan lines a logical value of “

1”

if the colors of the pixels fall within a predetermined color range;

performing a logical OR operation using corresponding pixels located on the first and second horizontal scan lines to create combined pixel data;

utilizing the combined pixel data to indicate regions of adjacent pixels in the video frame that fall within the predetermined color range;

creating a color map utilizing the combined pixel data from a plurality of successive video frames;

labeling the current video segment as a candidate video segment if the color map indicates the presence of a stable region of pixels falling within the predetermined color range for a predetermined number of successive video frames;

for each candidate video segment, selecting one video frame out of every N video frames and creating a histogram of the stable region of each of the selected video frames;

performing a first histogram comparison between the histograms corresponding to pairs of successively selected video frames;

when a first histogram difference resulting from the first histogram comparison is greater than a first threshold value, performing a second histogram comparison on the stable regions of pairs of consecutive video frames located between the pair of successively selected video frames which resulted in the first histogram difference being greater than the first threshold value;

indicating a shot change in the candidate video segment when the second histogram comparison yields a second histogram difference greater than a second threshold value; and

analyzing the audio signal for filtering out candidate video segments, wherein the audio signal is processed in audio frames of a predetermined size.

16. The method of claim 15, wherein the first and second horizontal scan lines are positioned near a location on the video frames one-third of the distance down from the top of the video frames.

17. A video segmentation method for segmenting television news video clips according to detection of a news anchor in the video clips, the method comprising:

receiving a video signal containing a plurality of video frames of a news broadcast;

applying a first horizontal scan line to video frames of the video signal, the first horizontal scan line selecting at least one row of pixels for analysis;

analyzing the pixels of the video frame located on the first horizontal scan line to determine if colors of the pixels fall within a predetermined color range for detecting skin color of the news anchor;

indicating regions of adjacent pixels in the video frame that fall within the predetermined color range;

creating a color map utilizing pixels located on the first horizontal scan line from a plurality of successive video frames;

labeling the current video segment as a candidate video segment if the color map indicates the presence of a stable region of pixels falling within the predetermined color range for a predetermined number of successive video frames;

for each candidate video segment, selecting one video frame out of every N video frames and creating a histogram of the stable region of each of the selected video frames;

performing a first histogram comparison between the histograms corresponding to pairs of successively selected video frames;

when a first histogram difference resulting from the first histogram comparison is greater than a first threshold value, performing a second histogram comparison on the stable regions of pairs of consecutive video frames located between the pair of successively selected video frames which resulted in the first histogram difference being greater than the first threshold value; and

indicating a shot change in the candidate video segment when the second histogram comparison yields a second histogram difference greater than a second threshold value.