VIDEO SCENE DETECTION

• US 20140093164A1
• Filed: 10/01/2012
• Published: 04/03/2014
• Est. Priority Date: 10/01/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Noorkami, Maneli, Chan, Yi Linda
• Granted Patent: US 8,818,037 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

Systems and related methods for detecting a scene in video content are provided. In one example, a scene detection program identifies a plurality of shots in the video content and selects a target shot. The program builds a forward window including the target shot and having shots that are temporally ahead of the target shot, and a rearward window having shots that are temporally behind the target shot. For each of the shots in the forward window the program determines a dissimilarity between a selected shot in forward window and each of the other shots in the rearward window. If one or more of the dissimilarities is less than a scene boundary threshold, the program determines that the scene does not begin at the target shot. If none of the dissimilarities is less than the scene boundary threshold, the program determines that the scene begins at the target shot.

Claims

1. A method for detecting a scene in video content, comprising:

identifying a plurality of shots in the video content;

selecting a target shot in the plurality of shots;

building a forward window including the target shot and having a first number of the plurality of shots that are temporally ahead of the target shot;

building a rearward window having a second number of the plurality of shots that are temporally behind the target shot;

for each of the shots in the forward window, determining a dissimilarity between a selected shot and each of the other shots in the rearward window;

if one or more of the determined dissimilarities is less than a scene boundary threshold, then determining that the scene does not begin at the target shot; and

if none of the determined dissimilarities is less than the scene boundary threshold, then determining that the scene begins at the target shot.

2. The method of claim 1, wherein identifying a plurality of shots comprises identifying a gradual transition start frame in one or more of a plurality of frames in the video content, and further comprising:

if the target shot includes the gradual transition start frame, then determining that the scene begins at the target shot.

3. The method of claim 2, wherein identifying a gradual transition start frame further comprises:

for a target frame (i) in the plurality of frames, determining a cardinality of a first bin of a Y component color histogram of the target frame;

if the cardinality of the first bin is >

0.98, then;

determining a number GT_L of following frames that are temporally adjacent and forward of the target frame (i), where each of the following frames has a Y component color histogram with a first bin cardinality>

0.98; and

if a frame histogram similarity between a temporally adjacent previous frame (i−

1) and a temporally forward frame (i+GT_L+1) is less than a false positive similarity threshold, then determining that the target frame (i) is the gradual transition start frame and setting a length of a gradual transition to GT_L.

4. The method of claim 2, wherein identifying a gradual transition start frame further comprises:

for a target frame (i) in the plurality of frames;

calculating a first set of frame histogram differences between the target frame (i) and each of X consecutive following frames ending in an end frame (i+X), where X is a positive integer;

calculating a second set of frame histogram differences between the end frame (i+X) and the target frame (i) and between the end frame (i+X) and each of the consecutive following frames between the target frame and the end frame (i+X);

if

(a) the frame histogram differences in the first set are increasing towards the end frame (i+X), and

(b) the frame histogram differences in the second set are decreasing towards the end frame (i+X), and

(c) a false positive frame histogram difference between the target frame (i) and the end frame (i+X) is greater than a false positive difference threshold, and

(a), (b) and (c) are true for Y of the consecutive following frames, where Y is a positive integer, then

determining that the target frame (i) is the gradual transition start frame.

5. The method of claim 4, further comprising:

selecting a sample frame from among the target frame (i) and the X consecutive following frames; and

if a blur value of the sample frame is greater than a blur value threshold, then determining that the target frame (i) is the gradual transition start frame.

6. The method of claim 1, wherein identifying a plurality of shots further comprises:

calculating a frame histogram similarity between a first frame and a temporally adjacent second frame of the video content; and

using the frame histogram similarity to identify a hard cut transition between the first frame and the second frame.

7. The method of claim 1, wherein one shot of the first number of shots in the forward window is temporally adjacent to the target shot.

8. The method of claim 1, wherein the forward window and the rearward window have the same number of shots from the plurality of shots.

9. The method of claim 1, wherein determining a dissimilarity between a selected shot in the forward window and each of the other shots in the rearward window further comprises, for each of the shots in the forward window calculating a Euclidean distance between shot histograms of the selected shot in the forward window and each of the other shots in the rearward window.

10. The method of claim 1, further comprising:

calculating a frame histogram for each frame (i) in a plurality of frames in the video content;

using the frame histograms to identify a hard cut transition between a first frame and a temporally adjacent second frame (i) in the plurality of frames;

using the frame histograms to identify a gradual transition start frame (i) in the plurality of frames;

using the frame histograms to calculate shot histograms of each of the shots in the video content; and

using the shot histograms to determine that the scene begins at the target shot.

11. A scene detection system for detecting a scene in video content, comprising:

a computing device including a processor and memory;

a scene detection program executed by the processor using portions of the memory, the scene detection program configured to;

identify a plurality of shots in the video content;

select a target shot in the plurality of shots;

build a forward window including the target shot and having a first number of shots that are temporally ahead of the target shot;

build a rearward window having a second number of shots that are temporally behind the target shot;

for each of the shots in the forward window determine a dissimilarity between a selected shot and each of the other shots in the rearward window;

if one or more of the determined dissimilarities is less than a scene boundary threshold, then determine that the scene does not begin at the target shot; and

if none of the determined dissimilarities is less than the scene boundary threshold, then determine that the scene begins at the target shot.

12. The system of claim 11, wherein the scene detection program is further configured to:

identify a gradual transition start frame (i) in one or more of a plurality of frames in the video content, and

if the target shot includes the gradual transition start frame, then determine that the scene begins at the target shot.

13. The system of claim 12, wherein the scene detection program is further configured to:

for a target frame (i) in the plurality of frames, determine a cardinality of a first bin of a Y component color histogram of the target frame;

if the cardinality of the first bin is >

0.98, then;

determine a number GT_L of following frames that are temporally adjacent and forward of the target frame (i), where each of the following frames has a Y component color histogram with a first bin cardinality>

0.98; and

if a frame histogram similarity between a temporally adjacent previous frame (i−

1) and a temporally forward frame (i+GT_L+1) is less than a false positive similarity threshold, then determine that the target frame (i) is the gradual transition start frame and set a length of a gradual transition to GT_L.

14. The system of claim 12, wherein the scene detection program is further configured to:

for a target frame (i) in the plurality of frames;

calculate a first set of frame histogram differences between the target frame (i) and each of X consecutive following frames ending in an end frame (i+X), where X is a positive integer;

calculate a second set of frame histogram differences between the end frame (i+X) and the target frame (i) and between the end frame (i+X) and each of the consecutive following frames between the target frame and the end frame (i+X);

if

(a) the frame histogram differences in the first set are increasing towards the end frame (i+X), and

(b) the frame histogram differences in the second set are decreasing towards the end frame (i+X), and

(c) a false positive frame histogram difference between the target frame (i) and the end frame (i+X) is greater than a false positive difference threshold, and

(a), (b) and (c) are true for Y of the consecutive following frames, where Y is a positive integer, then

determine that the target frame (i) is the gradual transition start frame.

15. The system of claim 14, wherein the scene detection program is further configured to:

select a sample frame from among the target frame (i) and the X consecutive following frames; and

if a blur value of the sample frame is greater than a blur value threshold, then determine that the target frame (i) is the gradual transition start frame.

16. The system of claim 11, wherein the scene detection program is further configured to:

calculate a frame histogram similarity between a first frame and a temporally adjacent second frame of the video content; and

use the frame histogram similarity to identify a hard cut transition between the first frame and the second frame.

17. The system of claim 11, wherein the scene detection program is further configured to determine a dissimilarity between the selected shot in the forward window and each of the other shots in the rearward window by calculating a Euclidean distance between shot histograms of the selected shot in forward window and each of the other shots in the rearward window.

18. The system of claim 11, wherein the scene detection program is further configured to:

calculate a frame histogram for each frame (i) in a plurality of frames in the video content;

use the frame histograms to identify a hard cut transition between a first frame and a temporally adjacent second frame (i) in the plurality of frames;

use the frame histograms to identify a gradual transition start frame (i) in the plurality of frames;

use the frame histograms to calculate shot histograms of each shot in the video content; and

use the shot histograms to determine that the scene begins at the target shot.

19. A method for detecting a scene in video content, comprising:

analyzing each frame of a plurality of frames in the video content to identify a plurality of shots in the video content, wherein analyzing each frame of the plurality of frames comprises;

for a target frame (i) in the plurality of frames, determining a cardinality of a first bin of a Y component color histogram of the target frame;

if the cardinality of the first bin is >

0.98, then;

determining a number GT_L of following frames that are temporally adjacent and forward of the target frame (i), where each of the following frames has a Y component color histogram with a first bin cardinality>

0.98; and

if a frame histogram similarity between a temporally adjacent previous frame (i−

1) and a temporally forward frame (i+GT_L+1) is less than a false positive similarity threshold, then determining that the target frame (i) is a gradual transition start frame and setting a length of a gradual transition to GT_L;

selecting a target shot in the plurality of shots;

building a forward window including the target shot and having a first number of the plurality of shots that are temporally ahead of the target shot;

building a rearward window having a second number of shots that are temporally behind the target shot;

for each of the shots in the forward window, determining a dissimilarity between a selected shot and each of the other shots in the rearward window;

if one or more of the determined dissimilarities is less than a scene boundary threshold, then determining that the scene does not begin at the target shot; and

if none of the determined dissimilarities is less than the scene boundary threshold, then determining that the scene begins at the target shot.

20. The method of claim 19, further comprising:

for the target frame (i) in the plurality of frames;

calculating a first set of frame histogram differences between the target frame (i) and each of X consecutive following frames ending in an end frame (i+X), where X is a positive integer;

calculating a second set of frame histogram differences between the end frame (i+X) and the target frame (i) and between the end frame (i+X) and each of the consecutive following frames between the target frame and the end frame (i+X);

if

(a) the frame histogram differences in the first set are increasing towards the end frame (i+X), and

(b) the frame histogram differences in the second set are decreasing towards the end frame (i+X), and

(c) a false positive frame histogram difference between the target frame (i) and the end frame (i+X) is greater than a false positive difference threshold, and

(a), (b) and (c) are true for Y of the consecutive following frames, where Y is a positive integer, then

determining that the target frame (i) is the gradual transition start frame.