Method and apparatus for shot detection

• US 6,870,956 B2
• Filed: 06/14/2001
• Issued: 03/22/2005
• Est. Priority Date: 06/14/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Dong, Qi, Wei
• Primary Examiner(s): Miriam, Daniel
• Application Number:
  US09/882,787
• Publication Number:
  US 20020196974A1
• Time in Patent Office: 1,377 Days
• Field of Search: 382/168, 382/170, 382/236, 382/239, 382/268, 382/270, 382/272, 382/274, 382/275, 382/305, 348/700
• US Class Current: 382/170
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A system identifies an abrupt transition in content between two frames, and determines whether the abrupt transition was caused by a shot boundary between the two frames or by a flashlight event. Identification of the abrupt transition in content includes calculating a difference in light intensity histograms between the current frame and a preceding frame, and comparing the histogram difference to a dynamically determined threshold. Further, an average light intensity based method is used to determine whether the abrupt transition was caused by a shot boundary or by a flashlight event.

Claims

1. A method comprising;

identifying an abrupt transition in average light intensity between two frames, wherein identifying an abrupt transition comprises;

calculating a difference in light intensity histograms between the current frame and a preceding frame; and

comparing the histogram difference to a dynamically determined threshold, wherein an abrupt transition is indicated if the histogram difference exceeds the threshold, and wherein dynamically determining the threshold comprises;

calculating an average and standard deviation value of the histogram difference in a sliding window of frames;

generating one or more thresholds as multiples of the calculated average; and

determining whether the calculated standard deviation falls below a standard deviation threshold before adopting the generated one or more thresholds; and

determining whether the abrupt transition was caused by a shot boundary between the two frames or by a flashlight event.

2. A method according to claim 1, wherein generating one or more thresholds comprises:

calculating a high threshold that is four- to five-times the calculated average; and

calculating a low average that is two- to three-limes the calculated average.

3. A method according to claim 2, wherein the high threshold is used to determine whether an abrupt transition has occurred.

4. A method according to claim 2, wherein the low threshold is used to indicate whether a gradual transition has occurred, if the histogram difference does not exceed the high threshold.

5. A method according to claim 4, further comprising:

comparing the histogram difference to the low threshold;

accumulating the histogram difference of two or more frames if the histogram difference exceeds the low threshold; and

determining that a gradual transition has occurred if the accumulated histogram difference exceeds the high threshold.

6. A method according to claim 1, wherein determining a cause of the abrupt transition comprises:

calculating a difference in average minimal intensity value between a window of frames preceding the current frame and a window of frames subsequent to the current frame;

generating a ratio of the average intensity change of the current frame to the calculated difference in average minimal intensity values; and

concluding that the abrupt transition is caused by a flashlight event if the ratio does not exceed a threshold.

7. A method according to claim 6, wherein calculating a difference in average minimal intensity values comprises:

identifying a window of frames on either side of the current frame;

generating an average minimal intensity value for each window from a first and second minimal intensity value of each frame, respectively; and

calculating the difference between the generated average minimal intensity values.

8. A method according to claim 6, wherein the threshold is equal to one (1), wherein a ratio that deviates from one indicates that the abrupt transition is due to a shot cut, while a ratio close to one indicates that the abrupt transition is due to a flashlight event.

9. A computer readable medium storing a plurality of executable instructions which, when executed, implement a method according to claim 1.

10. A media analysis agent comprising:

a shot boundary detector to statistically analyze one or more attributes associated with content in video frames to detect abrupt and gradual transitions in the video content indicative of a shot boundary; and

a flashlight detector, responsive to the shot boundary detector, to distinguish abrupt transitions in the video content caused by flashlight events from those caused by actual shot boundaries, to reduce false-positive identification of flashlight events as shot boundaries; and

an adaptive threshold selection module, responsive to the shot boundary detector, to dynamically set one or more thresholds for use in shot boundary identification based, at least in part, on one or more attributes of video content, wherein the adaptive threshold selection module calculates an average and standard deviation of a histogram difference in a sliding window of frames surrounding the current frame, and generates a proposed low threshold value (Ts) and a proposed high threshold value (Tb) from the average histogram difference, and wherein the proposed low threshold is two- to three-times the calculated average histogram difference, while the proposed high threshold is four- to five-times the calculated average histogram difference.

11. A media analysis agent according to claim 10, wherein the adaptive threshold selection module certifies the proposed thresholds if the calculated standard deviation does not reach a standard deviation threshold.

12. A media analysis agent according to claim 11, wherein if the adaptive threshold selection module cannot certify the proposed thresholds, prior thresholds are used by shot boundary detector.

13. A media analysis agent according to claim 10, wherein the high threshold is used to identity an abrupt transition, while the low threshold is used to indicate a potential gradual transition.

14. A media analysis agent according to claim 13, wherein shot boundary detector compares the histogram difference to a low threshold if the difference does not exceed the high threshold, and accumulates the histogram differences of frames wherein the difference exceeds the low threshold.

15. A media analysis agent according to claim 14, wherein shot boundary detector identifies a gradual transition when the accumulated histogram differences exceed the high threshold.

16. A media analysis agent according to claim 10, wherein the shot boundary detector calculates a difference in light intensity histograms between a current frame and a preceding frame, and compares the histogram difference to a dynamically determined threshold, wherein an abrupt transition is identified by a histogram difference that exceeds the threshold.

17. A media analysis agent according to claim 16, wherein shot boundary detector invokes an instance of flashlight detector to determine whether the abrupt transition is the result of a flashlight event.

18. A media analysis agent according to claim 10, wherein flashlight detector calculates a difference in average minimal intensity values between a first sliding window preceding the current frame, and a second sliding window subsequent to the current frame.

19. A media analysis agent according to claim 18, wherein flashlight detector calculates the average intensity change of the current frame, and generates a ratio of the difference in average minimal intensity values to the average intensity change of the current frame.

20. A media analysis agent according to claim 19, wherein flashlight detector determines that the abrupt change was due to a flashlight event if the ratio does not exceed a threshold value.

21. A media analysis agent according to claim 20, wherein the threshold value is one (1).

22. A media analysis agent according to claim 20, wherein the flashlight detector concludes that the abrupt transition was caused by a shot boundary if the ratio exceeds the threshold value.

23. A computing system comprising:

a memory, to receive media content; and

a media processing system, coupled to the memory device, including a media analysis agent according to claim 10 to segment the received media content into uniquely identifiable shots.

24. One or more computer-readable media having stored thereon a computer program that, when executed by one or more processors, causes the one or more processors to;

calculate a difference in light intensity histograms between two frames;

compare the histogram difference to a dynamically determine threshold, wherein an abrupt transition is indicated if the histogram difference exceeds the threshold, wherein to determine the dynamic threshold, the one or more processors;

calculate an average and standard deviation value of the histogram difference in a sliding window of frames;

generate one or more thresholds as multiples of the calculated average; and

determine whether the calculated standard deviation falls below a standard deviation threshold before adopting the generated one or more thresholds; and

determine whether the abrupt transition was caused by a shot boundary between the two frames or by a flashlight event.

25. One or more computer-readable media as recited in claim 24 wherein to determine a cause of the abrupt transition, the one or more processors:

calculate a difference in average minimal intensity values between a window of frames preceding the current frame and a window of frames subsequent to the current frame;

generate a ratio of the average intensity change of the current frame to the calculated difference in average minimal intensity values; and

conclude that the abrupt transition is caused by a flashlight event if the ratio does not exceed a threshold.

26. One or more computer-readable media as recited in claim 25 wherein the threshold is equal to one, wherein a ration that deviates from one indicates that the abrupt transition is due to a shot cut, while a ratio close to one indicates that the abrupt transition is due to a flashlight event.