Detection of transitions in video sequences

• US 5,990,980 A
• Filed: 05/28/1998
• Issued: 11/23/1999
• Est. Priority Date: 12/23/1997
• Status: Expired due to Term

• Current Assignee: MediaTek, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Golin, Stuart Jay
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/086,373
• Time in Patent Office: 544 Days
• Field of Search: 348/699-701, 348/416, 345/328
• US Class Current: 348/700
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/60 using transform coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

Frame dissimilarity measure (FDM) values are generated for pairs of frames in a video sequence that are separated by a specified timing window size, where each FDM value is the ratio of a net dissimilarity measure and a cumulative dissimilarity measure. In one embodiment, a first threshold condition is applied to identify peaks in the FDM data that could correspond to transitions between shots in a video sequence. A second threshold condition is applied to the FDM data adjacent to the detected peaks to identify frames at the start and end of the transition. The first threshold condition determines whether the FDM values exceed a first specified threshold level, while the second threshold condition determines whether the FDM values fall below a second specified threshold level. The net and cumulative dissimilarity measures may be based on histograms for the two frames. The present invention is effective in detecting gradual transitions between shots, such as wipes, dissolves, and fades, as well as abrupt transitions, such as cuts.

Claims

1. A method for identifying transitions in a video sequence, comprising the steps of:

(a) generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size, wherein the FDM value is the ratio of a net dissimilarity measure and a cumulative dissimilarity measure; and

(b) analyzing the FDM values to identify transitions in the video sequence.

2. The invention of claim 1, wherein, for each pair of frames m and n, the FDM value is generated from histograms for two or more frames corresponding to the pair of frames m and n.

3. The invention of claim 1, wherein:

the net dissimilarity measure D_net (m,n) is defined by;

space="preserve" listing-type="equation">D.sub.net (m,n).tbd.Σ

.sub.i |h.sub.i (m)-h.sub.i (n)|.sup.γ

wherein;

h₁ (m) is the i^th bin of a histogram corresponding to frame m;

h_i (n) is the is bin of a histogram corresponding to frame n; and

γ

is a specified parameter; and

the cumulative dissimilarity measure D_net (m,n) is defined by;

space="preserve" listing-type="equation">D.sub.cum (m,n).tbd.Σ

.sup.n-1.sub.k-m D.sub.net (k,k+1).

4. The invention of claim 1, further comprising the step of comparing the relative positions in the video sequence of start and end frames for each identified transition to discriminate against short-term anomalies in the video sequence.

5. The invention of claim 1, wherein step (b) comprises the steps of:

(1) applying a first threshold condition to an FDM value to determine if a frame in the corresponding pair of frames corresponds to a transition; and

(2) if the first threshold condition is satisfied, then applying a second threshold condition to one or more other FDM values to identify start and end frames for the transition.

6. The invention of claim 5, further comprising the step of characterizing noise in the FDM values for the video sequence, wherein the first and second threshold conditions are functions of the noise.

7. The invention of claim 6, wherein the noise is characterized by:

(1) biasing the FDM values toward small values; and

(2) filtering the biased FDM values.

8. The invention of claim 6 wherein:

the first threshold condition is satisfied when the FDM value for a frame n exceeds the maximum of a first threshold value T₁ and the product of a first threshold multiplier M₁ and the noise for frame n; and

the second threshold condition is satisfied when the FDM value for a frame m falls below the maximum of a second threshold value T₂ and the product of a second threshold multiplier M₂ and the noise for frame m.

9. The invention of claim 5, wherein the FDM values are filtered before applying the first and second threshold conditions.

10. The invention of claim 5, wherein:

the first threshold condition is satisfied when an FDM value is greater than a first threshold level; and

the second threshold condition is satisfied when an FDM value is less than a second threshold level.

11. The invention of claim 10, wherein the first threshold level is greater than the second threshold level.

12. The invention of claim 5, wherein:

the first threshold condition is used to identify peaks in the FDM values; and

the second threshold condition is used to determine boundaries for the peaks.

13. The invention of claim 1, wherein each frame is divided into two or more sub-frames and an FDM value is generated for each sub-frame in order to distinguish object motion from transitions between shots.

14. The invention of claim 1, wherein step (b) comprises the steps of:

(1) applying a first threshold condition to an FDM value to determine if a frame in the corresponding pair of frames corresponds to a start frame for a possible transition;

(2) if the first threshold condition is satisfied, then applying a second threshold condition to one or more other FDM values to determine if the possible transition is a probable transition; and

(3) if the second threshold condition is satisfied, then applying the first threshold condition to one or more other FDM values to determine an end frame for the probable transition.

15. An apparatus for identifying transitions in a video sequence, comprising:

(a) means for generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size, wherein the FDM value is the ratio of a net dissimilarity measure and a cumulative dissimilarity measure; and

(b) means for analyzing the FDM values to identify transitions in the video sequence.

16. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to implement a method for identifying transitions in a video sequence, the method comprising the steps of:

(a) generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size, wherein the FDM value is the ratio of a net dissimilarity measure and a cumulative dissimilarity measure; and

(b) analyzing the FDM values to identify transitions in the video sequence.

17. A method for identifying transitions in a video sequence, comprising the steps of:

(a) generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size;

(b) applying a first threshold condition to an FDM value to determine if the corresponding pair of frames corresponds to a transition; and

(c) if the first threshold condition is satisfied, then applying a second threshold condition to one or more other FDM values to identify start and end frames for the transition.

18. The invention of claim 17, wherein the FDM value is the ratio of a net dissimilarity measure and a cumulative dissimilarity measure, wherein:

the net dissimilarity measure D_net (m,n) is defined by;

space="preserve" listing-type="equation">D.sub.net (m,n).tbd.Σ

.sub.i |h.sub.i (m)-h.sub.i (n)|.sup.γ

wherein;

h_i (m) is the i^th bin of a histogram corresponding to frame m;

h_i (n) is the i^th bin of a histogram corresponding to frame n; and

γ

is a specified parameter; and

the cumulative dissimilarity measure D_cum (m,n) is defined by;

space="preserve" listing-type="equation">D.sub.com (m,n).tbd.Σ

.sup.n-1.sub.k=m D.sub.net (k,k+1).

19. The invention of claim 17, further comprising the step of comparing the relative positions of the start and end frames in the video sequence to discriminate against short-term anomalies in the video sequence.

20. The invention of claim 17, further comprising the step of characterizing noise in the FDM values for the video sequence, wherein the first and second threshold conditions are functions of the noise.

21. The invention of claim 17, wherein:

the first threshold condition is satisfied when an FDM value is greater than a first threshold level to identify peaks in the FDM values; and

the second threshold condition is satisfied when an FDM value is less than a second threshold level to determine boundaries for the peaks.

22. The invention of claim 17, wherein:

step (b) comprises the step of applying the first threshold condition to identify the start frame of a possible transition;

step (c) comprises the step of applying the second threshold condition to determine if the possible transition is a probable transition; and

further comprising the step of applying the first threshold condition to one or more other FDM values to determine the end frame for the probable transition, if the second threshold condition is satisfied.

23. An apparatus for identifying transitions in a video sequence, comprising:

(a) means for generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size;

(b) means for applying a first threshold condition to an FDM value to determine if the corresponding pair of frames corresponds to a transition; and

(c) means for applying a second threshold condition to one or more other FDM values to identify start and end frames for the transition, if the first threshold condition is satisfied.

24. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor, cause the processor to implement a method for identifying transitions in a video sequence, the method comprising the steps of:

(a) generating a frame dissimilarity measure (FDM) value for each of a plurality of pairs of frames in the video sequence that are separated by a timing window size;

(b) applying a first threshold condition to an FDM value to determine if the corresponding pair of frames corresponds to a transition; and

(c) if the first threshold condition is satisfied, then applying a second threshold condition to one or more other FDM values to identify start and end frames for the transition.