Method for content-based temporal segmentation of video

• US 6,195,458 B1
• Filed: 07/29/1997
• Issued: 02/27/2001
• Est. Priority Date: 07/29/1997
• Status: Expired due to Term

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Naphade, Milind R., Mehrotra, Rajiv, Warnick, James, Gunsel, Bilge, Ferman, Ahmet M.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US08/902,545
• Time in Patent Office: 1,309 Days
• Field of Search: 382/173, 382/236, 382/225, 382/305, 382/164, 382/232, 345/328, 348/700-701, 348/722, 348/699, 348/412-413, 348/595, 348/597, 348/412.1, 348/413.1, 348/415.1, 348/416.1, 375/240.01, 375/240.12, 375/240.13, 375/240.15, 375/240.16, 375/240.08, 386/52-53
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A method for performing content-based temporal segmentation of video sequences, the method comprises the steps of transmitting the video sequence to a processor, identifying within the video sequence a plurality of type-specific individual temporal segments using a plurality of type-specific detectors; analyzing and refining the plurality of type-specific individual temporal segments identified in the identifying the plurality of type-specific individual temporal segments step; and outputting a list of locations within the video sequence of the identified type-specific individual temporal segments.

Claims

1. A method for performing content-based temporal segmentation of video sequences comprising the steps of:

(a) transmitting the video sequences to a processor;

(b) identifying within the video sequences camera shot temporal segments using frame-to-frame difference metrics followed by a clustering operation on the frame-to-frame difference metrics; and

(c) outputting a list of locations within the video sequences of the camera shot temporal segments based on the clustering operation on the frame-to-frame difference metrics.

2. The method of claim 1, wherein the frame-to-frame difference metrics includes individually or in combination a frame-to-frame color histogram difference or frame-to-frame pixel intensity difference.

3. The method of claim 1, wherein the clustering operation includes performing a k-means clustering operation.

4. The method as in claim 1, wherein the clustering operation is followed by a camera-shot temporal-segment refinement operation.

5. The method as in claim 4, wherein the refinement operation includes a local maxima analysis of a frame-to-frame color histogram difference based on the camera shot temporal segment locations from the clustering operation on the frame-to-frame difference metrics.

6. The method as in claim 1, wherein step (b) includes segmenting the video sequences into temporally smaller sequences based upon a pre-analysis of the frame-to-frame difference metrics prior to clustering.

7. The method as in claim 6, wherein the frame-to-frame difference metrics is the difference of the variance of the frame color histogram difference.

8. A computer program product, comprising:

a computer readable storage medium having a computer program stored thereon for performing the steps of;

(a) transmitting video sequences to a processor;

(b) identifying within the video sequences camera shot temporal segments using frame-to-frame difference metrics followed by a clustering operation on the frame-to-frame difference metrics; and

(c) outputting a list of locations within the video sequences of the camera shot temporal segments based on the clustering operation on the frame-to-frame difference metrics.

9. The computer program product of claim 8, wherein the frame-to-frame difference metrics includes individually or in combination a frame-to-frame color histogram difference or frame-to-frame pixel intensity difference.

10. The computer program product of claim 8, wherein the clustering operation includes performing a k-means clustering operation.

11. The computer program product of claim 8, wherein the clustering operation is followed by a camera-shot temporal-segment refinement operation.

12. The computer program product of claim 11, wherein the refinement operation includes a local maxima analysis of a frame-to-frame color histogram difference based on the camera shot temporal segments from the clustering operation on the frame-to-fame difference metrics.

13. The computer program product of claim 8, wherein step (b) includes segment the video sequences into temporally smaller sequences based upon a pre-analysis of the frame-to-fame difference metrics prior to clustering.

14. The computer program product of claim 13, wherein the frame-to-frame difference metrics is the variance of the frame color histogram difference.