Method and system for segmentation, classification, and summarization of video images

• US 7,151,852 B2
• Filed: 10/21/2003
• Issued: 12/19/2006
• Est. Priority Date: 11/24/1999
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Gong, Yihong, Liu, Xin
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): Seth, Manav
• Application Number:
  US10/691,222
• Publication Number:
  US 20040170321A1
• Time in Patent Office: 1,155 Days
• Field of Search: 382168-169, 382224-226, 382/236, 382/305, 382/173, 715/723, 348700-701, 348/703, 348588-591, 707/3, 375/240, 375/240.02, 375/240.21, 375/240.26
• US Class Current: 382/173
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

In a technique for video segmentation, classification and summarization based on the singular value decomposition, frames of the input video sequence are represented by vectors composed of concatenated histograms descriptive of the spatial distributions of colors within the video frames. The singular value decomposition maps these vectors into a refined feature space. In the refined feature space produced by the singular value decomposition, the invention uses a metric to measure the amount of information contained in each video shot of the input video sequence. The most static video shot is defined as an information unit, and the content value computed from this shot is used as a threshold to cluster the remaining frames. The clustered frames are displayed using a set of static keyframes or a summary video sequence. The video segmentation technique relies on the distance between the frames in the refined feature space to calculate the similarity between frames in the input video sequence. The input video sequence is segmented based on the values of the calculated similarities. Finally, average video attribute values in each segment are used in classifying the segments.

Claims

1. A method for segmenting an input video sequence, said input video sequence comprising a plurality of frames, said method comprising:

grouping said plurality of frames into a plurality of video shots, wherein grouping is determined using a minimum time for frame inclusion, a maximum time for frame exclusion and a range of time from the minimum time to the maximum time, for which further analysis is performed;

computing a similarity between each of said plurality of frames and a frame preceding said each of said plurality of frames in time;

segmenting said input video sequence into said plurality of video shots according to said computed similarity.

2. The method of claim 1, wherein said similarity is calculated using a refined feature space representation of said input video sequence.

3. The method of claim 2, wherein said refined feature space representation is created using a singular value decomposition of said input video sequence.

4. The method of claim 3, wherein said singular value decomposition is performed using frames selected with a fixed interval from said input video sequence.

5. The method of claim 4, wherein said selected frames are arranged into a feature frame matrix, and wherein said singular value decomposition is performed on said feature frame matrix.

6. The method of claim 5, wherein said performed singular value decomposition produces a matrix, each column of said produced matrix comprising a frame in said refined feature space representing a frame in said input video sequence.

7. The method of claim 1, further comprising (d) extracting features from each of said plurality of video shots.

8. The method of claim 1, wherein said computed similarity is compared to at least a first threshold similarity and a second threshold similarity, and said input video sequence is segmented according to a result of said comparison.

9. The method of claim 1, wherein if in said computed similarity is below a first threshold similarity, said each of said plurality of frames is put into a one of said plurality of video shots containing a precedent in time frame.

10. The method of claim 1, wherein if in said computed similarity is above a second threshold similarity, said each of said plurality of frames is designated as a shot boundary.

11. The method of claim 1, wherein if in said computed similarity is between a first threshold similarity and a second threshold similarity, said each of said plurality of frames is put into a one of said plurality of video shots according to a further analysis performed using additional frames from said plurality of frames.

23. The method of claim 1, further comprising extracting features from each of said plurality of video shots and using said extracted features to index said plurality of video shots.

24. The method of claim 23, wherein said extracted features are features of a video frame representative of said each of said plurality of video shots.

12. A computer-readable medium containing a program for segmenting an input video sequence, said input video sequence comprising a plurality of frames, said program comprising:

grouping said plurality of frames into a plurality of video shots, wherein grouping is determined using a minimum time for frame inclusion, a maximum time for frame exclusion and a range of time from the minimum time to the maximum time, for which further analysis is performed;

computing a similarity between each of said plurality of frames and a subsequent in time frame;

segmenting said input video sequence into a plurality of shots according to said computed similarity.

13. The computer-readable medium of claim 12, wherein said similarity is calculated using a refined feature space representation of said input video sequence.

14. The computer-readable medium of claim 13, wherein said refined feature space representation is created using a singular value decomposition of said input video sequence.

15. The computer-readable medium of claim 14, wherein said singular value decomposition is performed using frames selected with a fixed interval from said input video sequence.

16. The computer-readable medium of claim 15, wherein said selected frames are arranged into a feature frame matrix, and wherein said singular value decomposition is performed on said feature frame matrix.

17. The computer-readable medium of claim 16, wherein said performed singular value decomposition produces a matrix, each column of said produced matrix comprising a frame in said refined feature space representing a frame in said input video sequence.

18. The computer-readable medium of claim 12, wherein said program further comprises extracting features from each of said plurality of video shots.

19. The computer-readable medium of claim 12, wherein in said computed similarity is compared to at least two threshold similarities, and said input video sequence is segmented according to a result of said comparison.

20. The computer-readable medium of claim 12, wherein if in said computed similarity is below a first threshold similarity, said each of said plurality of frames is put into a one of said plurality of video shots containing a precedent in time frame.

21. The computer-readable medium of claim 12, wherein if in said computed similarity is above a second threshold similarity, said each of said plurality of frames is designated as a shot boundary.

22. The computer-readable medium of claim 12, wherein if in said computed similarity is between a first threshold similarity and a second threshold similarity, said each of said plurality of frames is put into a one of said plurality of video shots according to a further analysis performed using additional frames from said plurality of frames.

25. The computer-readable medium of claim 12, wherein said program further comprises (c) extracting features from each of said plurality of video shots and using said extracted features to index said plurality of video shots.

26. The computer-readable medium of claim 25, wherein said extracted features are features of a video frame representative of said each of said plurality of video shots.