Video skimming system utilizing the vector rank filter

• US 6,771,268 B1
• Filed: 03/03/2000
• Issued: 08/03/2004
• Est. Priority Date: 04/06/1999
• Status: Expired due to Term

• Current Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Crinon, Regis J.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/518,610
• Time in Patent Office: 1,614 Days
• Field of Search: 345/418, 345/419, 345/475, 345/619, 345/620, 345/625, 345/716, 345/723, 345/730, 382/225, 704/251, 386/46, 386/68, 386/69, 386/70
• US Class Current: 345/475
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content

Abstract

Automated summarization of digital video sequences is accomplished using a vector rank filter. The consecutive frames of a digital video sequence can be represented as feature vectors which are successively accumulated in a set of vectors. The distortion of the set by the addition of each successive vector or the cumulative distance from each successive vector to all other vectors in the set is determined by a vector rank filter. When the distortion exceeds a threshold value the end of a video segment is detected. Each frame in a video segment can be ranked according to its relative similarity to the other frames of the set by applying the vector rank filter to the feature vectors representing the video frames. To produce a summary of a video sequence which is most representative of the content of the sequence, frames are chosen that correspond to vectors that are the least distant to or produce the least distortion of the set of vectors representing the segment. The ranking of the relative distortion can be used as the basis for selecting more than one frame from each segment to produce a hierarchy of summaries containing greater numbers of the frames having the most representative content.

Claims

1. A method of identifying a key video frame within a plurality of video frames comprising the steps of:

(a) dividing said plurality of video frames into at least two segments;

(b) characterizing said plurality of video frames as a plurality of feature vectors;

(c) identifying for each of at least two segments a first level key feature vector that minimizes a distortion of said feature vectors included in said segment;

(d) identifying a second level key feature vector that minimizes distortion of a set of first level key feature vectors; and

(e) identifying a video frame corresponding to said second level key feature vector as said key video frame.

2. The method of claim 1 wherein said first level key feature vector that minimizes said distortion of said plurality of feature vectors is identified by a vector rank filter.

3. The method of claim 1 wherein said second level key feature vector that minimizes said distortion of said set of first level key feature vectors is identified by a vector rank filter.

4. The method of claim 1 wherein at least one feature vector of said plurality of feature vectors is a vector signal describing an image histogram.

5. The method of claim 1 wherein at least one feature vector of said plurality of feature vectors is a vector signal describing an ordered set of image samples.

6. The method of claim 1 wherein at least one feature vector of said plurality of feature vectors is a vector signal which includes a component describing luminance of said video frame.

7. The method of claim 1 wherein at least one feature vector of said plurality of feature vectors is a vector signal which includes a component describing the average intensity of said video frame.

8. A method of identifying a key video frame within a plurality of video frames comprising the steps of:

(a) dividing said plurality of video frames into at least two segments;

(b) characterizing said plurality of video frames as a plurality of feature vectors;

(c) identifying for each of at least two segments a first level key feature vector that minimizes a distortion of said feature vectors included in said segment;

(d) accumulating a set of said first level key feature vectors corresponding to said segments occurring in a predetermined period;

(e) identifying a second level key feature vector that minimizes distortion of said set of first level key feature vectors; and

(f) identifying a video frame corresponding to said second level key feature vector as said key video frame.

9. A method of summarizing of a plurality of video frames comprising the steps of:

(a) dividing said plurality of video frames into at least two segments, each segment comprising at least one video frame;

(b) defining a feature vector representative of each of said video frames;

(c) identifying as a key feature vector a feature vector that minimizes a distortion of said feature vectors included in said segment;

(d) ranking a saliency of a key feature vector relative to other key feature vectors;

(f) selecting a video frame for said summary on a basis of said saliency ranking of a corresponding a key feature vector.

10. The method of claim 9 wherein said saliency is a function of a distortion of a set of key feature vectors produced by a feature vector included in said set.

11. The method of claim 9 wherein said saliency is a function of a distortion of a set of key feature vectors produced by a feature vector included in said set and a number of video frames included in said segment represented by said key feature vector.