IDENTIFYING SCENE BOUNDARIES USING GROUP SPARSITY ANALYSIS

• US 20150161450A1
• Filed: 02/13/2015
• Published: 06/11/2015
• Est. Priority Date: 08/03/2012
• Status: Active Grant

• Current Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Original Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Inventors: Kumar, Mrityunjay, Loui, Alexander C., Pillman, Bruce Harold
• Granted Patent: US 9,424,473 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06V 10/513 Sparse representations
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes

Abstract

A method for identifying a set of key video frames from a video sequence comprising extracting feature vectors for each video frame and applying a group sparsity algorithm to represent the feature vector for a particular video frame as a group sparse combination of the feature vectors for the other video frames. Weighting coefficients associated with the group sparse combination are analyzed to determine video frame clusters of temporally-contiguous, similar video frames. The video sequence is segmented into scenes by identifying scene boundaries based on the determined video frame clusters.

Claims

1-15. -15. (canceled)

16. A method for determining scene boundaries within a video sequence including a time sequence of video frames, each video frame including an array of image pixels having pixel values, comprising:

a) selecting a set of video frames from the video sequence, using a user interface to enable a user to manually indicate a starting point and an ending point for the set of video frames;

b) extracting a feature vector for each video frame in the set of video frames;

c) applying a group sparsity algorithm to represent the feature vector for a particular video frame as a group sparse combination of the feature vectors for the other video frames in the set of video frames, each feature vector for the other video frames in the group sparse combination having an associated weighting coefficient, wherein the weighting coefficients for feature vectors corresponding to other video frames that are most similar to the particular video frame are non-zero, and the weighting coefficients for feature vectors corresponding to other video frames that are most dissimilar from the particular video frame are zero;

d) analyzing the weighting coefficients to determine a video frame cluster of temporally-contiguous, similar video frames that includes the particular video frame;

e) repeating steps c)-d) for a plurality of particular video frames to provide a plurality of video frame clusters;

f) identifying one or more scene boundaries corresponding scenes in the video sequence based on the locations of boundaries between the determined video frame clusters; and

g) storing an indication of the identified scene boundaries in a processor-accessible memory;

wherein the method is performed, at least in part, using a data processor, wherein a subset of image pixels corresponding to a central region of the video frame are cropped out of the video frame or the video frame is spatially sub-sampled to provide a smaller image including a subset of the image pixels.

17. The method of claim 16 wherein the extraction of the feature vector for a video frame includes:

forming a frame vector including pixel values from the video frame;

defining a basis function set including a plurality of basis functions, each basis function being a vector having an identical length to the frame vector;

forming products between the frame vector and of the basis functions to determine corresponding feature coefficients;

forming the feature vector by collecting the feature coefficients for each basis function.

18. The method of claim 17 wherein the frame vector includes pixel values for a subset of the image pixels in the video frame selected according to a predefined sampling pattern.

19. The method of claim 17 wherein the basis functions are vectors of random numbers.

20. The method of claim 17 wherein the basis functions are Fourier transform basis functions, discrete cosine transform basis functions, or wavelet basis functions.

21. The method of claim 16 wherein the determined video frame cluster includes only the particular video frame and video frames that follow the particular video frame in the video sequence.

22. The method of claim 21 wherein the video frames included in the video frame cluster are determined by identifying the next video frame having an insignificant weighting coefficient that is smaller than a predefined threshold.

23. The method of claim 16 wherein the video frame cluster includes the temporally-contiguous video frames where the corresponding feature vectors have weighting coefficients that exceed a predefined threshold.

24. The method of claim 16 wherein similarity values are determined between adjacent video frame clusters, and wherein adjacent video frame clusters having similarity values that exceed a predefined threshold are merged to form video segments corresponding to scenes in the video sequence, and wherein boundaries of the video segments define the scene boundaries.

25. The method of claim 24 wherein the similarity values are determined by comparing representative video frames selected from within the video frame clusters.

26. The method of claim 24 wherein the similarity values are determined responsive to global color histograms determined for the video frame clusters.

27. The method of claim 24 wherein the similarity values are determined responsive to block-based color histograms determined for blocks of pixels in one or more video frames in the video frame clusters.

28. The method of claim 16 wherein indications of the determined scene boundaries are stored as metadata in association with a video file used to store the video sequence.

29. The method of claim 28 wherein the metadata is stored in the video.

30. The method of claim 16 wherein video frames within one or more video segments are extracted and stored in a separate video file.

31. The method of claim 16, wherein scene boundary information is stored as metadata in association with a video file either within the video file, in a separate file associated with the video file, or extracted and stored as a separate video file.