METHOD FOR CLASSIFICATION OF VIDEOS

• US 20130156304A1
• Filed: 06/29/2011
• Published: 06/20/2013
• Est. Priority Date: 07/01/2010
• Status: Active Grant

• Current Assignee: Telefonica SA
• Original Assignee: Telefonica SA
• Inventors: Obrador, Pere, Oliver Ramirez, Nuria, Moorty, Anush
• Granted Patent: US 8,942,469 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method for classifying a video regarding a subjective characteristic, the method comprising:

• • measuring a plurality of basic features (11) per frame thus obtaining a plurality of basic features measurements;
  • creating a plurality of second-level features by pooling (12) said basic features (11) measurements using a plurality of statistics of said basic features measurements in a determined period of time of footage;
  • creating a plurality of video features by pooling (13) said plurality of second-level features using a plurality of statistics of said second level features along the duration of the video;
  • choosing at least one video feature of said plurality of video features for classifying a video regarding a subjective characteristic.

Claims

1. A method for classifying a video regarding a subjective characteristic which is an aesthetic quality of the video perceived by human, the method comprising:

measuring a plurality of basic features (11) per frame thus obtaining a plurality of basic features measurements;

creating a plurality of second-level features by pooling (12) said basic features (11) measurements using a plurality of statistics of said basic features measurements in a determined period of time of footage;

creating a plurality of video features by temporal pooling (13) said plurality of second-level features using a plurality of statistics of said second level features along the duration of the video;

creating an additional video feature by measuring the frame rate of said video and using a structural similarity index (SSIM) algorithm as a measure of similarity between frames for computing said frame rate;

choosing at least one video feature of said plurality of video features for classifying a video regarding the aesthetic quality, the step of choosing said at least one video feature comprising;

comparing, for a determined combination of video features, an output value from a classifier, which is trained using the determined combination of features, with a training set of values predefined by human input.

2. The method of claim 1, wherein said plurality of basic features (11) measurements comprises at least one of the following features measurements:

at least one of the following motion features measurements;

motion ratio and size ratio, wherein said motion ratio is the ratio of motion magnitudes between the foreground and the background regions of the video and said size ratio is the ratio of sizes of said foreground and background regions;

a focus of the region of interest measurement;

a colourfulness measurement;

a luminance measurement;

a colour harmony measurement, wherein said measurement considers the effect that the combination of different colours has on the value of each frame;

a blockiness quality measurement, wherein said blockiness is the consequence of portions of a frame breaking into little squares;

a rule of thirds measurement, wherein said rule of thirds feature accounts for how important compositional elements of the image should be situated.

3. The method of claim 2, wherein said motion features are computed as follows:

performing motion estimation, by low-pass filtering the video frame and down-sampling said video frame by a factor of 2;

computing for each pixel location in said frame, the magnitude of the motion vector;

running a k-means algorithm with 2 clusters to segregate the motion vectors into two classes;

hiseogramming, within each one of said classes, the motion vector magnitudes and choosing the magnitude of the motion vector corresponding to the maximum of the histogram;

motion ratio is computed as f2 =(m_b+1)/(m_f+1), wherein m_f and m_b denote the magnitude of the motion vectors for each of said classes;

size ratio is computed as (s_b+1)/(s_f+1), wherein s_f and s_b denote the size (in pixels) of each of said foreground and background regions respectively.

5. The method of claim 2, wherein the red, green and blue (RGB) values of the frame are used in the following expressions:

α

=R−

B, β

=0.5×

(R+G)−

B, m_a=E[a], m_b=E[b], s^2a=E[(a−

m_a)²], and

s^2b=E[(b−

m_b)²]

wherein, E is the expectation operator, μ

is the mean and σ

, the standard deviation, to compute said colourfulness feature measurement as;

f₅=√

{square root over (s^2a+s^2b)}+0.3·

√

{square root over (m^2a+m^2b)};

6. The method of claim 2, wherein said luminance measurement is computed as the mean value of the luminance within a frame.

7. The method of claim 2, wherein said colour harmony measurement is computed as follows:

computing the normalized hue-histogram of each frame;

performing a convolution of said hue-histogram with each one of seven harmonic templates over the hue channel in the hue, saturation, value (HSV) space;

selecting the maximum of said convolution as a measure of similarity of the frame'"'"'s histogram to one of said particular templates;

selecting the maximum value of said measures of similarity as the colour harmony feature value.

8. The method of claim 2, wherein said blockiness quality measurement is computed by looking for blockiness artefacts.

9. The method of claim 2, wherein said rule of thirds measurement is computed as the minimum distance of the centroid of the region of interest to one of the four intersections of the lines that divide the image into nine equal rectangles.

10. The method of claim 1, wherein said plurality of statistics of basic features (11) measurements used to create said second-level features comprises at least one of the following:

mean, median, minimum, maximum, first quartile and third quartile.

11. The method of claim 1, wherein said plurality of statistics of second-level features measurements used to create video features comprises average and standard deviation.

12. The method of claim 1, wherein the frame rate together with the following video features are selected to characterize a video as high/low appealing, said following video features being referred as statistic of second level feature-statistic of basic feature-basic Feature:

Mean—

third quartile—

colourfulness

Standard Deviation—

median—

rule of thirds

Mean—

first quartile—

focus of the region of interest

Mean—

maximum—

luminance

Mean—

first quartile—

blockiness quality

Standard Deviation—

median—

focus if the region of interest.

13. A system comprising means adapted to perform the method according to any preceding claim.

14. A computer program comprising computer program code means adapted to perform the method according to claim 12 when said program is run on a computer, a digital signal processor, a field-programmable gate array, an application-specific integrated circuit, a micro-processor, a micro-controller, or any other form of programmable hardware.

15. The method of claim 3, wherein said plurality of statistics of basic features (11) measurements used to create said second-level features comprises at least one of the following:

mean, median, minimum, first quartile and third guar

16. The method of claim 15, wherein said plurality of statistics of second-level features measurements used to create video features comprises average and standard deviation.

17. The method of claim 16, wherein the frame rate together with the following video features are selected to characterize a video as high/low appealing, said following video features being referred as statistic of second level feature-statistic of basic feature-basic Feature:

Mean—

third quartile—

colourfulness

Standard Deviation—

median—

rule of thirds

Mean—

first quartile—

focus of the region of interest

Mean—

maximum—

luminance

Mean—

first quartile—

blockiness quality

Standard Deviation—

median—

focus if the region of interest.

18. The method of claim 9, wherein said plurality of statistics of basic features (11) measurements used to create said second-level features comprises at least one of the following:

mean, median, minimum, maximum, first quartile and third quartile.

19. The method of claim 18, wherein said plurality of statistics of second-level features measurements used to create video features comprises average and standard deviation.

20. The method of claim 19, wherein the frame rate together with the following video features are selected to characterize a video as high/low appealing, said following video features being referred as statistic of second level feature-statistic of basic feature-basic Feature:

Mean—

third quartile—

colourfulness

Standard Deviation—

median—

rule of thirds

Mean—

first quartile—

focus of the region of interest

Mean—

maximum—

luminance

Mean—

first quartile—

blockiness quality

Standard Deviation—

median—

focus if the region of interest.

4. The method of 2, wherein said focus on the region of interest measurement is computed by extracting said region of interest and setting the median of the level of focus of said region of interest.