Image processing for analyzing video content

• US 7,593,618 B2
• Filed: 03/15/2002
• Issued: 09/22/2009
• Est. Priority Date: 03/29/2001
• Status: Active Grant

• Current Assignee: British Telecommunications (BT Group Plc) (BT Group PLC)
• Original Assignee: British Telecommunications PLC (BT Group PLC)
• Inventors: Hanjalic, Alan, Xu, Li-Qun
• Primary Examiner(s): Harvey; David E
• Application Number:
  US10/471,429
• Publication Number:
  US 20040088289A1
• Time in Patent Office: 2,748 Days
• Field of Search: 386/46, 386/52, 386/69, 375/240.26, 382/173, 715/723
• US Class Current: 386/239
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion

Abstract

A sequence of intensity measures is determined corresponding to each of a sequence of video frames. Such can be used for selection of videos according to user preferences and for providing highlights from a video sequence. Low level video characteristics are used which may be related to arousal and valence affects conveyed to a viewer while watching the video recording.

Claims

1. An automated method of determining a sequence of intensity measures, each intensity measure corresponding to each of a sequence of recorded video frames, the method comprising using a programmed computer to:

determine a motion activity measure for each video frame of said sequence to provide a sequence of motion measures;

filter the sequence of motion activity measures;

provide a sequence of intensity measures using said filtered sequence of motion activity measures;

determine a measure of cut frequency to provide a sequence of cut frequency measures;

filter the sequence of cut frequency measures; and

combine the filtered measures to provide said sequence of intensity measures.

2. An automated method according to claim 1, in which the recorded video frames have an associated audio track and further comprising using said computer to:

determine a measure of sound energy in high frequencies for the audio track to provide a sequence of high frequency measures;

filter the sequence of sound energy measures;

normalize the filtered sequence of sound energy measures; and

combine the normalized sequence of sound energy measures with the filtered measures to provide said sequence of intensity measures.

3. An automated method according to claim 1, which also forms an affect measure for a sequence of recorded video frames having an associated audio track, by further comprising use of said computer to:

determine a pitch measurement for the audio track to provide a sequence of pitch measures;

filter the sequence of pitch measures;

provide a sequence of valence measures using said filtered sequence of pitch measures; and

combine the sequence of valence measures with the sequence of intensity measures to provide a sequence of affect measures.

4. An automated method of selecting frames from a sequence of recorded video frames comprising using said computer to determine a sequence of intensity measures corresponding to each of the sequence of video frames using the method according to claim 1, and selecting frames from the sequence of video frames for which the intensity measure is greater than a predetermined threshold.

5. An automated method of selecting frames from a sequence of recorded video frames comprising using said computer to determine a sequence of intensity measures corresponding to each of the sequence of video frames using the method according to claim 1, and in which the threshold is determined to select a predetermined number of frames.

6. An automated method according to claim 3, in which the affect measure is generated by using said computer to:

define ranges of the intensity measures and valence measures; and

count for respective combinations of intensity range and valence range the number of video frames whose measures fall into the respective range combination, to produce a multi-element affect measure.

7. An automated method according to claim 6, in which the counts are normalized by division by the number of frames in the video sequence.

8. An automated method according to claim 6, including forming a generic affect measure by combining the affect measures obtained for a plurality of video frame sequences.

9. An automated method according to claim 6, including the step of comparing an affect measure obtained for one sequence of video frames with the affect measure obtained for other video sequences.

10. An automated method of forming an affect measure for a sequence of recorded video frames having an associated audio track, said method comprising using a programmed computer to:

determine a sequence of intensity measures using a method according to claim 2;

determine a pitch measurement for the audio track to provide a sequence of pitch measures;

filter the sequence of pitch measures;

provide a sequence of valence measures using said filtered sequence of pitch measures; and

combine the sequence of valence measures with the sequence of intensity measures to provide a sequence of affect measures.

11. An automated method of generating an affect measure for a sequence of recorded video frames having an accompanying sound track, said method comprising using a programmed computer to:

(a) generate a sequence of intensity measures for the frames of the sequence, said intensity measures depending on the degree of visual activity in the sequence;

(b) generate a sequence of valence measures for the frames of the sequence, said valence measures depending on the pitch of the sound track;

(c) define ranges of the intensity measures and valence measures; and

(d) count for respective combinations of intensity range and valence range the number of video frames whose measures fall into the respective range combination, to produce a multi-element affect measure.

12. An automated method according to claim 11, in which the counts are normalized by division by the number of frames in the recorded video sequence.

13. An automated method according to claim 11, including forming a generic affect measure by combining the affect measures obtained for a plurality of recorded video frame sequences.

14. An automated method according to claim 11, including the step of comparing an affect measure obtained for one sequence of video frames with the affect measure obtained for other recorded video sequences.

15. An automated method of determining a sequence of intensity measures, each intensity measure corresponding to each of a sequence of recorded video frames having an associated audio track, the method comprising using a programmed computer to:

determine a motion activity measure for each frame of said sequence to provide a sequence of motion measures;

filter the sequence of motion activity measures;

provide the sequence of intensity measures using said filtered sequence of motion activity measures;

determine a measure of sound energy in high frequencies for the audio track to provide a sequence of high frequency measures;

filter the sequence of sound energy measures;

normalize the filtered sequence of sound energy measures; and

combine the normalized sequence of sound energy measures with the filtered measures to provide said sequence of intensity measures.

16. An automated method of determining a sequence of intensity measures, each intensity measure corresponding to each of a sequence of recorded video frames having an associated audio track and then forming an affect measure for a sequence of said video frames, said method comprising using a programmed computer to:

determine a motion activity measure for each frame of said sequence to provide a sequence of motion measures;

filter the sequence of motion activity measures;

provide the sequence of intensity measures using said filtered sequence of motion activity measures;

determine a pitch measurement for the audio track to provide a sequence of pitch measures;

filter the sequence of pitch measures;

provide a sequence of valence measures using said filtered sequence of pitch measures; and

combine the sequence of valence measures with the sequence of intensity measures to provide a sequence of affect measures.