Image processing

• US 20040088289A1
• Filed: 09/11/2003
• Published: 05/06/2004
• Est. Priority Date: 03/29/2001
• Status: Active Grant

• Current Assignee: British Telecommunications (BT Group Plc) (BT Group PLC)
• Original Assignee: British Telecommunications (BT Group Plc) (BT Group PLC)
• Inventors: Xu, Li-Qun, Hanjalic, Alan
• Granted Patent: US 7,593,618 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion

Abstract

The present invention relates to a method of determining a sequence of intensity measures corresponding to each of a sequence of video frames, the method being used particularly but not exclusively for selection of videos according to user preferences and for providing highlights from a video sequence.

The method uses low level video characteristics which may be related to the to arousal and valence affects conveyed to a viewer whilst watching the video recording.

Claims

1. A method of determining a sequence of intensity measures, each intensity measure corresponding to each of a sequence of video frames;

the method comprising steps of

determining a motion activity measure for each of said sequence to provide a

sequence of motion measures;

filtering the sequence of motion activity measures; and

providing the sequence of intensity measures using said filtered sequence of motion activity measures.

2. A method according to claim 1 further comprising

determining a measure of cut frequency to provide a sequence of cut frequency measures;

filtering the sequence of cut frequency measures; and

combining the filtered measures to provide said sequence of intensity measures.

3. A method according to claim 1 or claim 2 in which the video frames have an associated audio track and further comprising

determining a measure of sound energy in high frequencies for the audio track to provide a sequence of high frequency measures;

filtering the sequence of sound energy measures;

normalising the filtered sequence of sound energy measures; and

combining the normalised sequence of sound energy measures with the filtered measures to provide said sequence of intensity measures.

4. A method of forming an affect measure for a sequence of video frames having an associated audio track, comprising:

determining a sequence of intensity measures using a method according to any one of the preceding claims;

determining a pitch measurement for the audio track to provide a sequence of pitch measures;

filtering the sequence of pitch measures;

providing a sequence of valence measures using said filtered sequence of pitch measures; and

combining the sequence of valence measures with the sequence of intensity measures to provide a sequence of affect measures.

5. A method of selecting frames from a sequence of video frames comprising

determining a sequence of intensity measures corresponding to each of the sequence of video frames using the method according to any one of claims 1 to 3;

and

selecting frames from the sequence of video frames for which the intensity measure is greater than a predetermined threshold.

6. A method of selecting frames from a sequence of video frames comprising

determining a sequence of intensity measures corresponding to each of the sequence of video frames using the method according to any one of claims 1 to 3;

and

selecting frames from the sequence of video frames for which the intensity measure is greater than a threshold;

in which the threshold is determined in order to select a predetermined number of frames.

7. A method according to claim 4 in which the affect measure is generated by:

defining ranges of the intensity measures and valence measures; and

counting for respective combinations of intensity range and valence range the number of video frames whose measures fall into the respective range combination, to produce a multi-element affect measure.

9. A method according to claim 7 or 8 in which the counts are normalised by division by the number of frames in the video sequence.

10. A method according to claim 7, 8 or 9 including forming a generic affect measure by combining the affect measures obtained for a plurality of video frame sequences.

11. A method according to any one of claims 7 to 10 including the step of comparing an affect measure obtained for one sequence of video frames with the affect measure obtained for other video sequences.

8. A method of generating an affect measure for a sequence of video frames having an accompanying sound track, comprising:

(a) generating a sequence of intensity measures for the frames of the sequence, said intensity measures depending on the degree of visual activity in the sequence;

(b) generating a sequence of valence measures for the frames of the sequence, said valence measures depending on the pitch of the sound track; and

(c) defining ranges of the intensity measures and valence measures;

(d) counting for respective combinations of intensity range and valence range the number of video frames whose measures fall into the respective range combination, to produce a multi-element affect measure.