System for selecting a keyframe to represent a video

• US 6,782,049 B1
• Filed: 02/09/1999
• Issued: 08/24/2004
• Est. Priority Date: 01/29/1999
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Swain, Michael J., Dufaux, Frederic
• Primary Examiner(s): LEE, RICHARD J
• Application Number:
  US09/248,525
• Time in Patent Office: 2,023 Days
• Field of Search: H04/N.712, 375/240.16, 375/240.01, 348/699, 348/700, 348/701-702, 345/328, 345/723, 382/164, 382/168, 382/171-172, 382/173
• US Class Current: 375/240.01
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G11B 2220/41 Flat as opposed to hierarchical combination, e.g. library of tapes or discs, CD changer, or groups of record carriers that together store one title

Abstract

A key frame representative of a sequence of frames in a video file is selected. The sequence of frames is divided into shots, sequences of frames captured by a single continuous operation of the camera, by detecting shot boundaries. Shot boundaries are detected by measuring changes in motion activity between frames using measures of pixel intensity and histogram differences between frames. The most interesting shot in the sequence of frames is selected using measures of motion activity, spatial activity, skin pixel color and shot length. The key frame is selected in the most interesting shot using measures of spatial activity and motion activity.

Claims

1. A method of extracting a key frame from a sequence of frames, comprising the steps of:

detecting shot boundaries in the sequence of frames based on pixel-wise frame difference, distribution of a luminance histogram, luminance and a discontinuity in pixel-wise frame difference dependent on the result of a first test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a first threshold value and dependent on the result of a second test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a second threshold value, which is less than the first threshold value, and the minimum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is less than a third threshold value which is less than the second threshold value;

selecting a key shot from shots within the detected shot boundaries; and

selecting a key frame in the selected key shot.

2. The method as claimed in claim 1 wherein the step of selecting a key shot is based on level of skin color pixels.

3. The method as claimed in claim 2 wherein the step of selecting a key shot is based on motion between frames.

4. The method as claimed in claim 2 wherein the step of selecting a key shot is based on spatial activity within frames.

5. The method as claimed in claim 2 wherein the step of selecting a key shot is based on shot length.

6. The method as claimed in claim 2 wherein the step of selecting a key shot is based on motion between frames, spatial activity within frames, level of skin pixels and shot length.

7. The method as claimed in claim 1 wherein the step of selecting a key frame is based on frame activity.

8. The method as claimed in claim 7 wherein the frame activity is based on spatial activity within frame.

9. The method as claimed in claim 7 wherein the frame activity is based on motion between frames.

10. A method for extracting a key frame from a sequence of frames comprising the steps of:

detecting shot boundaries in the sequence of frames based on discontinuity in pixel-wise frame difference and distribution of a luminance histogram;

the result of a first test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a first threshold value and the result of a second test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a second threshold value and the minimum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is less than a third threshold value which is less than the second threshold value;

selecting a key shot from shots within the detected shot boundaries dependent on level of skin color pixels and spatial activity within frames, motion between frames and shot length; and

selecting a key frame in the selected key shot dependent on frame activity based on spatial activity within the frame and motion between frames.

11. A computer program product for extracting a key frame from a sequence of frames, the computer program product comprising a computer usable medium having computer readable code thereon, including program code which:

detects shot boundaries in the sequence of frames based on pixel-wise frame difference, distribution of a luminance histogram luminance and a discontinuity in pixel-wise frame difference dependent on the result of a first test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a first threshold value and dependent on the result of a second test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a second threshold value which is less than the first threshold value and the minimum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is less than a third threshold value which is less than the second thresholds value;

selects a key shot from shots within the detected shot boundaries; and

selects a key frame in the selected key shot.

12. The computer program product as claimed in claim 11 wherein the step of selecting a key shot is based on level of skill color pixels.

13. The computer program product as claimed in claim 12 wherein the step of selecting a key shot is based on motion between frames.

14. The computer program product as claimed in claim 12 wherein the step of selecting a key shot is based on spatial activity within frames.

15. The computer program product as claimed in claim 12 wherein the step of selecting a key shot is selected based on shot length.

16. The computer program product as claimed in claim 12 wherein the step of selecting a key shot is based on motion between frames, spatial activity within frames and shot length.

17. The computer program product as claimed in claim 11 wherein the step of selecting a key frame is based on frame activity.

18. The computer program product as claimed in claim 17 wherein the frame activity is based on spatial activity within frame.

19. The computer program product as claimed in claim 17 wherein the frame activity is based on motion between frames.