System for selecting a keyframe to represent a video

• US 7,643,657 B2
• Filed: 07/26/2004
• Issued: 01/05/2010
• Est. Priority Date: 01/29/1999
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Dufaux, Frederic, Swain, Michael J.
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): DUNN, MISHAWN N
• Application Number:
  US10/899,838
• Publication Number:
  US 20050002452A1
• Time in Patent Office: 1,989 Days
• Field of Search: 386/46, 386/52, 386/95, 386/124, 707/104.1, 382/115, 382/164, 382/168, 382171-173, 375/240.01, 375/240.16, 348699-702
• US Class Current: 382/115
• CPC Class Codes: G06F 16/785 using colour or luminescence
• CPC Class Codes: G11B 2220/41 Flat as opposed to hierarchical combination, e.g. library of tapes or discs, CD changer, or groups of record carriers that together store one title

Abstract

A key frame representative of a sequence of frames in a video file is selected. The sequence of frames is divided into shots, sequences of frames captured by a single continuous operation of the camera, by detecting shot boundaries. Shot boundaries are detected by measuring changes in motion activity between frames using measures of pixel intensity and histogram differences between frames. The most interesting shot in the sequence of frames is selected using measures of motion activity, spatial activity, skin pixel color and shot length. The key frame is selected in the most interesting shot using measures of spatial activity and motion activity.

Claims

1. A method of extracting a single representative key frame from a sequence of frames, the sequence of frames including a plurality of shots, comprising the steps of:

detecting shot boundaries in the sequence of frames to identify shots within the detected shot boundaries;

selecting from the identified shots a single most interesting shot representative of the sequence of frames based on shot length;

selecting the single representative key frame in the selected most interesting shot; and

wherein the detecting shot boundaries, the selecting from the identified shots, and the selecting the single representative key frame are performed by an apparatus.

2. A method as claimed in claim 1 wherein the step of detecting shot boundaries is based on pixel-wise frame difference.

3. A method as claimed in claim 2 wherein the step of detecting shot boundaries is based on distribution of a luminance histogram.

4. A method as claimed in claim 3 wherein the step of detecting shot boundaries is based on luminance.

5. A method as claimed in claim 4 wherein the step of detecting shot boundaries is based on a discontinuity in pixel-wise frame difference.

6. A method as claimed in claim 5 wherein the step of detecting shot boundaries is dependent on the result of a first test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a first threshold value.

7. A method as claimed in claim 6 wherein the step of detecting shot boundaries is dependent on the result of a second test to determine if the maximum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is greater than a second threshold value, which is less than the first threshold, and the minimum of a forward discontinuity in pixel-wise frame difference and a backward discontinuity in pixel-wise frame difference relative to a distribution of a luminance histogram is less than a third threshold value which is less than the second threshold.

8. A method as claimed in claim 1 wherein the step of selecting a single most interesting shot is based on level of skin color pixels.

9. A method as claimed in claim 8 wherein the step of selecting a single most interesting shot is based on motion between frames.

10. A method as claimed in claim 8 wherein the step of selecting a single most interesting shot is based on spatial activity within frames.

11. A method as claimed in claim 8 wherein the step of selecting a single most interesting shot is based on motion between frames, spatial activity within frames, and level of skin pixels.

12. A method as claimed in claim 1 wherein the step of selecting a key frame is based on frame activity.

13. A method as claimed in claim 12 wherein the step of selecting a key frame based on spatial activity within frame.

14. A method as claimed in claim 12 wherein the step of selecting a key frame is based on motion between frames.

15. A method of selecting a single most interesting multi-frame shot from a sequence of multi-frame shots comprising:

determining a level of skin pixels having a human flesh color of individual frames in each multi-frame shot;

selecting the most interesting multi-frame shot representative of the sequence of multi-frame shots based on the level of skin pixels; and

wherein the determining and the selecting are performed by an apparatus.

16. A method as claimed in claim 15 wherein the single most interesting shot is selected based on motion between frames.

17. A method as claimed in claim 15 wherein the single most interesting shot is selected based on spatial activity within frames.

18. A method as claimed in claim 15 wherein the single most interesting shot is selected based on shot length.

19. A method as claimed in claim 15 wherein the single most interesting shot is selected based on motion between frames, spatial activity within frames and shot length.

20. A method of selecting a single representative key frame from a sequence of frames, the sequence of frames including a plurality of shots, comprising:

selecting a single most interesting shot representative of the sequence of frames from the plurality of shots based on shot length;

determining frame activity of individual frames;

selecting the key frame from the most interesting shot based on the frame activity; and

wherein the selecting a single most interesting shot, the determining frame activity, and the selecting the key frame are performed by an apparatus.

21. A method as claimed in claim 20 wherein the frame activity is based on spatial activity within each frame.

22. A method as claimed in claim 20 wherein the frame activity is based on motion between frames.

23. An apparatus for extracting a key frame from a sequence of frames, comprising:

a keyframe extractor routine;

means, within the keyframe extractor routine, for detecting shot boundaries in the sequence of frames to identify shots within the detected shot boundaries;

means, within the keyframe extractor routine, for selecting a single most interesting shot representative of the sequence of frames from the identified shots within based on shot length; and

means, within the keyframe extractor routine, for selecting the key frame in the selected most interesting shot.

24. An apparatus as claimed in claim 23 wherein the means for detecting shot boundaries is based on pixel-wise frame difference and distribution of a luminance histogram.

25. An apparatus as claimed in claim 23 wherein the means for selecting a single most interesting shot is based on motion between frames, and spatial activity within frames and, level of skin pixels.

26. An apparatus as claimed in claim 23 wherein the means for selecting a key frame is based on frame activity.

27. An apparatus for extracting a single representative key frame from a sequence of frames, the sequence of frames including a plurality of shots, comprising:

a keyframe extractor routine responsive to a request for extracting a key frame from the sequence of frames, the extractor comprising;

a shot boundary detector detecting boundaries in the sequence of frames identify shots within the detected shot boundaries;

a key shot selector which selects a single most interesting shot representative of the sequence of frames from the identified shots is based on shot length; and

a key frame selector which selects a key frame in the selected most interesting shot.

28. An apparatus as claimed in claim 27 wherein the shot boundary detector detects shot boundaries based on pixel-wise frame difference and distribution of a luminance histogram.

29. An apparatus as claimed in claim 27 wherein the key shot selector selects the most interesting shot based on motion between frames, spatial activity within frames, level of skin pixels.

30. An apparatus as claimed in claim 27 wherein the key frame selector selects the most interesting frame based on frame activity.

31. A computer system comprising:

a memory system; and

a keyframe extractor routine located in the memory system responsive to a request for extracting a single representative key frame from a sequence of frames, the sequence of frames including a plurality of shots, which;

detects shot boundaries in the sequence of frames to identify shots within the detected shot boundaries;

selects a single most interesting shot representative of the sequence of frames from the identified shots based on shot length; and

selects the key frame in the selected most interesting shot.

32. A computer system as claimed in claim 31 wherein the keyframe extractor detects shot boundaries based on pixel-wise frame difference and distribution of a luminance histogram.

33. A computer system as claimed in claim 31 wherein the keyframe extractor selects the most interesting shot based on motion between frames, spatial activity within frames, and level of skin pixels.

34. A computer system as claimed in claim 31 wherein the keyframe extractor selects the key frame based on frame activity.