System and method for producing a page using frames of a video stream

• US 7,760,956 B2
• Filed: 05/12/2005
• Issued: 07/20/2010
• Est. Priority Date: 05/12/2005
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Untulis, Charles A., Lee, Dominic, Vondran, Gary L. Jr., Zhang, Tong, Chen, Mei, Lin, Xiaofan, Cheatle, Stephen Philip, Atkins, C. Brian
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Rosario; Dennis
• Application Number:
  US11/127,106
• Publication Number:
  US 20060257048A1
• Time in Patent Office: 1,895 Days
• Field of Search: 382/254, 382/282, 382/148, 382/299, 382/300, 382/325, 345/951, 715/204, 715/243, 715/724
• US Class Current: 382/254
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method and system for automatically producing a page using frames of a video stream. A plurality of video frames from a video stream or clip are read, and multiple frames are extracted from the video stream, based on the content of each frame. Frames which neighbor the extracted frames in the video stream are also extracted, with the number of neighboring frames to be extracted being controlled by parameters provided to the page production system. The resolution of the images in the extracted video frames is enhanced, based on the information in each extracted frame and each respective extracted neighboring frame. The enhanced images are automatically cropped to enhance the important features in each image, and the cropped images are composed into one or more pages that are representative of the content of the video stream.

Claims

1. A computer-implemented method of processing a sequence of video frames, comprising:

extracting key frames from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores;

resolution-enhancing image content of each of the keyframes, based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence;

generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and

determining a layout of the sequence of cropped images on one or more pages; and

producing a graphical representation of the layout.

2. The method according to claim 1, wherein the identifying comprises determining a respective color histogram for each of the video frames and selecting ones of the video frames as respective ones of the candidate frames based on comparisons between the respective ones of the color histograms.

3. The method according to claim 1, wherein the extracting comprises for each of the keyframes extracting the one or more video frames that neighbor the keyframe from the sequence.

4. The method according to claim 1, wherein the resolution-enhancing comprises:

computing a motion map for each pairing of an extracted frame and a respective extracted neighboring frame in a sequence of base images, wherein each computed motion map comprising a set of motion vectors mapping extracted frame image pixels to respective extracted neighboring frame image pixels;

assigning respective regions of a target image to motion classes based on the computed motion maps, the target image having a target resolution level and the base images having a base resolution level equal to or lower than the target resolution level; and

computing pixel values for the target image based on corresponding pixel value contributions from the base images selected in accordance with the motion classes assigned to the target image regions.

5. The method according to claim 1, wherein the cropping comprises identifying a respective region of interest in respective ones of the keyframes and cropping the keyframes such that the respective cropped images contain the respective regions of interest.

6. The method according to claim 5, wherein the cropping comprises:

identifying the regions of interest based on respective saliency images that are derived from the keyframes; and

determining one or more crop boundaries for each of the regions of interest based on application of one or more image composition rules.

7. The method according to claim 1, wherein the cropping comprises removing black top and bottom margins of the keyframes.

8. The method according to claim 1, wherein the cropping comprises cropping each of the keyframes to respective ones of the cropped images that comprise identified regions of interest.

9. The method according to claim 1, wherein the layout comprises images corresponding to cropped and uncropped ones of the keyframes.

10. The method according to claim 1, wherein the determining comprises for each of the pages;

establishing a respective candidate tree representing a recursive partitioning of the page into regions containing respective ones of the cropped images; and

in response to a determination that the candidate tree is a feasible candidate tree arranging the respective cropped images within the page in accordance with the candidate tree.

11. The method according to claim 1, wherein the determining comprises assigning the cropped frame images to respective ones of the pages based on the temporal sequence of the keyframes.

12. The method according to claim 1, wherein the producing comprises creating a poster from the layout.

13. The method according to claim 1, wherein the producing comprises creating a DVD cover page from the layout.

14. The method according to claim 1, wherein the producing comprises creating a video index page from the layout.

15. The method according to claim 1, wherein the producing comprises creating a storybook from the layout.

16. The method according to claim 1, wherein the producing comprises creating a postcard from the layout.

17. The method according to claim 1, wherein the producing comprises creating a calendar from the layout.

18. The method according to claim 1, wherein the producing comprises displaying the layout on a display monitor.

19. The method according to claim 1, wherein the cropping comprises cropping the keyframes in accordance with a user-selected cropping control parameter value.

20. The method of claim 1, wherein the cropping is performed before the resolution-enhancing.

21. The method of claim 1, wherein the resolution-enhancing is performed before the cropping.

22. A computer-implemented method of processing a sequence of video, comprising:

extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores;

generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes;

determining a layout of the sequence of cropped images on one or more pages; and

producing a graphical representation of the layout.

23. Apparatus for processing a page sequence of video frames, comprising:

a computer-readable medium storing computer-readable instructions; and

a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising;

extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores;

resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence;

generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes; and

determining a layout of the sequence of cropped images on one or more panes; and

producing a graphical representation of the layout.

24. At least one computer readable medium encoded with computer instructions adapted to be executed by a computer to perform a method of processing a sequence of video frames, the method comprising:

extracting keyframes from the sequence, wherein the extracting comprises identifying candidate frames from the sequence based on an analysis of content associated with the video frames, segmenting the candidate frames into respective clusters, assigning respective scores to each of the candidate frames, and selecting each of the keyframes from a respective one of the clusters based on the assigned scores;

resolution-enhancing image content of each of the keyframes based on estimated motion between the keyframe and one or more of the video frames that neighbor the keyframe in the sequence;

generating a sequence of resolution-enhanced cropped images respectively containing the image content, wherein the generating comprises automatically cropping the keyframes and

determining a layout of the sequence of cropped images on one or more pages; and

producing a graphical representation of the layout.