System and method for producing a page using frames of a video stream

• US 20060257048A1
• Filed: 05/12/2005
• Published: 11/16/2006
• Est. Priority Date: 05/12/2005
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Zhang, Tong, Chen, Mei, Lin, Xiaofan, Cheatle, Stephen Philip, Atkins, C. Brian, Untulis, Charles A., Lee, Dominic, Vondran, Gary L. Jr.
• Granted Patent: US 7,760,956 B2
• Time in Patent Office: Days
• US Class Current: 382/276
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44

Abstract

A method and system for automatically producing a page using frames of a video stream. A plurality of video frames from a video stream or clip are read, and multiple frames are extracted from the video stream, based on the content of each frame. Frames which neighbor the extracted frames in the video stream are also extracted, with the number of neighboring frames to be extracted being controlled by parameters provided to the page production system. The resolution of the images in the extracted video frames is enhanced, based on the information in each extracted frame and each respective extracted neighboring frame. The enhanced images are automatically cropped to enhance the important features in each image, and the cropped images are composed into one or more pages that are representative of the content of the video stream.

Claims

1. A computer-implemented method for producing a page using frames of a video stream, comprising:

automatically extracting multiple frames from a video stream, based on frame content;

enhancing resolution of images contained within each of the extracted frames, using information from neighboring frames;

automatically cropping the enhanced images within each of the extracted frames to produce cropped images; and

composing one or more pages having a layout of the cropped images.

2. The method according to claim 1, including:

selecting a set of candidate frames from the video stream by performing an analysis on each video frame, wherein the analysis is selected to detect a content in the video stream;

arranging the selected candidate frames into a set of clusters; and

selecting a candidate frame for extraction from each cluster based on an importance score for each candidate frame.

4. The method according to claim 1, including:

extracting frames neighboring each extracted frame; and

enhancing the spatial resolution of each extracted frame based on the extracted frame and its extracted neighboring frames.

5. The method according to claim 4, including:

computing a motion map for each pairing of an extracted frame and a respective extracted neighboring frame in a sequence of base images, wherein each computed motion map comprising a set of motion vectors mapping extracted frame image pixels to respective extracted neighboring frame image pixels;

assigning respective regions of a target image to motion classes based on the computed motion maps, the target image having a target resolution level and the base images having a base resolution level equal to or lower than the target resolution level; and

computing pixel values for the target image based on corresponding pixel value contributions from the base images selected in accordance with the motion classes assigned to the target image regions.

6. The method according to claim 1, wherein each enhanced frame is automatically cropped to keep only the important portion of the enhanced frame.

7. The method according to claim 1, including:

processing the enhanced images to identify one or more features relevant to the composition of the enhanced images;

selecting at least one compositional rule from a plurality of predetermined compositional rules, based on the relevance of the selected compositional rules to one or more of the identified features; and

determining one or more suitable crop boundaries for the enhanced images by applying the selected compositional rules.

8. The method according to claim 1, including:

removing black top and bottom margins of the extracted frames prior to cropping the images of the frames.

9. The method according to claim 1, wherein each extracted frame is automatically cropped to keep only the important portion.

10. The method according to claim 1, wherein the composed page has a layout comprising cropped and uncropped images.

11. The method according to claim 1, including:

establishing a candidate tree associated with the cropped images;

determining if the candidate tree is a feasible candidate tree; and

in response to the candidate tree being determined to be feasible, arranging the cropped images within a page in accordance with the candidate tree.

12. The method according to claim 1, including placing the cropped frame images onto pages using the temporal sequence of the extracted frames.

13. The method according to claim 1, including:

creating a poster from the composed frame images.

14. The method according to claim 1, including:

creating a DVD cover page from the composed frame images.

15. The method according to claim 1, including:

creating a video index page from the composed frame images.

16. The method according to claim 1, including:

creating a storybook from the composed frame images.

17. The method according to claim 1, including:

creating a postcard from the composed frame images.

18. The method according to claim 1, including:

creating a calendar from the composed frame images.

19. The method according to claim 1, wherein the composed page is displayed to a user.

20. The method according to claim 1, wherein the steps of the method can be directed by control parameters.

3. The method according to step 2, wherein the analysis comprises a color histogram difference comparison of the frames of the video stream.

21. A computer-implemented method for producing a page using frames of a video stream, comprising:

automatically extracting multiple frames from a video stream, based on frame content;

automatically cropping images within each of the extracted frames to produce cropped images; and

composing a page having a layout of the cropped images.

22. The method according to claim 21, including:

extracting frames neighboring the extracted frames; and

enhancing resolution of images contained within each of the extracted frames, using information from neighboring frames.

23. A computer-based system for producing a page using frames of a video stream, comprising:

a processor for executing computer instructions that;

automatically extract one or more frames from a video stream comprising a plurality of frames;

automatically extract one or more pairs of neighboring frames adjacent to the extracted frames in the video stream;

enhance resolution of the images contained within each of the extracted frames based on the content of the extracted neighboring frames;

automatically crop the enhanced frame images to produce cropped images; and

compose one or more pages comprising the cropped images.

24. A computer readable medium encoded with computer instructions, which, when executed by a computer, provide for the automatic production of one or more pages using frames of a video stream, wherein the instructions are provided for:

inputting a video stream comprising a plurality of frames;

automatically extracting frames from the video stream;

automatically enhancing resolution of images contained within the extracted frames;

automatically cropping the enhanced images to produce cropped images; and

composing one or more pages from the cropped images, wherein the composed pages are representative of the video stream.