VIDEO RETARGETING

• US 20090251594A1
• Filed: 04/02/2008
• Published: 10/08/2009
• Est. Priority Date: 04/02/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Cha, Zhang, Zhengyou, Liu, Zicheng, Hua, Gang, Shan, Ying
• Granted Patent: US 9,240,056 B2
• Time in Patent Office: Days
• US Class Current: 348/441
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: H04N 1/3875 combined with enlarging or reducing enlarging or reducing per se H04N1/393
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 7/0122 the input and the output signals having different aspect ratios

Abstract

Videos are retargeted to a target display for viewing with little to no geometric distortion or video information loss. Salient regions of video frames may be determined using scale-space spatiotemporal information. Video information loss may be a result of spatial loss, due to cropping, and resolution loss, due to resizing. A desired cropping window may be determined using a coarse-to-fine searching strategy. Video frames may be cropped with a window that matches an aspect ratio of the target display, and resized isotropically to match a size of the target display.

Claims

1. A method for transforming a video to a target display, the method comprising:

cropping respective frames of the video using one or more cropping windows with a same aspect ratio as the target display, the cropping comprising;

determining salient areas of the respective frames of the video; and

temporally detecting and tracking salient cropping regions of respective frames of the video; and

resizing the one or more cropping windows of the respective frames of the video to fit the target display.

2. The method of claim 1, determining salient areas of the respective frames of the video comprising using extended spectral residual saliency detection.

3. The method of claim 2, using extended spectral residual saliency detection comprising determining saliency using phase spectrum information from a combination of a consecutive sequence of frames from the video.

4. The method of claim 2, using extended spectral residual saliency detection comprising determining saliency using phase spectrum information from a combination of a series of differently scaled frames of one or more of the respective frames of the video.

5. The method of claim 3, comprising determining saliency using phase spectrum information from a combination of a series of differently scaled frames of one or more of the respective frames of the video.

6. The method of claim 1, temporally detecting and tracking salient cropping regions of the respective frames of the video comprising determining one or more desired cropping windows for the respective frames of the video using a coarse to fine search.

7. The method of claim 6, determining one or more desired cropping windows for the respective frames of the video using a coarse to fine search comprising:

performing a coarse search over a solution space for one or more coarse salient optimal cropping windows from one or more of the respective frames of the video;

performing a gradient-based fine search of the one or more coarse salient optimal cropping windows over one or more of the respective frames of the video; and

determining one or more optimal salient cropping windows for the respective frames of the video based on the gradient-based fine search.

8. The method of claim 7, performing a gradient-based fine search using a gradient descent step with backtracking line search.

9. The method of claim 7, performing a gradient-based fine search comprising determining a cropping window with an optimal video spatial information loss to video resolution information loss ratio.

10. The method of claim 1, temporally detecting and tracking salient cropping regions of the respective frames of the video comprising using smoothing cropping parameters.

11. The method of claim 10, the smoothing cropping parameters comprising motion smoothness constraints.

12. The method of claim 11, the motion smoothness constraints comprising at least one of:

zooming and motion speed constraints during cropping and scaling of one or more of the respective frames of the video; and

zooming and motion acceleration constraints during cropping and scaling of one or more of the respective frames of the video.

13. The method of claim 10, using smoothing cropping parameters comprising using an exponential moving average scheme.

14. A system for transforming a video to a target display, the system comprising:

a video frame cropper configured to crop respective video frames of the video to a same aspect ratio as the target display, the cropper comprising;

a saliency determiner configured to generate saliency images of the respective video frames; and

a cropping window generator configured to temporally detect and track salient cropping regions of the respective video frames; and

a video frame resizer configured to resize a cropped area of the video frames to fit the target display.

15. The system of claim 14, the saliency determiner comprising:

a frame rescaler configured to create a plurality of different sizes of the video frames;

a spectral residual detector configured to detect a plurality of phase spectrum information from a video frame;

a scale spectral residual combiner configured to combine phase spectrum information from a plurality of different sizes of a video frame; and

a frame spectral residual combiner configured to combine phase spectrum information from a sequence of consecutive video frames.

16. The system of claim 14, the cropping window generator comprising:

a coarse salient cropping window generator configured to generate coarse cropping window information for the saliency image of the video frame;

a frame salient cropping window generator configured to generate fine search cropping window information for the saliency image of the video frame; and

a frame salient cropping window determiner configured to determine a cropping window for the saliency image of the video frame.

17. The system of claim 14, the video frame cropper comprising a cropping window applicator configured to:

apply the cropping window to the video frame; and

generate a cropped video frame.

18. The system of claim 15, comprising:

a scale/frame spectral residual information combiner configured to combine information from the scale spectral residual combiner and the frame spectral residual combiner for respective video frames; and

a saliency image generator configured to generate a saliency image for respective video frames.

19. The system of claim 16, the coarse salient cropping window generator configured to perform a coarse search for one or more coarse salient cropping windows of a saliency image of a video frame.

20. A method for transforming a video to a target display, the method comprising:

cropping respective frames of the video using a cropping area with a same aspect ratio as the target display, the cropping comprising;

determining salient areas of the respective frames of the video using extended spectral residual saliency detection, the determining comprising;

determining saliency using phase spectrum information from a combination of a consecutive sequence of frames from the video; and

determining saliency using phase spectrum information from a combination of a series of differently scaled frames of one or more of the respective frames of the video; and

temporally detecting and tracking salient cropping regions of the respective frames of the video comprising;

determining one or more desired cropping windows for the respective frames of the video using a coarse to fine search, the determining comprising;

performing a coarse search over a solution space for one or more coarse salient desired cropping windows for one or more of the respective frames of the video;

performing a gradient-based fine search of the one or more coarse salient desired cropping windows over one or more of the respective frames of the video, comprising;

 

using a gradient descent step with backtracking line search; and

 

determining a cropping window with a desired video spatial information loss to video resolution information loss ratio; and

determining a desired salient cropping window for one or more of the respective frames of the video based on the gradient-based fine search; and

using smoothing cropping parameters; and

resizing the cropping area of the respective frames of the video to fit the target display.