System and Method for Retargeting Video Sequences

• US 20130050574A1
• Filed: 08/29/2011
• Published: 02/28/2013
• Est. Priority Date: 08/29/2011
• Status: Active Grant

• Current Assignee: VID SCALE Inc. (InterDigital, Inc.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Lu, Taoran, Yuan, Zheng, Huang, Yu, Wu, Dapeng Oliver, Yu, Hong Heather
• Granted Patent: US 8,891,009 B2
• Time in Patent Office: Days
• US Class Current: 348/441
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content

Abstract

A system and method for retargeting video sequences are provided. A method for retargeting a video includes a plurality of frames includes determining saliency information for the plurality of frames, determining a cost metric for the video, and retargeting the video based on the cost metric to produce a retargeted video. The cost metric considers loss due to cropping, scaling, temporal factors, and spatial factors. The retargeting makes use of a crop window for each frame in the plurality of frames.

Claims

1. A method for retargeting a video comprising a plurality of frames, the method comprising:

determining saliency information for the plurality of frames;

determining a cost metric for the video, wherein the cost metric considers loss due to cropping, scaling, temporal factors, and spatial factors; and

retargeting the video based on the saliency information and the cost metric to produce a retargeted video, wherein the retargeting makes use of a crop window for each frame in the plurality of frames.

2. The method of claim 1, wherein the spatial factors comprise cropping loss and scaling loss for each frame in the plurality of frames.

3. The method of claim 1, wherein the temporal factors comprise motion information between frames in the plurality of frames.

4. The method of claim 1, wherein the saliency information comprises spatial saliency information, temporal saliency information, a fusion of spatial saliency information and temporal saliency information, or combinations thereof.

5. The method of claim 1, wherein determining saliency information comprises:

applying a spatial processing algorithm to the video to produce spatial saliency information; and

applying a motion processing algorithm to the video to produce temporal saliency information.

6. The method of claim 1, wherein determining saliency information comprises applying a domain fusion technique to spatial saliency information and temporal saliency information.

7. The method of claim 6, wherein the domain fusion technique comprises:

spatially masking the spatial saliency information; and

nonlinearly fusing the spatially masked spatial saliency information and the temporal saliency information.

8. The method of claim 1, wherein retargeting the video comprises:

determining a crop window for each border frame in the plurality of frames based on a cropping loss and a scaling loss for the frame;

optimizing a crop window path for inner frames between the crop windows of border frames based on a temporal factor and a spatial factor; and

producing the retargeted video from the optimized window path.

9. The method of claim 8, wherein determining a crop window for each border frame comprises searching the border frame for a first object of interest.

10. The method of claim 9, wherein the searching is performed using a brute force search.

11. The method of claim 8, wherein optimizing a crop window path comprises, for a sequence of frames in the video, determining an optimized crop window path for intermediate frames between a start border frame of the sequence and an end border frame of the sequence.

12. The method of claim 11, wherein the video comprises a plurality of sequences of frames, and wherein determining the optimized crop window path is repeated for each sequence in the plurality of sequences.

13. The method of claim 11, wherein the sequence comprises a plurality of frames of the video having related content.

14. The method of claim 11, wherein determining an optimized crop window path comprises searching for a minimum cost path between a first optimized crop window and a second optimized crop window.

15. The method of claim 14, wherein searching for a minimum cost path comprises minimizing an objective function of video retargeting performance.

16. The method of claim 11, further comprising, for the sequence,

determining a scaling factor for the sequence, and

applying the scaling factor to a crop window associated with each frame in the sequence.

17. The method of claim 16, wherein determining a scaling factor comprises:

combining a first scaling factor for a first optimized crop window and a second scaling factor of a second optimized crop window; and

adjusting the combined scaling factor based on a transition velocity.

18. The method of claim 1, wherein determining saliency information comprises determining saliency information for each frame in the plurality of frames.

19. A device comprising:

a saliency detect unit configured to determine saliency information for a video comprising a plurality of frames;

a cost unit configured to determine a cost metric for the video, and wherein the cost metric considers loss due to cropping, scaling, temporal factors, and spatial factors; and

a retargeting unit configured to retarget the video based on the saliency information and cost metric to produce a retargeted video, wherein the retargeting unit makes use of a crop window for each frame in the plurality of frames.

20. The device of claim 19, wherein the saliency detect unit comprises:

a spatial processor configured to produce spatial saliency information from the video;

a motion processor configured to produce temporal saliency information from the video; and

a domain fusion unit configured to fuse the spatial saliency information and the temporal saliency information to produce spatial-temporal saliency information.

21. The device of claim 20, wherein the domain fusion unit comprises:

a spatial masking unit configured to mask the spatial saliency information; and

a nonlinear fusing unit configured to fuse the masked spatial saliency information and the temporal saliency information.

22. The device of claim 19, wherein the retargeting unit comprises:

a crop window unit configured to determine a crop window for each frame in the plurality of frames;

a crop window path search unit configured to optimize a crop window path for the crop windows; and

a producing unit configured to generate the retargeted video based on the optimized crop window path and the crop windows.

23. The device of claim 22, wherein the crop window path search unit optimizes the crop window path based on a first optimized crop window for a start frame of a sequence of frames of the video, and a second optimized crop window for an end frame of the sequence.

24. The device of claim 23, wherein the video comprises a plurality of sequences, and wherein the crop window path search unit optimizes a crop window path for each sequence in the plurality of sequences.

25. The device of claim 22, wherein the retargeting unit further comprises a scaling unit coupled to the crop window path search unit, the scaling unit configured to determine a scaling factor for a sequence of frames of the video, and to apply the scaling factor to a crop window associated with each frame in the sequence.

26. A system comprising:

a video processor configured to retarget a video based on a cost metric to produce a retargeted video, where the video comprises a plurality of frames, wherein the cost metric considers loss due to cropping, scaling, temporal factors, and spatial factors, and wherein the video processor makes use of a crop window for each frame in the plurality of frames to retarget the video; and

a memory coupled to the video processor, the memory configured to store the retargeted video.

27. The system of claim 26, wherein the video processor comprises:

a saliency detect unit configured to determine saliency information for the video;

a cost unit configured to determine a cost metric for the video; and

a retargeting unit configured to retarget the video based on the cost metric to produce the retargeted video, wherein the retargeting unit makes use of crop windows for each frame in the plurality of frames.

28. The system of claim 27, wherein the saliency detect unit comprises:

a spatial processor configured to produce spatial saliency information from the video;

a motion processor configured to produce temporal saliency information from the video; and

a domain fusion unit configured to fuse the spatial saliency information and the temporal saliency information to produce spatial-temporal saliency information.

29. The system of claim 27, wherein the retargeting unit comprises:

a crop window unit configured to determine a crop window for each frame in the plurality of frames;

a crop window path search unit configured to optimize a crop window path for the crop windows; and

a producing unit configured to generate the retargeted video based on the optimized crop window path and the crop windows.

30. The system of claim 26, wherein the video processor comprises a microprocessor.

31. The system of claim 26, wherein the video processor comprises a digital signal processor.