Methods and Systems for Video Retargeting Using Motion Saliency

• US 20140044404A1
• Filed: 10/21/2013
• Published: 02/13/2014
• Est. Priority Date: 03/13/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Grundmann, Matthias, Kwatra, Vivek
• Granted Patent: US 9,167,221 B2
• Time in Patent Office: Days
• US Class Current: 386/230
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 9/87 Regeneration of colour television signals H04N9/80 takes precedence

Abstract

Methods and systems for video retargeting and view selection using motion saliency are described. Salient features in multiple videos may be extracted. Each video may be retargeted by modifying the video to preserve the salient features. A crop path may be estimated and applied to a video to retarget each video and generate a modified video preserving the salient features. An action score may be assigned to portions or frames of each modified video to represent motion content in the modified video. Selecting a view from one of the given modified videos may be formulated as an optimization subject to constraints. An objective function for the optimization may include maximizing the action score. This optimization may also be subject to constraints to take into consideration optimal transitioning from a view from a given video to another view from another given video, for example.

Claims

1. A method comprising:

receiving a plurality of videos, each video comprising a sequence of frames;

determining salient features in a content of each video, wherein the salient features include features selected based on motion content of the features over the sequence of frames;

determining a camera crop path for each video, wherein the camera crop path comprises a sequence of crop windows, the sequence of crop windows including the salient features;

applying the sequence of crop windows for each video to the sequence of frames of each video to generate a modified video for each video including the salient features of each respective video; and

selecting one of the modified videos, wherein selecting the one of the modified videos comprises performing an optimization over time to select the one of the modified videos, and wherein the optimization is subject to a constraint to limit switching between the modified videos.

2. The method of claim 1, wherein determining the camera crop path for each video comprises determining a crop window of a pre-defined size, wherein the crop window is defined to fit inside an original frame that is in a received video.

3. The method of claim 2, wherein determining the crop window of the pre-defined size comprises determining a size of the crop window large enough to include at least a threshold number of salient features.

4. The method of claim 2, wherein determining the crop window of the pre-defined size comprises:

determining a size of the crop window large enough to include at least a first threshold number of salient features; and

restricting the size of the crop window such that the crop window includes a second threshold number of salient features.

5. The method of claim 1, wherein determining the camera crop path comprises determining a given camera crop path comprising at least one of constant, linear, or parabolic segments.

6. The method of claim 1, wherein determining the sequence of crop windows comprises optimizing a weighted combination of a first derivative, a second derivative, and a third derivative of the camera crop path.

7. The method of claim 6, wherein optimizing the weighted combination of the first derivative, the second derivative, and the third derivative of the camera crop path comprises performing an L1 optimization subject to at least one constraint.

8. The method of claim 7, wherein the at least one constraint comprises an inclusion constraint that requires a given crop window to be contained within an original frame of a received video at all times.

9. The method of claim 1, wherein applying the sequence of crop windows of each video to the sequence of frames of each video to generate a modified video for each video including the salient features comprises removing content from frames of the respective video.

10. The method of claim 1, further comprising:

determining an action score for each modified video based on action content in each modified video, wherein selecting the one of the modified videos is further based on the action score determined for each modified video.

11. The method of claim 10, wherein determining the action score based on the action content comprises determining an average motion magnitude of the salient features in each modified video.

12. The method of claim 1, wherein the constraint includes a respective negative switching penalty assigned to each modified video, wherein the respective negative switching penalty decreases over time.

13. The method of claim 1, wherein determining the salient features in the content of each video comprises determining features of frames of each video that remain in respective frames of each video over the sequence of frames.

14. The method of claim 1, wherein receiving the plurality of videos comprises receiving a video from each of a plurality of non-moving cameras, and wherein the salient features in the content of each video include features that move in position relative to a background of the content of each video over the sequence of frames.

15. A non-transitory computer readable medium having stored therein instructions that, in response to execution by a computing device, cause the computing device to perform operations comprising:

receiving a plurality of videos, each video comprising a sequence of frames;

determining salient features in a content of each video, wherein the salient features include features selected based on motion content of the features over the sequence of frames;

determining a camera crop path for each video, wherein the camera crop path comprises a sequence of crop windows, the sequence of crop windows including the salient features;

applying the sequence of crop windows for each video to the sequence of frames of each video to generate a modified video for each video including the salient features of each respective video; and

selecting one of the modified videos, wherein selecting the one of the modified videos comprises performing an optimization over time to select the one of the modified videos, and wherein the optimization is subject to a constraint to limit switching between the modified videos.

16. The non-transitory computer readable medium of claim 15, wherein the operation of determining the camera crop path for each video comprises determining a crop window of a pre-defined size, wherein the crop window is defined to fit inside an original frame that is in a received video, and wherein the crop window includes at least a threshold number of salient features.

17. The non-transitory computer readable medium of claim 15, wherein the operations further comprise:

generating a display comprising content from the plurality of videos, wherein the display portrays content of the selected modified video at a given point in time.

18. The non-transitory computer readable medium of claim 15, wherein the operations further comprise:

determining an action score for each modified video based on action content in each modified video, wherein selecting the one of the modified videos is further based on the action score determined for each modified video.

19. A system comprising:

a motion saliency engine configured to receive a plurality of videos and to determine salient features in a content of each video, wherein the salient features include features selected based on motion content of the features over the sequence of frames;

a video retargeting engine in communication with the motion saliency engine and configured to determine a camera crop path for each video, wherein the camera crop path comprises a sequence of crop windows, the sequence of crop windows including the salient features, wherein the video retargeting engine is further configured to apply the sequence of crop windows for each video to the sequence of frames of each video to generate a modified video for each video including the salient features of each respective video; and

a view selection engine in communication with the motion saliency engine and the video retargeting engine and configured to select one of the modified videos, wherein to select the one of the modified videos the view selection engine is configured to perform an optimization over time to select the one of the modified videos, and wherein the optimization is subject to a constraint to limit switching between the modified videos.

20. The system of claim 19, wherein for the optimization to be subject to the constraint to limit switching between the modified videos, the view selection engine is configured to assign a respective negative switching penalty to each modified video, and wherein the respective negative switching penalty decreases over time.