Methods and Apparatus for Video Completion

• US 20130128121A1
• Filed: 11/24/2010
• Published: 05/23/2013
• Est. Priority Date: 09/14/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Agarwala, Aseem O., Goldman, Daniel, Leventhal, Daniel H.
• Granted Patent: US 9,013,634 B2
• Time in Patent Office: Days
• US Class Current: 348/607
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects

Abstract

Methods, apparatus, and computer-readable storage media for video completion that may be applied to restore missing content, for example holes or border regions, in video sequences. A video completion technique applies a subspace constraint technique that finds and tracks feature points in the video, which are used to form a model of the camera motion and to predict locations of background scene points in frames where the background is occluded. Another frame where those points were visible is found, and that frame is warped using the predicted points. A content-preserving warp technique may be used. Image consistency constraints may be applied to modify the warp so that it fills the hole seamlessly. A compositing technique is applied to composite the warped image into the hole. This process may be repeated until the missing content is filled on all frames.

Claims

1. A method, comprising:

obtaining an input video sequence of a scene comprising a plurality of frames and an indication of an object to be removed from the frames of the input video sequence;

tracking feature points in the input video sequence to generate feature tracks;

factoring the feature tracks into a low-dimensional subspace;

generating a prediction of background scene motion according to the low-dimensional subspace; and

processing a frame of the input video sequence as a target frame to fill a region in the target frame occluded by the indicated object with background content from unoccluded regions of one or more other frames of the input video sequence as source frames, wherein said processing the target frame is performed according to the prediction of background scene motion.

2. The method as recited in claim 1, further comprising performing said processing on each frame in the input video sequence that includes the indicated object.

3. The method as recited in claim 1, wherein said processing the target frame according to the prediction of background scene motion comprises:

identifying the unoccluded regions of the one or more source frames according to the prediction of background scene motion;

warping each source frame to the target frame according to the prediction of background scene motion; and

compositing at least a portion of the unoccluded region from each warped source frame into the occluded region of the target frame.

4. The method as recited in claim 3, wherein said warping is performed according to a content-preserving warp technique.

5. The method as recited in claim 3, further comprising applying one or more image consistency energy terms to improve alignment between the target frame and each warped source frame prior to said compositing.

6. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

obtain an input video sequence of a scene comprising a plurality of frames and an indication of an object to be removed from the frames of the input video sequence;

generate a prediction of background scene motion in the input video sequence; and

process a frame of the input video sequence as a target frame to fill a region in the target frame occluded by the indicated object with background content from unoccluded regions of one or more other frames of the input video sequence as source frames, wherein, to process a target frame, the program instructions are executable by the at least one processor to;

identify the unoccluded regions of the one or more source frames according to the prediction of background scene motion;

warp each source frame to the target frame according to the prediction of background scene motion;

apply one or more image consistency energy terms to improve alignment between the target frame and each warped source frame; and

composite at least a portion of the unoccluded region from each warped source frame into the occluded region of the target frame.

7. The system as recited in claim 6, wherein the program instructions are executable by the at least one processor to perform said processing on each frame in the input video sequence that includes the indicated object.

8. The system as recited in claim 6, wherein, to warp each source frame, the program instructions are executable by the at least one processor to apply a content-preserving warp technique.

9. The system as recited in claim 6, wherein the image consistency energy terms act to minimize differences in pixel values between the source frames and the target frame in a band around the region in the target frame occluded by the indicated object.

10. The system as recited in claim 6, wherein, to generate a prediction of background scene motion in the input video sequence, the program instructions are executable by the at least one processor to apply a subspace constraint technique that:

finds and tracks feature points in the input video sequence to generate feature tracks;

factors the feature tracks into a low-dimensional subspace; and

generates the prediction of background scene motion according to the low-dimensional subspace.

11. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining an input video sequence of a scene comprising a plurality of frames and an indication of an object to be removed from the frames of the input video sequence;

generating a prediction of background scene motion in the input video sequence; and

processing a frame of the input video sequence as a target frame to fill a region in the target frame occluded by the indicated object with background content from unoccluded regions of one or more other frames of the input video sequence as source frames, wherein said processing a target frame comprises;

identifying the unoccluded regions of the one or more source frames according to the prediction of background scene motion;

warping each source frame to the target frame according to the prediction of background scene motion;

applying one or more image consistency energy terms to improve alignment between the target frame and each warped source frame; and

compositing at least a portion of the unoccluded region from each warped source frame into the occluded region of the target frame.

12. The non-transitory computer-readable storage medium as recited in claim 11, wherein the program instructions are computer-executable to implement performing said processing on each frame in the input video sequence that includes the indicated object.

13. The non-transitory computer-readable storage medium as recited in claim 11, wherein said warping is performed according to a content-preserving warp technique.

14. The non-transitory computer-readable storage medium as recited in claim 11, wherein the image consistency energy terms act to minimize differences in pixel values between the source frames and the target frame in a band around the region in the target frame occluded by the indicated object.

15. The non-transitory computer-readable storage medium as recited in claim 11, wherein, in said generating a prediction of background scene motion in the input video sequence, the program instructions are computer-executable to implement applying a subspace constraint technique that:

finds and tracks feature points in the input video sequence to generate feature tracks;

factors the feature tracks into a low-dimensional subspace; and

generates the prediction of background scene motion according to the low-dimensional subspace.

16. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

obtaining an input video sequence of a scene comprising a plurality of frames, wherein the frames in the input video sequence have been cropped;

tracking feature points in the input video sequence to generate feature tracks;

factoring the feature tracks into a low-dimensional subspace; and

generating the prediction of background scene motion according to the low-dimensional subspace; and

processing each frame of the input video sequence as a target frame to fill a cropped region in the target frame with content from background regions of one or more other frames of the input video sequence as source frames, wherein said processing the target frame is performed according to the prediction of background scene motion.

17. The non-transitory computer-readable storage medium as recited in claim 16, wherein, in said processing the target frame, the program instructions are computer-executable to implement:

identifying the background regions of the one or more source frames according to the prediction of background scene motion;

warping each source frame to the target frame according to the prediction of background scene motion; and

compositing at least a portion of the background region from each warped source frame into the cropped region of the target frame.

18. The non-transitory computer-readable storage medium as recited in claim 17, wherein the program instructions are computer-executable to implement performing said warping according to a content-preserving warp technique.

19. The non-transitory computer-readable storage medium as recited in claim 17, wherein the program instructions are computer-executable to implement applying one or more image consistency energy terms to improve alignment between the target frame and each warped source frame prior to said compositing.

20. The non-transitory computer-readable storage medium as recited in claim 16, wherein the input video sequence is a stabilized output video sequence generated according to a video stabilization technique that generates the stabilized output video sequence from an original video sequence.