AUTOMATIC COMPOSITION OF VIDEO WITH DYNAMIC BACKGROUND AND COMPOSITE FRAMES SELECTED BASED ON FRAME AND FOREGROUND OBJECT CRITERIA

• US 20170094194A1
• Filed: 03/24/2016
• Published: 03/30/2017
• Est. Priority Date: 09/28/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Adsumilli, Balineedu Chowdary, Lustig, Ryan
• Granted Patent: US 10,051,206 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A processing device generates composite images from a sequence of images. The composite images may be used as frames of video. A foreground/background segmentation is performed at selected frames to extract a plurality of foreground object images depicting a foreground object at different locations as it moves across a scene. The foreground object images are stored to a foreground object list. The foreground object images in the foreground object list are overlaid onto subsequent video frames that follow the respective frames from which they were extracted, thereby generating a composite video.

Claims

1. A method for generating a composite output video from an input video having a sequence of frames, the method comprising:

selecting from the sequence of frames, a range of frames for processing;

performing, by a processing device, a foreground/background segmentation on each of the frames in the range of frames to extract a plurality of candidate foreground object images based on a predictive model, each of the candidate foreground object images comprising a representation of a foreground object depicted in a corresponding video frame with background pixels subtracted;

selecting, based on an image metric, a selected foreground object image from the plurality of candidate foreground object images;

storing the selected foreground object images to a foreground object list; and

overlaying the foreground object images in the foreground object list on a current video frame to generate a composite video frame.

2. The method of claim 1, wherein selecting the selected foreground object image comprises:

determining an image quality metric for each of the candidate foreground object images; and

determining that the selected foreground object image has a highest quality metric.

3. The method of claim 1, wherein selecting the selected foreground object image comprises:

determining a face detection likelihood on each of the candidate foreground object images; and

determining that the selected foreground object image has a highest face detection likelihood.

4. The method of claim 1, wherein selecting the selected foreground object image comprises:

determining a motion parameter for each of the candidate foreground object images; and

determining that the selected foreground object image has a motion parameter best matching a predefined motion criteria.

5. The method of claim 1, further comprising:

training the predictive model based on a plurality of training video frames, the predictive model to predict whether a pixel in a given video frame belongs to a background model or the foreground object.

6. The method of claim 5, further comprising:

determining if a frame number of the current video frame is a multiple of a predefined integer X; and

responsive to the frame number of the current video frame being the multiple of the predefined integer X, updating the predictive model.

7. The method of claim 1, wherein performing the foreground/background segmentation comprises:

obtaining a preliminary foreground object image;

applying a filter to reduce noise in the preliminary foreground object image to generate a filtered image;

detecting a filled convex hull region in the preliminary foreground object image;

adding extra pixels from the filtered image to the preliminary foreground object image to generate a temporary image;

discarding pixels in the temporary image outside the filled convex hull region to generate a noisy convex hull image; and

closing gaps in foreground regions of the noisy convex hull image to generate the foreground object image.

8. The method of claim 1, wherein the predictive model comprises an adaptive Gaussian Mixture Model.

9. A non-transitory computer-readable storage medium storing instructions for generating a composite output video from an input video having a sequence of frames, the instructions when executed by a processor causing the processor to perform steps comprising:

selecting from the sequence of frames, a range of frames for processing;

performing a foreground/background segmentation on each of the frames in the range of frames to extract a plurality of candidate foreground object images based on a predictive model, each of the candidate foreground object images comprising a representation of a foreground object depicted in a corresponding video frame with background pixels subtracted;

selecting, based on an image metric, a selected foreground object image from the plurality of candidate foreground object images;

storing the selected foreground object images to a foreground object list; and

overlaying the foreground object images in the foreground object list on a current video frame to generate a composite video frame.

10. The non-transitory computer-readable storage medium of claim 9, wherein selected the selected foreground object image comprises:

determining an image quality metric for each of the candidate foreground object images; and

determining that the selected foreground object image has a highest quality metric.

11. The non-transitory computer-readable storage medium of claim 9, wherein selecting the selected foreground object image comprises:

determining a face detection likelihood on each of the candidate foreground object images; and

determining that the selected foreground object image has a highest face detection likelihood.

12. The non-transitory computer-readable storage medium of claim 9, wherein selecting the selected foreground object image comprises:

determining a motion parameter for each of the candidate foreground object images; and

determining that the selected foreground object image has a motion parameter best matching a predefined motion criteria.

13. The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed further cause the processor to perform a step of:

training the predictive model based on a plurality of training video frames, the predictive model to predict whether a pixel in a given video frame belongs to a background model or the foreground object.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions when executed further cause the processor to perform a step of:

determining if a frame number of the current video frame is a multiple of a predefined integer X; and

responsive to the frame number of the current video frame being the multiple of the predefined integer X, updating the predictive model.

15. The non-transitory computer-readable storage medium of claim 9, wherein performing the foreground/background segmentation comprises:

obtaining a preliminary foreground object image;

applying a filter to reduce noise in the preliminary foreground object image to generate a filtered image;

detecting a filled convex hull region in the preliminary foreground object image;

adding extra pixels from the filtered image to the preliminary foreground object image to generate a temporary image;

discarding pixels in the temporary image outside the filled convex hull region to generate a noisy convex hull image; and

closing gaps in foreground regions of the noisy convex hull image to generate the foreground object image.

16. The non-transitory computer-readable storage medium of claim 9, wherein the predictive model comprises an adaptive Gaussian Mixture Model.

17. A system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for generating a composite output video from an input video having a sequence of frames, the instructions when executed by the one or more processors causing the one or more processors to perform steps comprising;

selecting from the sequence of frames, a range of frames for processing;

performing a foreground/background segmentation on each of the frames in the range of frames to extract a plurality of candidate foreground object images based on a predictive model, each of the candidate foreground object images comprising a representation of a foreground object depicted in a corresponding video frame with background pixels subtracted;

selecting, based on an image metric, a selected foreground object image from the plurality of candidate foreground object images;

storing the selected foreground object images to a foreground object list; and

overlaying the foreground object images in the foreground object list on a current video frame to generate a composite video frame.

18. The system of claim 17, wherein selected the selected foreground object image comprises:

determining an image quality metric for each of the candidate foreground object images; and

determining that the selected foreground object image has a highest quality metric.

19. The system of claim 17, wherein selecting the selected foreground object image comprises:

determining a face detection likelihood on each of the candidate foreground object images; and

determining that the selected foreground object image has a highest face detection likelihood.

20. The system of claim 17, wherein selecting the selected foreground object image comprises:

determining a motion parameter for each of the candidate foreground object images; and

determining that the selected foreground object image has a motion parameter best matching a predefined motion criteria.