METHOD AND APPARATUS FOR PROCESSING VIDEO SEQUENCES

• US 20100128789A1
• Filed: 04/25/2008
• Published: 05/27/2010
• Est. Priority Date: 05/07/2007
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Huang, Yu, Llach, Joan, Sole, Joel
• Granted Patent: US 9,214,030 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

A method for processing a video sequence having a plurality of frames includes the steps of: extracting features from each of the frames, determining correspondences between the extracted features from two of the frames, estimating motion in the video sequence based on the determined correspondences, generating a background mosaic for the video sequence based on the estimated motion, and performing foreground-background segmentation on each of the frames based on the background mosaic.

Claims

1. A method for processing a video sequence comprised of a plurality of frames, said method comprising the steps of:

extracting a feature from each of said frames;

determining correspondences between said extracted feature from said frames;

estimating motion in said video sequence based on said determined correspondences;

generating an image including majority of background pixels in said frames based on said estimated motion; and

extracting foreground information from each of said frames based on said image.

2. The method of claim 1, wherein said feature extracting step includes extracting a scale invariant feature transform feature.

3. The method of claim 1, wherein said determining step includes checking for temporal consistency of said extracted feature along said frames.

4. The method of claim 1, wherein said estimating step includes using a modified random sample consensus algorithm.

5. The method of claim 1, wherein said generating step includes:

generating a forward warping matrix and a background warping matrix for each of said frames based on said estimated motion;

generating a forward warping error and a backward warping error for each of said frames based on said forward warping matrix and said background warping matrix;

generating a foreground/background mask for each of said frames based on said forward warping error and said backward warping error; and

generating said image by mapping said frames to a common coordinate system.

6. The method of claim 5, wherein said generating step further includes inpainting missing regions of said video sequence from surrounding regions of said video sequence.

7. The method of claim 1, wherein said foreground information extracting step includes extracting said foreground information using a mean shift method.

8. An apparatus for processing a video sequence comprised of a plurality of frames, said apparatus comprising:

means for extracting a feature from each of said frames;

means for determining correspondences between said extracted feature from said frames;

means for estimating motion in said video sequence based on said determined correspondences;

means for generating an image including majority of background pixels in said frames based on said estimated motion; and

means for extracting foreground information from each of said frames based on said image.

9. The apparatus of claim 8, wherein said feature extracting means extracts a scale invariant feature transform feature.

10. The apparatus of claim 8, wherein said determining means checks for temporal consistency of said extracted feature along said frames.

11. The apparatus of claim 8, wherein said estimating means uses a modified random sample consensus algorithm.

12. The apparatus of claim 8, wherein said generating means generates a forward warping matrix and a background warping matrix for each of said frames based on said estimated motion, generates a forward warping error and a backward warping error for each of said frames based on said forward warping matrix and said background warping matrix, generates a foreground/background mask for each of said frames based on said forward warping error and said backward warping error, and generates said image by mapping said frames to a common coordinate system.

13. The apparatus of claim 12, wherein said generating means inpaints missing regions of said video sequence from surrounding regions of said video sequence.

14. The apparatus of claim 8, wherein said foreground information extracting means extracts said foreground information using a mean shift method.

15. An apparatus for processing a video sequence comprised of a plurality of frames, said apparatus including circuitry capable of performing steps comprising:

Extracting a feature from each of said frames;

determining correspondences between said extracted features from said frames;

estimating motion in said video sequence based on said determined correspondences;

generating an image including majority of background pixels in said frames based on said estimated motion; and

extracting foreground information from each of said frames based on said image.

16. The apparatus of claim 15, wherein said feature extracting step includes extracting a scale invariant feature transform feature.

17. The apparatus of claim 15, wherein said determining step includes checking for temporal consistency of said extracted features along said frames.

18. The apparatus of claim 15, wherein said estimating step includes using a modified random sample consensus algorithm.

19. The apparatus of claim 15, wherein said generating step includes:

generating a forward warping matrix and a background warping matrix for each of said frames based on said estimated motion;

generating a forward warping error and a backward warping error for each of said frames based on said forward warping matrix and said background warping matrix;

generating a foreground/background mask for each of said frames based on said forward warping error and said backward warping error; and

generating said image by mapping said frames to a common coordinate system.

20. The apparatus of claim 19, wherein said generating step further includes inpainting missing regions of said video sequence from surrounding regions of said video sequence.

21. The apparatus of claim 15, wherein said foreground information extracting step includes extracting said foreground information using a mean shift method.