Method and apparatus for video stabilization

• US 20060215036A1
• Filed: 03/25/2005
• Published: 09/28/2006
• Est. Priority Date: 03/25/2005
• Status: Abandoned Application

• Current Assignee: University of Hong Kong (Government of Hong Kong Special Administrative Region), Multivision Intelligent Surveillance Limited (NutryFarm International Ltd.)
• Original Assignee: University of Hong Kong (Government of Hong Kong Special Administrative Region), Multivision Intelligent Surveillance Limited (NutryFarm International Ltd.)
• Inventors: Chow, Kam Pui, Mak, Fuk Sang, Chin, Yuk Lun Francis, Chung, Hing Yip, Leung, Chun Bong
• Application Number:
  US11/090,638
• Publication Number:
  US 20060215036A1
• Time in Patent Office: Days
• US Class Current: 348/208.400
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

A method and an apparatus for video stabilization is disclosed for detecting and eliminating unwanted camera motion from a sequence of video images. Motion vectors are first generated based on sample points using a block matching technique, from which a number of possible camera motions are estimated. Among the estimated camera motions, unwanted or undesirable camera motions are then detected and parameterized. A frame remapping process is then applied to relocate the pixels in the current frame, which acts in opposition to the dislocation of pixels due to unwanted camera motions in order to achieve video stabilization.

Claims

1. A video stabilization apparatus comprising:

memory for storing current frame data representative of a current frame and reference frame data representative of a reference frame; and

one or more processors connected to the memory and operable (a) to retrieve the current frame data and the reference frame data;

(b) to calculate motion vectors based on the current frame data and the reference frame data;

(c) to estimate a set of motion parameters based on the motion vectors;

(d) to select a final motion parameter from the set of motion parameters, and (e) to generate, based on the final motion parameter, output frame data representative of an output frame.

2. A video stabilization apparatus according to claim 1, wherein the one or more processors comprises a single processor for calculating the motion vectors, estimating the set of motion parameters, selecting the final motion parameter, and generating the output frame data based on the final motion parameter.

3. A video stabilization apparatus according to claim 1, wherein the one or more processors comprises a plurality of processing units, wherein each of the processing units is operable to perform at least one of calculating the motion vectors, estimating the set of motion parameters, selecting the final motion parameter, and generating the output frame data.

4. A video stabilization apparatus according to claim 1, wherein the calculating of the motion vectors is based on a set of sample points at predefined positions in the current frame.

5. A video stabilization apparatus according to claim 1, wherein the calculating of the motion vectors is based on a set of sample points at randomly selected positions in the current frame.

6. A video stabilization apparatus according to claim 1, wherein the generating of the output frame data includes remapping pixel coordinates of the current frame data to new coordinates based on the final motion parameter.

7. A video stabilization apparatus according to claim 1, wherein the one or more processors is further for rendering reference frame data based on the output frame data.

8. A video stabilization apparatus according to claim 1, wherein the calculating of the motion vectors is based on a set of sample points in the current frame.

9. A video stabilization apparatus according to claim 8, wherein the calculating of the motion vectors includes:

constructing, for each sample point, a sample block of pixels that includes the sample point; and

performing block matching motion estimation using the current frame data and the reference frame data to obtain the motion vectors.

10. A video stabilization apparatus according to claim 8, wherein the set of motion vectors includes a plurality of sample-point subsets, and wherein the estimating of the set of motion parameters includes checking sample-point subsets to determine whether they are valid sample-point subsets, and calculating the motion parameters based on the valid sample-point subsets.

11. A video stabilization apparatus according to claim 10, wherein a sample-point subset is considered valid if the sample points of the sample-point subset are non-collinear.

12. A method of video stabilization comprising:

storing current frame data representative of a current frame and reference frame data representative of a reference frame;

calculating motion vectors based on the current frame data and the reference frame data;

estimating a set of motion parameters based on the motion vectors;

selecting a final motion parameter from the set of motion parameters; and

generating, based on the final motion parameter, output frame data representative of an output frame.

13. A method of video stabilization according to claim 12, wherein the calculating of the motion vectors is based on a set of sample points at predefined positions in the current frame.

14. A method of video stabilization according to claim 12, wherein the calculating of the motion vectors is based on a set of sample points at randomly selected positions in the current frame.

15. A method of video stabilization according to claim 12, wherein the generating of the output frame data includes remapping pixel coordinates of the current frame data to new coordinates based on the final motion parameter.

16. A method of video stabilization according to claim 12, further comprising rendering reference frame data based on the output frame data.

17. A method of video stabilization according to claim 12, wherein the calculating of the motion vectors is based on a set of sample points in the current frame.

18. A method of video stabilization according to claim 17, wherein the calculating of the motion vectors includes:

constructing, for each sample point, a sample block of pixels that includes the sample point; and

performing block matching motion estimation using the current frame data and the reference frame data to obtain the motion vectors.

19. A method of video stabilization according to claim 17, wherein the set of motion vectors includes a plurality of sample-point subsets, and wherein the estimating of the set of motion parameters includes checking sample-point subsets to determine whether they are valid sample-point subsets, and calculating the motion parameters based on the valid sample-point subsets.

20. A method of video stabilization according to claim 19, wherein a sample-point subset is considered valid if the sample points of the sample-point subset are non collinear.

21. A method of camera motion estimation comprising:

selecting a subset of sample points from a set of sample points, the subset including at least three non-collinear sample points;

calculating motion parameters based on the selected subset of sample points and motion vectors associated with the selected subset of sample points; and

repeating the selection of the sample points and the calculating of the motion parameters, thereby obtaining a set of camera motion parameters, for no more than a pre-defined number of iterations.

22. A method of camera motion estimation according to claim 21, wherein the repeating is performed for less than the predefined number of iteration if all possible subsets have been exhausted.

23. A method of camera motion estimation according to claim 21, wherein the calculating of motion parameters includes calculating affine motion parameters.

24. A method comprising:

forming a plurality of at least partially remapped video frames, including forming different ones of the plurality of remapped video frames based on different ones of a plurality of camera motion parameters;

determining respective degrees of similarity between the remapped video frames and a reference frame;

selecting as a final camera motion parameter the camera motion parameter associated with the remapped video frame that a highest degree of similarity to the reference frame; and

if the highest degree of similarity obtained during the determining is indicative of an estimated camera motion that does not appear to include unwanted or undesirable motions, setting the final camera motion parameter to have no effect on a frame remapping process.

25. A method of video stabilization comprising:

storing current frame data representative of a current frame and reference frame data representative of a reference frame;

calculating a motion parameter based on the current frame data and the reference frame data;

generating, based on the motion parameter, output frame data representative of an output frame;

determining a degree of similarity between the output frame and the reference frame; and

generating a new reference frame using the output frame data if the degree of similarity is above a predetermined threshold.

26. A method of video stabilization according to claim 25, wherein the determining of the degree of similarity includes sampling data of the output frame for each iteration of the generating of the output frame.

27. A method of video stabilization according to claim 25, wherein the determining of the degree of similarity includes sampling data of the output frame at fixed intervals of the generating of the output frame.