Video image stabilization and registration

• US 6,459,822 B1
• Filed: 07/26/1999
• Issued: 10/01/2002
• Est. Priority Date: 08/26/1998
• Status: Expired due to Term

• Current Assignee: National Aeronautics and Space Administration (Government of the United States of America)
• Original Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Hathaway, David H., Meyer, Paul J.
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/364,919
• Time in Patent Office: 1,163 Days
• Field of Search: 382/286, 382/173, 382/219, 382/220, 382/276, 382/278, 382/279, 382/296, 382/298, 382/299, 382/300, 345/436-439, 345/132, 345/138, 345/502, 345/648, 345/649, 345/660, 345/672, 358/413, 358/408, 358/430, 375/240.16
• US Class Current: 382/296
• CPC Class Codes: G06T 3/02 Affine transformations for image registration G06T3/147; for image mosaicing G06T3/4038
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 5/14 Picture signal circuitry for video frequency region cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00

Abstract

A method of stabilizing and registering a video image in multiple video fields of a video sequence provides accurate determination of the image change in magnification, rotation and translation between video fields, so that the video fields may be accurately corrected for these changes in the image in the video sequence. In a described embodiment, a key area of a key video field is selected which contains an image which it is desired to stabilize in a video sequence. The key area is subdivided into nested pixel blocks and the translation of each of the pixel blocks from the key video field to a new video field is determined as a precursor to determining change in magnification, rotation and translation of the image from the key video field to the new video field.

Claims

1. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

subdividing a selected area of a first video field into nested pixel blocks including. multiple levels of progressively smaller pixel block subdivisions, the area containing the video image;

determining horizontal and vertical translation of each of the pixel blocks in each of the pixel block subdivision levels from the first video field to a second video field; and

determining change in magnification of the image from the first video field to the second video field by dividing each of relative translations of pairs of the pixel blocks by respective distances between centers the pairs of pixel blocks.

2. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the'"'"'steps of:

subdividing a selected area of a first video field into nested pixel blocks including multiple levels of progressively smaller pixel block subdivisions, the area containing the video image;

determining horizontal and vertical translation of each of the pixel blocks in each of the pixel block subdivision levels from the first video field to a second video field; and

determining rotation of the image from the first video field to the second video field by calculating the inverse tangent of each of relative translations of pairs of the pixel blocks divided by respective distances between centers of the pixel block pairs.

3. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

subdividing a selected area of a first video field into nested pixel blocks including multiple levels of progressively smaller pixel block subdivisions, the area containing the video image;

determining horizontal and vertical translation of each of the pixel blocks in each of the pixel block subdivision levels from the first video field to a second video field; and

determining translation of the image from the first video field to the second video field by determining a change in magnification of the image from the first video field to the second video field, determining rotation of the image from the first video field to the second video field, correcting the horizontal and vertical translations of each of the pixel blocks in the smallest pixel block subdivision for the change in magnification and rotation of the image from the first video field to the second video field, and averaging the corrected horizontal and vertical pixel block translations.

4. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field; and

calculating a change in magnification of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing a relative horizontal displacement between a pair of the pixel blocks in one of the rows of pixel blocks by a distance between respective centers of the pair of pixel blocks.

5. The method according to claim 4, wherein the dividing step is performed for each pair of the pixel blocks in each row of pixel blocks in the area.

6. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field;

, and

calculating a change in magnification of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing a relative vertical displacement between a pair of the pixel blocks in one of the columns of pixel blocks by a distance between respective centers of the pair of pixel blocks.

7. The method according to claim 6, wherein the dividing step is performed for each pair of the pixel blocks in each column of pixel blocks in the area.

8. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field; and

calculating a change in magnification of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing relative vertical displacements between respective pairs of pixel blocks in each column of pixel blocks by respective distances between centers of the pixel block column pairs, dividing relative horizontal displacements between respective pairs of pixel blocks in each row of pixel blocks by respective distances between centers of the pixel block row pairs, and averaging together the results of both of the dividing steps.

9. The method according to claim 8, further comprising the step of weighting the results of both of the dividing steps prior to the averaging step.

10. The method according to claim 9, wherein the weighting step further comprises assigning greater weight to results of the dividing steps having relatively greater respective distances between the pixel block pair centers.

11. The method according to claim 9, further comprising the step of excluding from the averaging and weighting steps results of the dividing steps which are more than one standard deviation from the average of the results of the dividing steps.

12. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field, and

calculating rotation of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing a relative horizontal displacement between a pair of the pixel blocks in one of the columns of pixel blocks by a distance between respective centers of the pair of pixel blocks.

13. The method according to claim 12, wherein the dividing step is performed for each pair of the pixel blocks in each column of pixel blocks in the area.

14. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field; and

calculating rotation of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing a relative vertical displacement between a pair of the pixel blocks in one of the rows of pixel blocks by a distance between respective centers of the pair of pixel blocks.

15. The method according to claim 14, wherein the dividing step is performed for each pair of the pixel blocks in each row of pixel blocks in the area.

16. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field; and

calculating rotation of the image from the first video field to the second video field,

wherein the calculating step is performed by dividing relative vertical displacements between respective pairs of pixel blocks in each row of pixel blocks by respective distances between centers of the pixel block row pairs, dividing relative horizontal displacements between respective pairs of pixel blocks in each column of pixel blocks by respective distances between centers of the pixel block column pairs, and averaging together the results of both of the dividing steps.

17. The method according to claim 16, further comprising the step of weighting the results of both of the dividing steps prior to the averaging step.

18. The method according to claim 17, wherein the weighting step further comprises assigning greater weight to results of the dividing steps having relatively greater respective distances between the pixel block pair centers.

19. The method according to claim 17, further comprising the step of excluding from the averaging and weighting steps results of the dividing steps which are more than one standard deviation from the average of the results of the dividing steps.

20. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

dividing an area of a first video field of the video sequence into rows and columns of pixel blocks, the area containing the image;

determining a horizontal and vertical translation of each of the pixel blocks from the first video field to a second video field;

utilizing the pixel block horizontal and vertical displacements to calculate a change in magnification of the image from the first video field to the second video field;

utilizing the pixel block horizontal and vertical displacements to calculate rotation of the image from the first video field to the second video field;

correcting the horizontal and vertical displacements of the pixel blocks for the change in magnification and rotation from the first video field to the second video field;

averaging the corrected horizontal displacements of the pixel blocks to determine horizontal displacement of the image from the first video field to the second video field; and

averaging the corrected vertical displacements of the pixel blocks to determine vertical displacement of the image from the first video field to the second video field.

21. A method of stabilizing a video image of interest displayed in multiple video fields of a video sequence, the method comprising the steps of:

determining a change in magnification of the image from a first video field to a second video field of the video sequence;

determining rotation of the image from the first video field to the second video field;

determining translation of the image from the first video field to the second video field;

de-magnifying a third video field of the video sequence using the opposite of the change in magnification of the image from the first video field to the second video field;

de-rotating the third video field using the opposite of the rotation of the image from the first video field to the second video field; and

de-translating the third video field using the opposite of the translation of the image from the first video field to the second video field.

22. The method according to claim 21, further comprising the step of determining a change in magnification, rotation and translation of the image from the first video field to the de-magnified, de-rotated and de-translated third video field.

23. The method according to claim 22, further comprising the steps of:

determining a change in magnification of the image from the first video field to the third video field by adding the change in magnification of the image from the first video field to the second video field to the change in magnification of the image from the first video field to the de-magnified third video field;

determining rotation of the image from the first video field to the third video field by adding the rotation of the image from the first video field to the second video field to the rotation of the image from the first video field to the derotated third video field; and

determining translation of the image from the first video field to the third video field by adding the translation of the image from the first video field to the second video field to the translation of the image from the first video field to the de-translated third video field.