Moving video image estimation system

• US 5,214,504 A
• Filed: 01/31/1991
• Issued: 05/25/1993
• Est. Priority Date: 01/31/1990
• Status: Expired due to Term

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Naoi, Satoshi, Toriu, Takashi
• Primary Examiner(s): Britton, Howard W.
• Application Number:
  US07/648,986
• Time in Patent Office: 845 Days
• Field of Search: 358/105, 358/136
• US Class Current: 375/240.02
• CPC Class Codes: G06F 17/175 of multidimensional data
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 7/0132 the field or frame frequency of the incoming video signal being multiplied by a positive integer, e.g. for flicker reduction

Abstract

A moving video image estimation system wherein based on an original video image of time n and time n+1, the centroid, the principal axix of inertia, the moment about the principal axis of inertia and the moment about the axis perpendicular to the principal axis of inertia are obtained. By using this information, an affine transformation for transforming the original video image at time n to the original video image at time n+1 is obtained. Based on the infinitesimal transformation (A), {exp (At), and exp (A(t-1))} obtained by making the affine transformation continuous with regard to time is executed on the original video image at time n and time n+1. The results are synthesized to perform an interpolation between the frames. {exp (a(t-1))} is applied to the original video system time n+1. The video image after time n+1 is thereby protected.

Claims

1. A moving video image estimation system comprising:

moment calculating means for calculating primary and secondary moments of a video image;

affine transformation calculating means utilizing an output of said moment calculating means to determine affine transformation (G) by which a first frame video image (I₁) is geometrically transformed into a second frame video image (I₂);

affine transformation continuing means for multiplying by a constant (t) an infinitesimal transformation (A), given by the relation G=exp(A), and obtaining an exponential transformation {exp(At)} as a continued affine transformation; and

affine transformation executing means for performing said continued affine transformation {exp(At)} to thereby obtain a video image of a time after an arbitrary time (t) from a time corresponding to said first frame video image.

2. The moving video image estimation system according to claim 1 , wherein said affine transformation executing means executes said continued affine transformation {exp(At)} on the first frame video image (I₁) to thereby obtain a video image occurring after an arbitrary time (t) from a time corresponding to said first frame video image.

3. The moving video image estimation system according to claim 1, wherein

moments of said first and second frame video images are calculated to extract characteristics of respective video images,

the affine transformation for coupling the characteristics of said first and second frame video images is determined,

the exponential transformation {exp(At)} is obtained as the continued affine transformation after the infinitesimal transformation (A) of said affine transformation (G) is multiplied by the constant (t) and

said continued affine transformation {exp(At)} is executed on said first frame video image (I₁) to thereby assume a video image at the arbitrary time (t).

4. The moving video image estimation system according to claim 1, wherein said affine transformation calculating means calculates an inverse transformation (G^-1) of said affine transformation (G) and said affine transformation executing means executes a transformation of the product {exp(At)}G^-1 of said continued affine transformation and said inverse transformation on said second frame video image (I₂) to thereby obtain a video image after said second frame video image.

5. The moving video image estimation system according to claim 4, wherein

said moment calculating means comprises an input portion for receiving video image data, a video image memory for first storing said first frame video image and finally storing said second frame video image, a first moment memory for storing a moment of said first frame video image and a second moment memory for storing a moment of said second frame video memory;

and

said affine transformation calculating means comprises a first affine transformation determining unit for determining said affine transformation (G) and a second affine transformation determining unit for calculating the inverse transformation (G^-1) of said affine transformation.

6. The moving video image estimation system according to claim 1 , wherein said affine transformation executing means executes a transformation {exp(A (t-1))} on said second frame video image (I₂) by using said continued affine transformation {exp(At)} to thereby obtain a video image after said second frame video image.

7. The moving video image estimation system according to claim 6, wherein said moment calculating means comprises an input portion for receiving video image data, a video image memory for first storing said first frame video image and finally storing said second frame video image, a first moment memory for storing a moment of said first frame video image and a second moment memory for storing a moment of said second frame video memory;

and

said affine transformation calculating means comprises an affine transformation determining unit for determining said affine transformation (G).

8. An moving video image estimation system comprising:

moment calculating means for calculating primary and secondary moments of a video image;

affine transformation calculating means utilizing an output of said moment calculating means for determining an affine transformation (G) by which a first frame video image (I₁) is geometrically transformed into a second frame video image (I₂);

first affine transformation continuing means for multiplying by a constant (t) and infinitesimal transformation (A), given by the relation of G=exp(A) with respect to said affine transformation (G) and for obtaining and exponential transformation {exp(At)};

second affine transformation continuing means for multiplying said infinitesimal transformation (A) by another constant (t-1) and obtaining an exponential transformation {exp(A(t-1))}; and

affine transformation executing means for obtaining a video image at an arbitrary time (t) between said first and second frames as a linear sum of results which is provided by executing a transformation {exp(At)}, provided as an output of said first affine transformation continuing means, onto said first frame video image (I₁) and of a result which is provided by executing a transformation {exp(A(t-1)}, provided as an output of said second affine transformation continuing means, onto said second frame video image (I₂).

9. The moving video image estimation system according to claim 8, wherein

an input unit is provided for inputting video image data,

a first video image memory is provided for storing video image data of said first frame,

a second video image memory is provided for storing video image data of said second frame,

a first moment memory is provided for storing a moment of said first frame video image,

a second moment memory is provided for storing a moment of said second frame video image,

a first affine transformation unit is provided for performing a continued affine transformation output from said first affine transformation continuing unit onto the first frame video image stored in said first video image memory,

a second affine transformation unit is provided for performing a continued affine transformation output from said second affine transformation continuing unit onto the second frame video image stored in said second video image memory and

a video image synthesizing unit is provided for calculating a linear sum of outputs of said first and second affine transformation units to thereby obtain a video image at an arbitrary time, wherein said system includes an affine transformation executing unit equivalent to said affine transformation executing means.

10. The moving video image estimation system according to claim 9, wherein

said input unit comprises a television camera for picking up a video image and an analog-to-digital converter for converting an output of said television camera into a digital signal,

said moment calculating unit comprises of a video image projection circuit for obtaining projections to the x axis, the y axis and the oblique axis of video image;

three projection memories for storing projections of the x axis, the y axis and the oblique axis output from said video image projection circuit; and

a moment calculating circuit for calculating a moment of a video image by utilizing outputs of said three projection memories,

said affine transformation calculating unit and said first and second affine transformation continuing units are formed of one portion of a microprocessor,

said first affine transformation unit is composed of a first affine transformation coefficient memory and a first affine circuit,

said second affine transformation unit is composed of a second affine transformation coefficient memory and a second affine transformation circuit and

said video image synthesizing unit is a video image synthesizing circuit.

11. The moving video image estimation system according to claim 9, wherein said moment calculating unit calculates projections of the x axis, the y axis and the oblique axis;

zero-order moment m₀ ;

primary moments m_X and m_Y ;

coordinates X and Y of gravity of a video image;

secondary moments m_XX, m_YY and m_XY ; and

central moments M_XX, M_YY and M_XY.

12. The moving video image estimation system according to claim 9, wherein said affine transformation calculating unit utilizes coordinates of centroid X and Y and central moments M_XX, M_YY and M_XY of said first and second frame video images output from said moment calculating unit to calculate a direction θ

of the principal axis of inertia of each video image, a moment κ

about the axis perpendicular to said principal axis of inertia and a moment λ

about said principal axis of inertia, thereby obtaining matrix element Lij and vector element Bi of matrix ##EQU20## which transforms a coordinate (x, y) into a coordinate (x'"'"', y'"'"') as an affine transformation coefficient.

13. The moving video image estimation system according to claim 9, wherein

said first affine transformation continuing unit determines the time t,

a matrix element hij is calculated in order to calculate the matrix element lij(t) of affine transformation,

the matrix element lij (t) of affine transformation is calculated by the matrix element hij,

a matrix element Pij is calculated in order to calculate a vector element bi(t) of affine transformation and the vector element bi(t) of affine transformation is calculated by using said element Pij.

14. The moving video image estimation system according to claim 9, wherein

said second affine transformation continuing unit determines the time t,

t-1 is substituted for t,

the matrix element hij is calculated in order to calculate a matrix element lij'"'"'(t) of affine transformation,

the matrix element lij'"'"'(t) of affine transformation is calculated by using said element hij,

a matrix element Pij is calculated in order to calculate a vector element bi'"'"'(t) of affine transformation and

the vector element bi'"'"'(t) of affine transformation is calculated by using said element Pij.

15. The moving video image estimation system according to claim 9, wherein

said first affine transformation continuing unit determines the time t,

the matrix element hij is calculated in order to calculate the matrix element lij(t) of affine transformation,

the matrix element lij(t) of affine transformation is calculated by using said matrix element hij,

the matrix element Pij is calculated in order to calculated the vector element bi(t) of affine transformation,

the vector element bi(t) of affine transformation is calculated by using said element Pij,

said second affine transformation continuing unit determines the time t,

t-1 is substituted for t,

the matrix element hij is calculated in order to calculate the matrix element lij'"'"'(t) of affine transformation,

the matrix element lij'"'"'(t) of affine transformation is calculated by using said element hij,

the matrix element Pij is calculated in order to calculate the vector element bi'"'"'(t) of affine transformation,

the vector element bi'"'"'(t) of affine transformation is calculated by using said element Pij,

said affine transformation executing unit executes the affine transformation on said first frame video image by using the elements lij(t) and bi(t) calculated by said first affine transformation continuing unit,

said affine transformation executing unit executes affine transformation on said second frame video image by using the elements lij'"'"'(t) and bi'"'"'(t) calculated by said second affine transformation continuing unit, and

executed results of said two affine transformations are synthesized to obtain an interpolated video image.