Apparatus for calculating optical flow and camera motion using correlation matching and system model in moving image

• US 6,683,677 B2
• Filed: 03/18/2002
• Issued: 01/27/2004
• Est. Priority Date: 12/24/2001
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Kyung Ok, Lim, Young Jae, Chon, Jae Choon, Yang, Young Kyu, Cho, Seong Ik
• Primary Examiner(s): Tarcza, Thomas H.
• Assistant Examiner(s): ANDREA, BRIAN K
• Application Number:
  US10/098,430
• Publication Number:
  US 20030117611A1
• Time in Patent Office: 680 Days
• Field of Search: 356/28, 356/28.5, 382/107, 348/142, 348/700, 348/701, 396/54, 396/55
• US Class Current: 356/28
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

An apparatus for calculating optical flow and camera motion using correlation matching and a system model in a moving image, in which the calculation of the optical flow is executed accurately in real time, even when an image rotation or zoom change has occurred. The apparatus comprises a feature point extracting section for extracting a feature point of a sequence of input images (previous image and current image); an optical flow calculating section for calculating an optical flow by use of the feature point extracted by the feature point extracting section; a first camera motion calculating section for calculating a camera motion by use of the optical flow calculated by the optical flow calculating section; a second camera motion calculating section for eliminating an incorrect optical flow from the optical flow calculated by the optical flow calculating section and again calculating camera motion; an optical flow location estimating section for estimating a location of the optical flow by estimating a distance difference between an estimated feature point location and a current feature point location in the previous image according to the camera motion calculated by the second camera motion calculating section; and a weighted value calculating section for calculating a weighted value according to the camera motion calculated by the second camera motion calculating section and providing the weighted value to the optical flow calculating section.

Claims

1. An apparatus for calculating an optical flow and a camera motion of a camera using correlation matching and a system model in moving images, the apparatus comprising:

feature point extracting means for extracting a feature point from each of a previous image and a current image in a sequence of input images;

optical flow calculating means for calculating an optical flow by determining a maximum edge connecting strength between the feature points extracted by the feature point extracting means;

first camera motion calculating means for calculating a camera motion using the optical flow calculated by the optical flow calculating means;

second camera motion calculating means for eliminating an incorrect optical flow from the optical flow calculated by the optical flow calculating means and calculating a second camera motion;

optical flow location estimating means for estimating a location of the optical flow by estimating a distance difference between an estimated feature point location and a current feature point location in the previous image according to the second camera motion calculated by the second camera motion calculating means; and

weighted value calculating means for calculating a plurality of weighted values according to the second camera motion calculated by the second camera motion calculating means and providing the weighted value to the optical flow calculating means, said weighted values changing according to zoom and optical axis rotation of said camera and including weighted values of correlation matching, location error matching and edge strength matching.

2. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the feature point extracting means comprises:

a SUSAN edge driving portion for extracting an edge image from the sequence of the input images;

and a local max portion for selecting a largest value in a mask size region set in the edge image extracted by the SUSAN edge extracting portion to extract a certain number of feature points.

3. The apparatus for calculating the optical flow and camera motion as claimed in claim 2, wherein the feature points are extracted by the local max portion in accordance with:

$S=\left\{\underset{e}{\sum <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>-{\left(\frac{I<br><br>\left(x,y\right)-I<br><br>\left(x+\mathrm{dx},y+\mathrm{dy}\right)}{T}\right)}^6\right\}/G$

wherein, I is a gray value, T is a threshold value for a difference between the gray values, and G is an edge strength difference between feature point locations of the previous and current images.

4. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the optical flow calculating means comprises:

a combining section for calculating the connecting strength of an edge using the distance difference between the current feature point location and the estimated feature point location in the previous image provided from the optical flow location estimating means, the weighted values of the correlation, location incorrectness, and edge strength matching provided from the weighted value calculating means, and the edge strength difference between the feature points of the previous and current images provided from the feature point extracting means; and

a feature point extracting section for extracting the feature points of the largest connecting strength by use of the connecting strength value provided from the combining section and providing the extracted feature points to the first and second camera motion calculating means.

5. The apparatus for calculating the optical flow and camera motion as claimed in claim 4, wherein the edge connecting strength E is calculated by the combining section in accordance with:

6. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the first camera motion calculating means comprises:

a pseudo inverse matrix calculating section for calculating a constant of a camera projection formula by use of the optical flow provided from the optical flow calculating means; and

a camera motion calculating section for dividing the camera motion by use of the constant of the camera projection formula provided from the pseudo inverse matrix calculating section, and providing the resultant to the second camera motion calculating means.

7. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the second camera motion calculating means comprises:

an optical flow direction and magnitude calculating section for calculating a direction and magnitude of the optical flow by use of a value of the camera motion calculated by the first camera motion calculating means;

an incorrect optical flow eliminating section for eliminating an incorrect optical flow from the optical flow provided by the optical flow calculating means depending upon the direction and magnitude of the optical flow calculated by the optical flow direction and magnitude calculating section; and

a camera motion calculating section for calculating the camera motion by use of the optical flow which is not eliminated by the incorrect optical flow eliminating section, and providing the camera motion to the weighted value calculating means and the optical flow location estimating means, respectively.

8. The apparatus for calculating the optical flow and camera motion as claimed in claim 7, wherein the direction and magnitude of the optical flow are calculated by the optical flow direction and magnitude calculating section in accordance with:

$\left(\begin{array}{c}{x}_{\mathrm{Ave}}\\ y_{\mathrm{Ave}}\end{array}\right)=\left(\begin{array}{c}{x}_t\\ y_t\end{array}\right)+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>t<br><br>\left(\begin{array}{c}-T_{x,y}/Z_t+T_{z,t}<br><br>x_t/Z_t+{\mathrm{\Omega <br><br>}}_{z,t}<br><br>y_t\\ -T_{y,t+1}/Z_t+T_{z,t}<br><br>y_t/Z_t-{\mathrm{\Omega <br><br>}}_{z,t}<br><br>x_t\end{array}\right)$

wherein, Δ

t is a sampling time, T_x, T_y and T_z are x, y and z-axis translation of the camera, respectively, and Ω

_z is z-axis rotation of the camera, x_t, and y_t are locations in the current frame image, and x_Ave and y_Ave are standard locations.

9. The apparatus for calculating the optical flow and camera motion as claimed in claim 8, wherein if a direction θ

of a standard optical flow and a distance Δ

between x_Ave and y_Ave are outside a predetermined range, the incorrect optical flow eliminating section determines the calculated optical flow as a mismatched optical flow to eliminate the incorrect optical flow.

10. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the optical flow location estimating means estimates the camera motion, depending upon the optical flow from which the incorrect optical flow is eliminated, and the camera motion provided from the second camera motion calculating means, in accordance with:

$\left(\begin{array}{c}{\hat{x}}_{t+1}\\ {\hat{y}}_{t+1}\end{array}\right)=\left(\begin{array}{c}{x}_t\\ y_t\end{array}\right)+\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>t<br><br>\left(\begin{array}{c}-{\hat{T}}_{x,t+1}/Z_t+{\hat{T}}_{z,t+1}<br><br>x_t/Z_t+{\widehat{\mathrm{\Omega <br><br>}}}_{z,t+1}<br><br>y_t\\ -{\hat{T}}_{y,t+1}/Z_t+{\hat{T}}_{z,t+1}<br><br>y_t/Z_t-{\mathrm{\Omega <br><br>}}_{z,t+1}<br><br>x_t\end{array}\right)$

wherein, {circumflex over (T)}_x,t+1, {circumflex over (T)}_y,t+1, {circumflex over (T)}_x,t+1, {circumflex over (Ω

)}_x,t+1 are camera motion estimated.

11. The apparatus for calculating the optical flow and camera motion as claimed in claim 1, wherein the weighted value calculating means comprises:

a correlation matching weighted value calculating section for calculating the correlation matching weighted value by use of the camera motion in which the incorrect optical flow has been eliminated by the second camera motion calculating means; and

a weighted value normalized calculating section for executing smoothing of a predetermined location estimated constant weighted value, an edge strength constant weighted value, and the calculated correlation matching weighted value, and providing each of the smoothed values to the optical flow calculating means.

12. The apparatus for calculating the optical flow and camera motion as claimed in claim 11, wherein the correlation matching weighted value is calculated by the correlation matching weighted value calculating section in accordance with:

13. The apparatus for calculating the optical flow and camera motion as claimed in claim 11, wherein the smoothing of the predetermined location estimated constant weighted value, the edge strength constant weighted value, and the calculated correlation matching weighted value are executed by the weighted value normalized calculating section in accordance with:

$W_G=\frac{W_1}{W_1+W_{\mathrm{Cl},\mathrm{Init}}+W_{S,\mathrm{Init}}}$$W_{\mathrm{Cl}}=\frac{W_{\mathrm{Cl},\mathrm{Init}}}{W_1+W_{\mathrm{Cl},\mathrm{Init}}+W_{S,\mathrm{Init}}}$$W_S=\frac{W_{S,\mathrm{Init}}}{W_1+W_{\mathrm{Cl},\mathrm{Init}}+W_{S,\mathrm{Init}}}$

wherein, W_Cl,Init and W_S,Init are the location estimated weighted value and the edge strength constant weighted value, respectively.