Three-frame technique for analyzing two motions in successive image frames dynamically

• US 5,067,014 A
• Filed: 12/13/1990
• Issued: 11/19/1991
• Est. Priority Date: 01/23/1990
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Bergen, James R., Hingorani, Rajesh, Peleg, Shmuel, Burt, Peter J.
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Fox, David
• Application Number:
  US07/624,343
• Time in Patent Office: 341 Days
• Field of Search: 382/6, 382/54, 358/105, 358/109, 358/125, 358/136, 364/516, 342/64, 342/76
• US Class Current: 382/107
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

An iterative process, implemented by a feedback loop, responds to all the image data in the respective analysis regions of three consecutive frames of a motion picture, to provide, after a plurality of cycles of operation thereof, an accurate estimation of the motion of either one or both of two differently moving patterns defined by the image data of these respective analysis regions. The analysis region of each frame is preferably large, and may occupy the entire frame area of a frame. The type of differently moving patterns include (1) separation by a motion bounty, (2) overlapping transparent surface in motion, (3) "picket fence" motion (4) masking of a small and/or low-contrast pattern by a dominant pattern, and (5) two-component aperture effects. Also, the operation of this iterative process inherently accurately estimates the motion of image data defining a single moving pattern.

Claims

1. A feedback loop for iteratively motion-processing image-data pixels within respective analysis regions of a first-occurring frame, a second-occurring frame, and a third-occurring frame of a motion-picture during each of successive cycles of operation thereof;

wherein the image-data pixels within said analysis regions of these motion-picture frames may define first and second differently-moving patterns;

said feedback loop comprising;

first warp means for shifting the position of said first of said moving patterns within said first-occurring frame analysis region toward the position of said first of said moving patterns within said second-occurring frame analysis region by an amount determined by the value of a warp-control signal applied to said first warp means during any odd cycle of iterative operation of said feedback loop;

second warp means for shifting the position of said first of said moving patterns within said second-occurring frame analysis region toward the position of said first of said moving patterns within said third-occurring frame analysis region by an amount determined by the value of said warp-control signal applied to said second warp means during that odd cycle of iterative operation of said feedback loop;

first computation means coupled to said first warp means for performing a predetermined computation on corresponding image-data pixel values of said second-occurring frame and said shifted-pattern first-occurring frame for deriving first computation image data;

second computation means coupled to said second warp means for performing said predetermined computation on corresponding image-data pixel values of said third-occurring frame and said shifted-pattern second-occurring frame for deriving second computation image data; and

motion estimation means, responsive to both said first and said second computation image data derived during that odd cycle of iterative operation, for computing a first warp-control-signal value for application to said first and second warp means during the immediately following even cycle of iterative operation, which first computed value results in the position of said second of said moving patterns within said respective first-occurring and second-occurring frame analysis regions being shifted toward the position of said second of said moving patterns within said respective second-occurring and third-occurring frame analysis regions by an amount determined by said first computed value.

2. The feedback loop defined in claim 1, wherein:

said motion estimation means is responsive to both said first and said second computation image data derived during any even cycle of iterative operation for computing a second warp-control-signal value for application to said first and second warp means during the immediately following odd cycle of iterative operation, which second computed value results in the position of said first of said moving patterns within said respective first-occurring and second-occurring frame analysis regions being shifted closer to the position of said first of said moving patterns within said respective second-occurring and third-occurring frame analysis regions than said first of said moving patterns was shifted during the immediately preceding odd cycle of iterative operation.

3. The feedback loop defined in claim 2, wherein:

said warp-control-signal applied to said first and second warp means during the first cycle of iterative operation has any arbitrary value that results in the position of said first of said moving patterns within said respective first-occurring and toward the position of said first of said moving patterns within said respective second-occurring and third-occurring frame analysis regions.

4. The feedback loop defined in claim 2, wherein:

said motion estimation means is responsive to both said first and said second computation image data derived during each odd cycle of iterative operation above said first cycle for computing a particular first warp-control-signal value for application to said first and second warp means during the immediately following even cycle of iterative operation, which particular first computed value results in the position of said second of said moving patterns within said respective first-occurring and second-occurring frame analysis regions being shifted closer to the position of said second of said moving patterns within said respective second-occurring and third-occurring frame analysis regions than said second of said moving patterns was shifted during the immediately preceding even cycle of iterative operation.

5. The feedback loop defined in claim 4, wherein:

said warp-control-signal applied to said first and second warp means during the first cycle of iterative operation has any arbitrary value that results in the position of said first of said moving patterns within said respective first-occurring and toward the position of said first of said moving patterns within said respective second-occurring and third-occurring frame analysis regions.

6. The feedback loop defined in claim 5, wherein said first-occurring, second-occurring, and third-occurring frames comprise a group of the current three consecutive frames of the ongoing successive frames of said motion picture occurring in real time, whereby the three frames comprising said group are updated by one frame every frame period of said motion picture.

7. The feedback loop defined in claim 5, wherein each of the respective analysis regions of said first-occurring, said second-occurring frame, and said third-occurring frames occupies substantially the entire frame area of a frame.

8. The feedback loop defined in claim 1, wherein said first-occurring, second-occurring, and third-occurring frames comprise a group of the current three consecutive frames of the ongoing successive frames of said motion picture occurring in real time, whereby the three frames comprising said group are updated by one frame every frame period of said motion picture.

9. The feedback loop defined in claim 1, wherein each of the respective analysis regions of said first-occurring, said second-occurring frame, and said third-occurring frames occupies substantially the entire frame area of a frame.

10. The feedback loop defined in claim 1, wherein:

said first computation means comprises first subtraction means; and

said second computation means comprises second subtraction means.

11. The feedback loop defined in claim 1, wherein:

said first computation means consists of first subtraction means, whereby said first computation image data constitutes first difference image data; and

said second computation means consists of second subtraction means, whereby said second computation image data constitutes second difference image data.

12. A method for iteratively motion-processing image-data pixels within respective analysis regions of a first-occurring frame, a second-occurring frame, and a third-occurring frame of a motion-picture during each of successive cycles of operation thereof;

wherein the image-data pixels within said analysis regions of these motion-picture frames may define a first moving pattern having a motion p and a second moving pattern having a motion q different from p;

said method comprising the steps of;

a) setting the initial estimate of motion p of said first moving pattern at p_O ;

b) shifting the position of said first moving pattern within each of said respective first-occurring and second-occurring frame analysis regions toward the position of said first moving pattern within each of said respective second-occurring and third-occurring frame analysis regions by an amount in accordance with the latest estimate p_n of the motion p of said first moving pattern (where n indicates the ordinal number of that iterative cycle of operation that derived that estimate), and said estimate p_n of motion p has a first certain accuracy;

c) performing a predetermined computation on corresponding image-data pixel values of said second-occurring frame and said step (b) shifted-pattern first-occurring frame for deriving step (c) first computation image data D₁, and performing said predetermined computation on corresponding image-data pixel values of said third-occurring frame and said step (b) shifted-pattern second-occurring frame for deriving step (c) second computation image data D₂ ;

d) using step (c) computation image data D₁ and D₂ to derive an estimate q_n+1 of the motion q of said second moving pattern with a second certain accuracy which is higher than the accuracy of the estimate of motion q made by any preceding cycle of iterative operation;

e) shifting the position of said second moving pattern within each of said respective first-occurring and second-occurring frame analysis regions toward the position of said second moving pattern within each of said respective second-occurring and third-occurring frame analysis regions by an amount in accordance with the estimate q_n+1 of the motion q of said second moving pattern;

f) performing said predetermined computation on corresponding image-data pixel values of said second-occurring frame and said step (e) shifted-pattern first-occurring frame for deriving step (f) first computation image data D₁, and performing said predetermined computation on corresponding image-data pixel values of said third-occurring frame and said step (e) shifted-pattern second-occurring frame for deriving step (f) second computation image data D₂ ; and

g) using step (f) difference image data D₁ and D₂ to derive an estimate p_n+2 of the motion p of said first moving pattern with an accuracy which is higher than the first certain accuracy of the estimate p_n of motion p.

13. The method defined in claim 12, comprising the additional step of:

h) repeating, in order, at least some of steps (b) to (h).

14. The method defined in claim 12, wherein the initial estimate p₀ of motion p of said first moving pattern, set in step (a), has an arbitrary value.

15. The method defined in claim 12, wherein said first-occurring, second-occurring, and third-occurring frames comprise a group of the current three consecutive frames of the ongoing successive frames of said motion picture occurring in real time, whereby the three frames comprising said group are updated by one frame every frame period of said motion picture.

16. The method defined in claim 12, wherein each of the respective analysis regions of said first-occurring, said second-occurring frame, and said third-occurring frames occupies substantially the entire frame area of a frame.

17. The method defined in claim 12, wherein:

step c) consists of the step of subtracting corresponding image-data pixel values of said second-occurring frame and said step (b) shifted-pattern first-occurring frame for deriving step (c) first difference image data D₁, and subtracting corresponding image-data pixel values of said third-occurring frame and said step (b) shifted-pattern second-occurring frame for deriving step (c) second difference image data D₂ ; and

step f) consists of the step of subtracting corresponding image-data pixel values of said second-occurring frame and said step (e) shifted-pattern first-occurring frame for deriving step (f) first difference image data D₁, and subtracting corresponding image-data pixel values of said third-occurring frame and said step (e) shifted-pattern second-occurring frame for deriving step (f) second computation image data D₂.

18. The method defined in claim 17, comprising the additional step of:

h) repeating, in order, at least some of steps (b) to (h).