Method for determining sensor motion and scene structure and image processing system therefor

• US 5,259,040 A
• Filed: 10/04/1991
• Issued: 11/02/1993
• Est. Priority Date: 10/04/1991
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Hanna, Keith J.
• Primary Examiner(s): Couso, Jose L.
• Application Number:
  US07/771,715
• Time in Patent Office: 760 Days
• Field of Search: 382/39, 382/41, 382/44, 382/54, 382/56, 358/133, 358/135, 358/136, 358/105, 358/125
• US Class Current: 382/107
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/543 using regions

Abstract

The invention is a method for determining the motion of an image sensor through a scene directly from brightness derivatives of an image pair. A global image sensor motion constraint is combined with the local brightness constancy constraint to relate local surface models with the global image sensor motion model and local brightness derivatives. In an iterative process, the local surface models are refined using the image sensor motion as a constraint, and then the image sensor motion model is refined using the local surface models as constraints. The analysis is performed at multiple resolutions to enhance the speed of the process.

Claims

1. A digital image-processing method employing a feedback loop, which method, in response to a sequence of successive pairs of image frames in digital form which image frames depict a globally-moving scene in which a certain structural image characteristic of said scene changes with both said global motion and any intrinsic change in local structure of said scene, determines at least one of (1) the actual value of said global motion and (2) the actual local changes in said scene'"'"'s intrinsic structure;

wherein said feedback loop comprises estimate means, synthesize-flow means having an output of said estimate means applied as an input thereto, and warp means having an output of said synthesize-flow means applied as a warp-control input thereto and an output therefrom applied as a first input to said estimate means; and

wherein said method comprises the steps of;

a) applying, at a given resolution, the image structure of one image frame of each of said successive pairs through said warp means as said first input to said estimate means;

b) applying, at said given resolution, the image structure of the other image frame of each of said successive pairs directly as a second input to said estimate means;

c) initially storing in said estimate means respective preliminary estimates of the values of said global motion and of said intrinsic changes in said local structure

d) employing said estimate means to modify a certain one of the estimates of the value of said global motion and of said intrinsic changes in said local structure currently stored therein to that stored estimate of said certain one of the estimates that results in the difference between the measured changes in image structure defined by the first and second inputs actually applied to said estimate means by steps (a) and (b) and a prediction of these changes in image structure computed from said stored estimates being minimized, thereby deriving as an output from said estimate means a modified stored estimate of certain one of said stored estimates and the currently-stored estimate of said other of said stored estimates;

e) employing said synthesize-flow means to derive a warp-control input to said warp means determined by said output from said estimate means that is applied as an input to said synthesize-flow means, whereby the image structure of said one image frame of each of said successive pairs that is applied through said warp means as said first input to said estimate means by step (a) is warped in accordance with said warp-control input to said warp means to closer correspond to the image structure of said other image frame of each of said successive pairs applied directly as said second input to said estimate means by step (b);

f) employing said estimate means to modify the other of the estimates of the value of said global motion and of said intrinsic changes in said local structure currently stored therein to that stored estimate of said other of the estimates that results in the difference between the measured changes in image structure defined by the first and second inputs actually applied to said estimate means by steps (a) and (b) and a prediction of these changes in image structure computed from said stored estimates being minimized, thereby deriving as an output from said estimate means a modified stored estimate of said other of said stored estimates and the currently-stored estimate of said certain one of said stored estimates; and

g) repeating step (e).

2. In the method defined in claim 1, wherein:

said global motion is said certain one of said stored estimates.

3. In the method defined in claim 1, wherein:

said intrinsic changes in said local structure is said certain one of said stored estimates.

4. In the method defined in claim 1, further comprising the step of:

h) iteratively repeating steps (c) to (g) until said modified stored estimate of said certain one of said stored estimates derived in a step (d) iteration is substantially the same as the currently-stored estimate in that step (d) iteration, or until a given maximum number of iterations have taken place.

5. In the method defined in claim 4, wherein said given resolution of the image structure of said one image frame and said other image frame of each of said successive pairs is lower than an initial resolution of the image structure of said one image frame and said other image frame of each of said successive pairs;

comprising the further steps of;

i) substituting a higher-resolution image structure of said one image frame and said other image frame of each of said successive pairs than said given-resolution image structure for said given-resolution image structure; and

(j) repeating steps (a) to (h) for said substituted higher-resolution image structure of said one image frame and said other image frame of each of said successive pairs.

6. In the method defined in claim 1, further comprising the step of:

h) iteratively repeating steps (d) to (g) until both said modified stored estimate of said certain one of said stored estimates derived in a step (d iteration is substantially the same as the currently-stored estimate in that step (d) iteration and said modified stored estimate of said other of said stored estimates derived in a step.(f) iteration is substantially the same as the currently-stored estimate in that step.(f) iteration, or until a given maximum number of iterations have taken place.

7. In the method defined in claim 6, wherein said given resolution of the image structure of said one image frame and said other image frame of each of said successive pairs is lower than an initial resolution of the image structure of said one image frame and said other image frame of each of said successive pairs;

comprising the further steps of;

i) substituting a higher-resolution image structure of said one image frame and said other image frame of each of said successive pairs than said given-resolution image structure for said given-resolution image structure; and

(j) repeating steps (a) to (h) for said substituted higher-resolution image structure of said one image frame and said other image frame of each of said successive pairs.

8. In the method defined in claim 1, wherein:

said certain structural image characteristic is image-pixel brightness at at said given resolution of said image.

9. Digital image-processing apparatus responsive to a sequence of successive pairs of relatively-high resolution image frames in digital form which image frames depict a globally-moving scene in which a certain structure image characteristic of said scene changes with both said global motion and any intrinsic change in local image structure of said scene, wherein said apparatus comprises:

first and second pyramid processors;

first means including a switch and a frame delay for applying at said relatively-high resolution a certain ordinal one of the two image frames of each of said successive pairs directly as an input to one of said first and second pyramid processors and for applying at said relatively-high resolution the other ordinal one of the two image frames of each of said successive pairs through said frame delay as an input to the other of said first and second pyramid processors, whereby each of said first and second pyramid processors is capable of reducing the relatively-high resolution of each image frame applied to its input to a given relatively-high resolution at its output; and

a feedback loop comprising estimate means having first and second inputs and an output indicative to respective estimates of the value of said global motion and of said intrinsic changes in said local structure stored therein, synthesize-flow means having said output of said estimate means applied as an input thereto, and warp means having an output of said synthesize-flow means applied as a warp-control input thereto and an output therefrom applied as a first input to said estimate means;

second means for applying one of the outputs of said first and second pyramid processors as said first input of said estimate means through said warp means for directly applying the other of said outputs of said first and second pyramid processors as said second input of said estimate means;

wherein said estimate means is capable of using the two current estimates stored therein to first compute from the image structure directly applied to its second input a prediction of the image structure applied to its first input, and then modify a selectable one of said two stored current estimates to minimize the difference between the measured changes in image structure defined by the first and second inputs actually applied to said estimate means and said computed prediction of these changes in image structure;

wherein said synthesize-flow means derives a warp-comtrol input to said warp means, determined by said output from said estimate means, which causes the image structure applied through said warp means as said first input to said estimate means to be warped in accordance with said warp-control input to closer correspond to the image structure applied directly as said second input to said estimate means; and

third means for initially storing respective preliminary estimates of the value of said global motion and of said intrinsic changes in said local structure in said estimate means.

10. The apparatus defined in claim 9, wherein:

each of said first and second processors is capable of selectively providing at its output each image frame applied to its input at said given relatively-low reduced resolution or, alternatively, at a predetermined resolution that is higher than said given relatively-low resolution and is lower than or equal to said relatively-high resolution.