DIGITAL PROCESSING METHOD AND SYSTEM FOR DETERMINATION OF OPTICAL FLOW

• US 20130259317A1
• Filed: 05/30/2013
• Published: 10/03/2013
• Est. Priority Date: 10/15/2008
• Status: Active Grant

• Current Assignee: Cheytec Technologies, LLC
• Original Assignee: Spinella Ip Holdings Inc
• Inventors: Gaddy, William L.
• Granted Patent: US 8,712,095 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/207 for motion estimation over a hierarchy of resolutions multi-resolution motion estimation or hierarchical motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/53
• CPC Class Codes: G06T 7/269 using gradient-based methods

Abstract

A method and system for determining an optical flow field between a pair of images is disclosed. Each of the pair of images is decomposed into image pyramids using a non-octave pyramid factor. The pair of decomposed images is transformed at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant. Discrete-time derivatives of the second derivative image representations are estimated. An optical flow estimation process is applied to the discrete-time derivatives to produce a raw optical flow field. The raw optical flow field is scaled by the non-octave pyramid factor. The above-cited steps are repeated for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein spatiotemporal gradient estimations are warped by a previous raw optical flow estimation.

Claims

1. A method for determining an optical flow field between a pair of images, comprising the steps of:

(a) decomposing each of the pair of images into image pyramids;

(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;

(c) estimating discrete-time derivatives of the second derivative image representations;

(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;

(e) scaling the raw optical flow field; and

(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein each of the pair of decomposed images of step (b) is warped by using the motion vectors estimated in a previously scaled optical flow estimation.

2. The method of claim 1, further comprising the step of (g) transforming the second derivative image representations from color-space to a scalar color-distance space.

3. The method of claim 2, wherein step (g) is implemented using the Laplacian of the vector subtraction of color-distances.

4. The method of claim 3, wherein the Laplacian operator is applied to intensities of all color channels of the second-derivative image representations separately.

5. The method of claim 1, further comprising employing a pre-scaling step via bilinear interpolation to ensure that the highest resolution pyramid scale devolves to integer image sizes of (W′

,H′

), wherein W′ and

H′

are given by

W′

=floor(W/N)×

pyramid_factor^N

H′

=floor(W/N)×

pyramid_factor^N.

6. The method of claim 1, wherein the first pyramid scale is the lowest resolution pyramid scale and another pyramid scale is next higher resolution pyramid scale.

7. The method of claim 1, wherein step (c) is accomplished using a spatiotemporal gradient estimation method.

8. The method of claim 1, wherein the optical flow estimation process employs a least-squares-fit method.

9. The method of claim 1, further comprising the step of processing the raw optical flow field with at least one non-linear filter that imposes a non-linear, piecewise-linearity constraint on the optical flow field.

10. The method of claim 9, wherein the at least one non-linear filter is at least one of a total variation minimization (L1 and L2 norm) filter, an anisotropic diffusion filter, an elliptic steerable filter that reacts to local image structures, and a kernel regression filter.

11. The method of claim 1, further comprising the steps of performing multiple-pixel motion-compensation and a sum-of-square-differences calculation on the raw optical flow field to check inverse consistency of the raw optical flow field on a per-pixel-sample basis.

12. The method of claim 1, further comprising the step of applying a 2D structure tensor eigensystem analysis to the discrete-time derivatives to provide a feedback mechanism for selecting a pyramid scaling factor at subsequent lower resolution pyramid scales.

13. The method of claim 12, wherein when small eigenvalues below a threshold percentage of pixels for a single pyramid scale in excess of other pyramid scales is encountered during the 2D structure tensor eigensystem analysis, a finer pyramid scale is used for at least one of the subsequent pyramid scales.

14. The method of claim 1, wherein the raw optical field flow is processed in reverse order.

15. The method of claim 14, wherein a motion vector is used to sample the second image of the pair of images in five or more locations of a Robert'"'"'s Cross configuration.

16. The method of claim 1, further comprising the steps of processing the raw optical flow field for all pyramid scales by a (non-linear) 3×

3 median filter operation to eliminate single-pixel outliers and processing the filtered raw optical flow field with an L2 norm total variation operator for a variable number of iterations.

17. A non-transitory computer-readable storage medium including instructions that, when accessed by a processing system, cause the processing system to perform operations comprising:

(a) decomposing each of a pair of images into image pyramids;

(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;

(c) estimating discrete-time derivatives of the second derivative image representations;

(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;

(e) scaling the raw optical flow field; and

(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein each of the pair of decomposed images of step (b) is warped by using the motion vectors estimated in a previously scaled optical flow estimation.

18. A method for determining an optical flow field between a pair of images, comprising the steps of:

(a) decomposing each of the pair of images into image pyramids;

(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;

(c) estimating discrete-time derivatives of the second derivative image representations;

(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;

(e) scaling the raw optical flow field; and

(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein each of the pair of decomposed images of step (b) is motion compensated by using motion vectors estimated in a previously scaled optical flow estimation.

19. The method of claim 18, further comprising the step of (g) transforming the second derivative image representations from color-space to a scalar color-distance space.

20. The method of claim 19, wherein step (g) is implemented using the Laplacian of the vector subtraction of color-distances.

21. The method of claim 20, wherein the Laplacian operator is applied to intensities of all color channels of the second-derivative image representations separately.

22. The method of claim 18, further comprising employing a pre-scaling step via bilinear interpolation to ensure that the highest resolution pyramid scale devolves to integer image sizes of (W′

,H′

), wherein W′ and

H′

are given by

W′

=floor(W/N)×

pyramid_factor^N

H′

=floor(W/N)×

pyramid_factor^N.

23. The method of claim 18, wherein the first pyramid scale is the lowest resolution pyramid scale and another pyramid scale is next higher resolution pyramid scale.

24. The method of claim 18, wherein step (c) is accomplished using a spatiotemporal gradient estimation method.

25. The method of claim 18, wherein the optical flow estimation process employs a least-squares-fit method.

26. The method of claim 18, further comprising the step of processing the raw optical flow field with at least one non-linear filter that imposes a non-linear, piecewise-linearity constraint on the optical flow field.

27. The method of claim 26, wherein the at least one non-linear filter is at least one of a total variation minimization (L1 and L2 norm) filter, an anisotropic diffusion filter, an elliptic steerable filter that reacts to local image structures, and a kernel regression filter.

28. The method of claim 18, further comprising the steps of performing multiple-pixel motion-compensation and a sum-of-square-differences calculation on the raw optical flow field to check inverse consistency of the raw optical flow field on a per-pixel-sample basis.

29. The method of claim 18, further comprising the step of applying a 2D structure tensor eigensystem analysis to the discrete-time derivatives to provide a feedback mechanism for selecting a pyramid scaling factor at subsequent lower resolution pyramid scales.

30. The method of claim 29, wherein when small eigenvalues below a threshold percentage of pixels for a single pyramid scale in excess of other pyramid scales is encountered during the 2D structure tensor eigensystem analysis, a finer pyramid scale is used for at least one of the subsequent pyramid scales.

31. The method of claim 18, wherein the raw optical field flow is processed in reverse order.

32. The method of claim 31, wherein a motion vector is used to sample the second image of the pair of images in five or more locations of a Robert'"'"'s Cross configuration.

33. The method of claim 18, further comprising the steps of processing the raw optical flow field for all pyramid scales by a (non-linear) 3×

3 median filter operation to eliminate single-pixel outliers and processing the filtered raw optical flow field with an L2 norm total variation operator for a variable number of iterations.

34. A non-transitory computer-readable storage medium including instructions that, when accessed by a processing system, cause the processing system to perform operations comprising:

(a) decomposing each of a pair of images into image pyramids;

(b) transforming the pair of decomposed images at a first pyramid scale to second derivative representations under an assumption that a brightness gradient of pixels in the pair of decomposed images is constant;

(c) estimating discrete-time derivatives of the second derivative image representations;

(d) applying an optical flow estimation process to the discrete-time derivatives to produce a raw optical flow field;

(e) scaling the raw optical flow field; and

(f) repeating steps (b)-(e) for the pair of images at another pyramid scale until all pyramid scales have been visited to produce a final optical flow field, wherein each of the pair of decomposed images of step (b) is motion compensated by using motion vectors estimated in a previously scaled optical flow estimation.