Method for temporally adaptive filtering of frames of a noisy image sequence using motion estimation

• US 5,600,731 A
• Filed: 05/09/1991
• Issued: 02/04/1997
• Est. Priority Date: 05/09/1991
• Status: Expired due to Term

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Sezan, Muhammed I., Ozkan, Mehmet K., Fogel, Sergei V.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Johns, Andrew W.
• Application Number:
  US07/697,868
• Time in Patent Office: 2,098 Days
• Field of Search: 358/105, 382/54, 382/107, 382/260, 382/261, 382/275, 382/254, 348/155, 348/618, 348/620
• US Class Current: 382/107
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

A method of temporally adaptive filtering of the frames of an image sequence is disclosed. To filter the kth frame, first, the motion trajectories that transverse the pixel locations at the kth frame are determined using a motion estimation algorithm. Motion trajectories pass through a predetermined number of frames neighboring the kth frame. Image values at the neighboring frames, along motion trajectories are then evaluated. Temporally adaptive linear minimum mean square error (LMMSE) estimates of the image values at the pixel locations of the kth frame are determined using the image values along the motion trajectories. These steps are repeated for each of the other frames of the image sequence.

Claims

1. A method of temporally adaptive filtering frames of an image sequence in which each image of the sequence has pixel locations, comprising:

(a) subdividing an image sequence into a plurality of frames;

(b) determining motion trajectories of pixels that pass through the pixel locations at a kth frame using a motion estimation algorithm;

(c) determining image values for a pixel originally located at a specific pixel location at the kth frame along the motion trajector of that pixel to the same or other pixel locations in other frames of the image sequence, and repeating for all pixel locations at the kth frame;

(d) determining temporally adaptive linear minimum mean square error (LMMSE) estimates of the image value at each of the pixel locations at the kth frame based upon the determined image values;

(e) repeating steps b, c and d for each of the other frames of the image sequence;

(f) modifying the image values of the pixels at the pixel locations at each frame of the image sequence based upon the determined temporally adaptive LMMSE estimates; and

(g) generating a temporally adaptive filtered image sequence.

2. The method of claim 1, wherein, for a pixel located at (m,n) in the kth frame, the motion trajectory τ

_m,n,k is defined as a set of locations containing (m,n) as well as displaced locations corresponding to (m,n) in N-1 neighboring frames at times k-M, . . . k-1, k+1, . . . , k+M, where N=2M+1.

3. The method of claim 1, wherein step (b) further comprises estimating displacement vectors with the motion estimation algorithm.

4. The method of claim 3, wherein components of the displacement vector estimates are real-valued, such that the motion trajectories include subpixel locations, and wherein step (c) further comprises (f) determining image values at the subpixel locations by interpolation.

5. The method of claim 1, wherein the motion estimation algorithm is a spatiotemporal-gradient method.

6. The method of claim 5, wherein the spatiotemporal-gradient method is the Fogel algorithm.

7. The method of claim 1, wherein step (b) includes the steps of:

(b) (1) forming a multiresolution representation of the image sequence;

(b) (2) constraining a displacement vector field estimate using optical flow and directional smoothness constraints;

(b) (3) obtaining a displacement vector field estimate using a variational principle; and

(b) (4) solving the variational principle.

8. The method of claim 1, wherein the temporally local LMMSE estimate of the image value at pixel location (m,n) of the kth frame is determined according to the equation:

##EQU15## where σ

_f² (m,n;

k)=max[0,σ

_g² (m,n;

k)-σ

_v² (m,n;

k)], and σ

_g² (m,n,;

k) is the sample variance, and μ

_f (m,n;

k)=μ

_g (m,n;

k) is the sample mean of a degraded image sequence computed over the motion trajectory τ

_m,n,k, and σ

_v² (m,n) denotes the sample variance of a noise process.

9. The method of claim 1, wherein step (b) includes estimating the motion trajectories from the kth frame to a predetermined number of contiguous frames in the image sequence.

10. The method of claim 9, wherein the predetermined number of contiguous frames are used to filter the kth frame.

11. The method of claim 1, wherein step (b) includes estimating the motion trajectories from the kth frame to a k-1 frame and a k+1 frame.

12. The method of claim 11, wherein the k-1, k and k+1 frames are filtered to produce three intermediate frames, these intermediate frames then being filtered to determine an estimate f(m,n,k) of the image value in step (d).

13. The method of claim 1, wherein the kth frame is filtered using N=2M+1 frames, the filtering being performed in M steps, and the LMMSE estimation is applied to groups of three frames where N is a predetermined number of frames.

14. The method of claim 1, wherein the temporally local LMMSE estimate of the image value at pixel location (m,n) of the kth frame is determined according to the equation:

##EQU16## where σ

_f² (m,n;

k)=max[0,σ

_g² (m,n;

k)-σ

_v² (m,n;

k)], and σ

_g² (m,n;

k) is the sample variance, and μ

^a_f (m,n;

k)= ##EQU17## is a motion-adaptive sample mean where the weight function is given by

space="preserve" listing-type="equation">w(i,j,l)≐

K(1+a.sup.2 max[ε

.sup.2, (g(m,n,k)-g(i,j,l)).sup.2 ]).sup.-1,

and the constant K is defined as

space="preserve" listing-type="equation">K≐

β

(i,j,k-1)β

(i,j,k)β

(i,j,k+1)/[β

(i,j,k-1)β

(i,j,k)+β

(i,j,k-1)β

(i,j,k+1)+β

(i,j,k)β

(i,j,k+1)]

the quantity β

being given by,

space="preserve" listing-type="equation">β

(i,j,l)≐

1+a.sup.2 (max[ε

.sup.2, (g(m,n,k)-g(i,j,l)).sup.2 ]).

15. The method of claim 1, wherein the temporally local LMMSE estimate of the image value at pixel location (m,n) of the kth frame is determined according to the equation f(m,n,k)=μ

^a_f (m,n,k) where μ

^a_f (m,n,k) is a motion-adaptive sample mean value and f(m,n,k) is a filtered pixel value.

16. A method of temporally adaptively filtering frames of an image sequence in which each image of the sequence has pixel locations, comprising:

(a) subdividing an image sequence into a plurality of frames;

(b) determining motion trajectories of pixels that pass through the pixel locations at a kth frame using a motion estimation algorithm;

(c) determining image values for a pixel originally located at a specific pixel location at the kth frame along the motion trajectory of that pixel to the same or other pixel locations in other frames of the image sequence, and repeating for all pixel locations at the kth frame;

(d) determining temporally adaptive estimates of image value at each of the pixel locations at the kth frame based upon the determined image values;

(e) repeating steps b, c and d for each of the other frames of the image sequence;

(f) modifying the image values of the pixels at the pixel locations at each frame of the image sequence based upon the determined temporally adaptive LMMSE estimates; and

(g) generating a temporally adaptive filtered image sequence.

17. The method of claim 16, wherein step (d) includes median filtering along the motion trajectories to perform temporal noise filtering.

18. The method of claim 16, wherein step (d) includes edge-preserving spatial filtering.