Apparatus for assessing the visiblity of differences between two image sequences

• US 5,719,966 A
• Filed: 10/15/1996
• Issued: 02/17/1998
• Est. Priority Date: 03/29/1996
• Status: Expired due to Fees

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Lubin, Jeffrey, Brill, Michael Henry, van der Wal, Gooitzen Siemen
• Primary Examiner(s): Rogers, Scott A.
• Application Number:
  US08/730,275
• Time in Patent Office: 490 Days
• Field of Search: 382/260, 382/264, 382/302, 382/263, 382/254, 382/274, 382/275, 382/276, 382/279, 382/303, 382/304, 382/240
• US Class Current: 382/260
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/60 using transform coding
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

An apparatus for assessing the visibility of differences between two input image sequences. The apparatus comprises a pair of pre-filtered processors, a pair of pyramid constructors and a JND processor. Each pre-filter processor receives a separate input image sequence and performs temporal filtering to produce a low-pass temporal response and a bandpass temporal response. In turn, the pyramid constructors receive the temporal responses and generate contrast pyramids for the temporal responses. The JND processor evaluates the contrast pyramids to produce an image metric which is used to assess the visibility of differences between the two input image sequences.

Claims

1. Apparatus for assessing visibility of differences between a first input image sequence and a second input image sequence, where each image sequence having a plurality of images, said apparatus comprising:

a pre-filter processor for filtering the first input image sequence into a first low-pass temporal response and a first high-pass temporal response and filtering the second input image sequence into a second low-pass temporal response and a second high-pass temporal response;

a pyramid constructor, coupled to said pre-filter processor, for generating a first contrast pyramid, having a plurality of levels, for each image of said first and second low-pass temporal responses and generating a second contrast pyramid, having a plurality of levels, for each image of said first and second high-pass temporal responses; and

a contrast pyramid processor, coupled to said pyramid constructor, for generating an image metric from said contrast pyramids.

2. The apparatus of claim 1, wherein said pre-filter processor comprises a first pre-filter processor for filtering the first input image sequence and a second pre-filter processor for filtering the second input image sequence;

and

wherein said pyramid constructor comprises a first pyramid constructor, coupled to both said first and second pre-filter processors, for generating said contrast pyramid for each image of said first and second low-pass temporal responses and a second pyramid constructor, coupled to both said first and second pre-filter processors, for generating said contrast pyramid for each image of said first and second high-pass temporal responses.

3. The apparatus of claim 2, wherein said contrast pyramid processor comprises:

a first maximum section for comparing said contrast pyramids from said pyramid constructors to determine a maximum contrast pyramid.

4. The apparatus of claim 3, wherein said contrast pyramid processor further comprises:

a first buffer, coupled to said first maximum section, for storing said maximum contrast pyramid having a plurality of levels;

a delay, coupled to said first buffer, for applying a delay to each of said level of said maximum contrast pyramid;

a filter, coupled to said delay, for expanding a level of said maximum contrast pyramid to a resolution of a next level;

a second buffer, coupled to said filter, for storing said expanded level of said maximum contrast pyramid; and

a second maximum section, coupled to said first and second buffers, for comparing a next level of said maximum contrast pyramid stored in said first buffer and said expanded level stored in said second buffer to determine a maximum level.

5. The apparatus of claim 4, wherein said contrast pyramid processor further comprises:

a look-up table, coupled between said first buffer and said delay, for applying a weighting function to each level of said maximum contrast pyramid.

6. The apparatus of claim 2, wherein each of said pre-filter processors comprises:

a first buffer for storing a low-pass response of a previous frame with respect to a current frame of the input image sequence;

a second buffer, coupled to said first buffer, for storing a low-pass response of a previous frame with respect to said frame stored in said first buffer;

an adder, coupled to said first buffer, for adding a current frame from the input image sequence with said low-pass response stored in said first buffer to produce a temporal low-pass response; and

a subtractor, coupled to said second buffer, for taking a difference between said temporal low-pass response and said low-pass response stored in said second buffer to produce a temporal high-pass response.

7. The apparatus of claim 6, wherein said temporal low-pass response is expressed as:

space="preserve" listing-type="equation">I.sub.L (t)=A*IL(t)+B*I.sub.L (t-1)

and wherein said temporal high-pass response is expressed as;

space="preserve" listing-type="equation">I.sub.H (t)=C*I.sub.L (t)-D*I.sub.L (t-2).

where A, B, C and D are constants, I_L (t) represents said temporal low-pass response, I_H (t) represents said temporal high-pass response, I(t) represents one of the input image sequence, I_L (t-1) represents said low-pass response stored within said first buffer and I_L (t-2) represents said low-pass response stored within said second buffer.

8. The apparatus of claim 2, wherein each of said pyramid constructors comprises:

two channels, where each of said channel receives a separate temporal response from one of said pre-filter processors.

9. The apparatus of claim 8, wherein each of said channel comprises:

a first delay for applying a delay to said temporal response from said pre-filter processor;

a first filter, coupled to said delay, for producing a first low-pass output;

a second delay, coupled to said first filter, for applying a delay to said first low-pass output;

a second filter, coupled to said second delay, for producing a second low-pass output;

a first subtractor, coupled to said first filter and first delay, for taking a difference between said delayed temporal response of said first delay and said first low-pass output to produce a high-pass output;

a third delay, coupled to said first subtractor, for applying a delay to said high-pass output;

a divider, coupled to said third delay and said second filter, for dividing said delayed high-pass output by said second low-pass output to produce a normalized high-pass output;

a look-up table, coupled to said divider, for applying a weighting function to said normalized high-pass output; and

a buffer, coupled to said first filter, for storing said first low-pass output.

10. The apparatus of claim 9, wherein said first filter is a Guassian filter and said second filter is a spread-tap filter.

11. The apparatus of claim 9, wherein said first delay is a four-line delay, said second delay is a eight-line delay and said third delay is a four-line delay.

12. The apparatus of claim 9, wherein said first low-pass output stored in said buffer is applied to said first delay to generate subsequent levels of a Guassian pyramid.

13. The apparatus of claim 9, wherein each of said pyramid constructor further comprises a second subtractor, coupled to said look-up tables from both channels, for taking a difference between said weighted normalized high-pass outputs to produce said contrast pyramid.

14. Apparatus for assessing visibility of differences between a first input image sequence and a second input image sequence, said apparatus comprising:

a first pre-filter processor for filtering the first input image sequence into a first low-pass temporal response and a first high-pass temporal response;

a second pre-filter processor for filtering the second input image sequence into a second low-pass temporal response and a second high-pass temporal response, wherein each of said pre-filter processors comprises a first buffer for storing a low-pass response of a previous frame with respect to a current frame of the input image sequence, a second buffer, coupled to said first buffer, for storing a low-pass response of a previous frame with respect to said frame stored in said first buffer, an adder, coupled to said first buffer, for adding a current frame from the input image sequence with said low-pass response stored in said first buffer to produce a temporal low-pass response, and a subtractor, coupled to said second buffer, for taking a difference between said temporal low-pass response and said low-pass response stored in said second buffer to produce a temporal high-pass response;

a first pyramid constructor, coupled to both said first and second pre-filter processors, for generating a contrast pyramid, having a plurality of levels, for each frame of said first and second low-pass temporal responses;

a second pyramid constructor, coupled to both said first and second pre-filter processors, for generating a contrast pyramid, having a plurality of levels, for each frame of said first and second high-pass temporal responses, wherein each of said pyramid constructors comprises two channels, where each of said channel receives a separate temporal response from one of said pre-filter processors; and

a processor, coupled to said pyramid constructors, for generating an image metric from said contrast pyramids.