Methods, Systems and Apparatus for Automatic Video Quality Assessment

• US 20130057703A1
• Filed: 09/02/2011
• Published: 03/07/2013
• Est. Priority Date: 09/02/2011
• Status: Active Grant

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Deshpande, Sachin G., Vu, Cuong
• Granted Patent: US 8,525,883 B2
• Time in Patent Office: Days
• US Class Current: 348/180
• CPC Class Codes: H04N 17/004 for digital television systems

Abstract

Aspects of the present invention are related to systems, methods and apparatus for automatic quality assessment of a video sequence. According to a first aspect of the present invention, a quality index may be generated by combining a spatial quality index and a temporal quality index. According to a second aspect of the present invention, a spatial quality index may be calculated using a modified exponential moving average model to pool multi-scale structural similarity indices computed from test frame—reference frame pairs. According to a third aspect of the present invention, a temporal quality index may be generated by averaging multi-scale structural similarity indices computed from difference image pairs, wherein one difference image is formed between reference frames and another difference image is formed between a reference frame and a test frame.

Claims

1. A method for determining a quality index for a test video sequence, said method comprising:

receiving, in a processor, a test video sequence;

receiving, in said processor, a reference video sequence corresponding to said test video sequence;

in said processor, calculating a spatial quality index using said test video sequence and said reference video sequence;

in said processor, calculating a temporal quality index using said test video sequence and said reference video sequence; and

in said processor, combining said spatial quality index and said temporal quality index to form a final quality index for said test video sequence.

2. A method as described in claim 1, wherein said test video sequence is a degraded version of said reference video sequence, a processed version of said reference video sequence or a previously compressed version of said reference video sequence.

3. A method as described in claim 1, wherein said combining comprises averaging said spatial quality index and said temporal quality index.

4. A method as described in claim 1, wherein:

said test video sequence comprises a first plurality of image frames;

said reference video sequence comprises a second plurality of image frames; and

said calculating a spatial quality index comprises;

calculating a multi-scale structural similarity (MS-SSIM) index for each image frame in said first plurality of image frames and a temporally corresponding image frame in said second plurality of image frames, thereby producing a plurality of MS-SSIM indices;

pooling said plurality of MS-SSIM indices, thereby producing a plurality of pooled MS-SSIM indices;

determining a minimum value from said plurality of MS-SSIM indices; and

setting said spatial quality index to said minimum value.

5. A method as described in claim 4, wherein said pooling said plurality of MS-SSIM indices comprises:

calculating an initial pooled MS-SSIM index by averaging a first plurality of said MS-SSIM indices in said plurality of MS-SSIM indices, wherein said first plurality of said MS-SSIM indices corresponds to a temporally initial portion of said MS-SSIM indices in said plurality of MS-SSIM indices;

calculating a first subsequent pooled MS-SSIM index by forming a linear combination of said initial pooled MS-SSIM index and a first next MS-SSIM index, wherein said first next MS-SSIM index is an immediately temporally subsequent MS-SSIM index to said temporally initial portion of said MS-SSIM indices in said plurality of MS-SSIM indices; and

calculating a second subsequent pooled MS-SSIM index by forming a linear combination of said first subsequent pooled MS-SSIM index and a second next MS-SSIM index, wherein said second next MS-SSIM index is an immediately temporally subsequent MS-SSIM index to said first next MS-SSIM index in said plurality of MS-SSIM indices.

6. A method as described in claim 5, wherein said initial portion of said MS-SSIM indices is associated with a portion of video at least one-half second in length.

7. A method as described in claim 5, wherein:

said calculating a temporal quality index comprises;

forming a first reference difference image between a first image frame in said second plurality of image frames and a second image frame in said second plurality of image frames, wherein said second image frame is an immediately temporally previous image frame to said first image frame in said second plurality of image frames;

forming a first test difference image between a test image frame in said first plurality of image frames, wherein said test image frame corresponds temporally to said first image frame, and said second image frame;

calculating a difference-frames multi-scale structural similarity (MS-SSIM) index using said first reference difference image and said first test difference image; and

averaging said difference-frames MS-SSIM index with a plurality of previously calculated difference-frames MS-SSIM indices.

8. A method as described in claim 7, wherein said combining comprises averaging said spatial quality index and said temporal quality index.

9. A method as described in claim 4, wherein:

said calculating a temporal quality index comprises;

forming a first reference difference image between a first image frame in said second plurality of image frames and a second image frame in said second plurality of image frames, wherein said second image frame is an immediately temporally previous image frame to said first image frame in said second plurality of image frames;

forming a first test difference image between a test image frame in said first plurality of image frames, wherein said test image frame corresponds temporally to said first image frame, and said second image frame;

calculating a difference-frames multi-scale structural similarity (MS-SSIM) index using said first reference difference image and said first test difference image; and

averaging said difference-frames MS-SSIM index with a plurality of previously calculated difference-frames MS-SSIM indices.

10. A method as described in claim 1, wherein:

said test video sequence comprises a first plurality of image frames;

said reference video sequence comprises a second plurality of image frames; and

said calculating a temporal quality index comprises;

forming a first reference difference image between a first image frame in said second plurality of image frames and a second image frame in said second plurality of image frames, wherein said second image frame is an immediately temporally previous image frame to said first image frame in said second plurality of image frames;

forming a first test difference image between a test image frame in said first plurality of image frames, wherein said test image frame corresponds temporally to said first image frame, and said second image frame;

calculating a multi-scale structural similarity (MS-SSIM) index using said first reference difference image and said first test difference image; and

averaging said MS-SSIM index with a plurality of previously calculated MS-SSIM indices.

11. A method for determining a quality index for a test video sequence, said method comprising:

receiving, in a processor, a test video sequence, wherein said test video sequence comprises a first plurality of image frames;

receiving, in said processor, a reference video sequence corresponding to said test video sequence, wherein said reference video sequence comprises a second plurality of image frames; and

in said processor, calculating a spatial quality index using said test video sequence and said reference video sequence, wherein said calculating comprises;

calculating a multi-scale structural similarity (MS-SSIM) index for each image frame in said first plurality of image frames and a temporally corresponding image frame in said second plurality of image frames, thereby producing a plurality of MS-SSIM indices;

pooling said plurality of MS-SSIM indices, thereby producing a plurality of pooled MS-SSIM indices;

determining a minimum value from said plurality of MS-SSIM indices; and

setting said spatial quality index to said minimum value.

12. A method as described in claim 11, wherein said pooling said plurality of MS-SSIM indices comprises:

calculating an initial pooled MS-SSIM index by averaging a first plurality of said MS-SSIM indices in said plurality of MS-SSIM indices, wherein said first plurality of said MS-SSIM indices corresponds to a temporally initial portion of said MS-SSIM indices in said plurality of MS-SSIM indices;

calculating a first subsequent pooled MS-SSIM index by forming a linear combination of said initial pooled MS-SSIM index and a first next MS-SSIM index, wherein said first next MS-SSIM index is an immediately temporally subsequent MS-SSIM index to said temporally initial portion of said MS-SSIM indices in said plurality of MS-SSIM indices; and

calculating a second subsequent pooled MS-SSIM index by forming a linear combination of said first subsequent pooled MS-SSIM index and a second next MS-SSIM index, wherein said second next MS-SSIM index is an immediately temporally subsequent MS-SSIM index to said first next MS-SSIM index in said plurality of MS-SSIM indices.

13. A method as described in claim 12, wherein said initial portion of said MS-SSIM indices is associated with a portion of video at least one-half second in length.

14. A method as described in claim 11 further comprising combining said spatial quality index with a temporal quality index.

15. A method as described in claim 14, wherein said combining comprises averaging said spatial quality index and said temporal quality index.

16. A method as described in claim 11, wherein said test video sequence is a degraded version of said reference video sequence, a processed version of said reference video sequence or a previously compressed version of said reference video sequence.

17. A method for determining a quality index for a test video sequence, said method comprising:

receiving, in a processor, a test video sequence, wherein said test video sequence comprises a first plurality of image frames;

receiving, in said processor, a reference video sequence corresponding to said test video sequence, wherein said reference video sequence comprises a second plurality of image frames; and

in said processor, calculating a temporal quality index using said test video sequence and said reference video sequence, wherein said calculating comprises;

forming a first reference difference image between a first image frame in said second plurality of image frames and a second image frame in said second plurality of image frames, wherein said second image frame is an immediately temporally previous image frame to said first image frame in said second plurality of image frames;

forming a first test difference image between a test image frame in said first plurality of image frames, wherein said test image frame corresponds temporally to said first image frame, and said second image frame;

calculating a multi-scale structural similarity (MS-SSIM) index using said first reference difference image and said first test difference image; and

averaging said MS-SSIM index with a plurality of previously calculated MS-SSIM indices.

18. A method as described in claim 17 further comprising combining said temporal quality index with a spatial quality index.

19. A method as described in claim 18, wherein said combining comprises averaging said spatial quality index and said temporal quality index.

20. A method as described in claim 17, wherein said test video sequence is a degraded version of said reference video sequence, a processed version of said reference video sequence or a previously compressed version of said reference video sequence.