Method and apparatus for estimating and compensating for jitter in digital video

• US 8,130,845 B2
• Filed: 11/02/2006
• Issued: 03/06/2012
• Est. Priority Date: 11/02/2006
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Brusnitsyn, Mikhail, Clarke, Ian, Zhou, Hui
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): VAZQUEZ COLON, MARIA E
• Application Number:
  US11/556,092
• Publication Number:
  US 20080107186A1
• Time in Patent Office: 1,951 Days
• Field of Search: 375/240.28
• US Class Current: 375/240.28
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

A method of estimating and compensating for jitter between two video frames in a digital video sequence comprises comparing a pixel subset in one video frame to candidate matching pixel subsets within a search area in the other video frame to obtain a preliminary matching pixel subset. Based on the location of the preliminary matching pixel subset, an adapted search area is formed and the comparing is continued within the adapted search area to obtain a final matching pixel subset. Motion is calculated between the video frames based on the final match thereby to estimate the jitter. The video frames are then transformed based on the estimated jitter.

Claims

1. A method of estimating jitter between two video frames in a digital video sequence, comprising:

comparing a pixel subset of one video frame to candidate pixel subsets within a search area in the other video frame to determine a preliminary matching pixel subset from the candidate pixel subsets;

determining an actual matching pixel subset from the other video frame based on the location of the preliminary matching pixel subset relative to said search area; and

calculating a global jitter vector based on the pixel subset and the actual matching pixel subset thereby to estimate the jitter;

checking the reliability of the actual matching pixel subset; and

only calculating the global jitter vector if the actual matching pixel subset is deemed to be reliable;

wherein reliability of the actual matching pixel subset is based on at least one of frame contrast measure and steepness measure; and

wherein said determining comprises;

determining if the preliminary matching pixel subset is adjacent a border of the search area;

if not, designating the preliminary matching pixel subset as the actual matching pixel subset;

otherwise, altering the search area; and

comparing the pixel subset to candidate pixel subsets within the altered search area to determine the actual matching pixel subset;

wherein said altering comprises;

shifting the search area in the direction of the preliminary matching pixel subset.

2. The method of claim 1, wherein said determining is based on the location of the preliminary matching pixel subset relative to the boundary of said search area.

3. The method of claim 1, wherein the preliminary matching pixel subset is designated as the actual matching pixel subset if the preliminary matching pixel subset does not have a border coincident with a border of the search area.

4. The method of claim 3, wherein said search area comprises pixels corresponding to the pixels of the pixel subset in the one video frame together with a band of surrounding pixels.

5. The method of claim 4, wherein the width of the band is equal to the distance the search area is shifted.

6. The method of claim 4, wherein said pixel subset comprises a set of pixel blocks, said pixel blocks being distributed across said one video frame.

7. The method of claim 6, wherein said comparing comprises comparing pixel blocks of the one video frame with corresponding pixel bocks within search areas of the other video frame to determine a preliminary matching set of pixel blocks.

8. The method of claim 7, wherein during the pixel block comparing, absolute differences between pixels of the pixel blocks and pixels of the search areas are calculated and accumulated thereby to determine the preliminary matching set of pixel blocks.

9. The method of claim 6, wherein said pixel blocks are evenly distributed across said one video frame.

10. The method of claim 1, further comprising:

prior to the comparing, converting the video frames to monochrome.

11. The method of claim 10, wherein the converting comprises:

converting each pixel in the video frames to black or white based on a luminance value of each pixel relative to a luminance threshold.

12. The method of claim 11, wherein the luminance threshold is adjustable.

13. The method of claim 12, wherein the luminance threshold is adjusted when an average luminance of pixels in the one video frame differs from the luminance threshold by greater than a difference threshold amount.

14. The method of claim 11, wherein the converting comprises:

calculating overall luminance measures for each of the video frames;

selecting a bit plane based on an average of the overall luminance measures; and

setting each pixel in the video frames to one of black and white based on the selected bit plane.

15. The method of claim 14, wherein the overall luminance measures are video frame luminance averages.

16. The method of claim 1, further comprising:

calculating rotational jitter between the video frames.

17. The method of claim 16, further comprising:

multiplying the rotational jitter by a correction factor to correct for accumulated error.

18. The method of claim 1, further comprising:

subtracting a weighted global jitter average of a previous M video frames from the global jitter vector to account for intentional panning during capture of the digital video sequence.

19. The method of claim 1, further comprising:

transforming the video frames based on the estimated jitter.

20. The method of claim 19, wherein the transforming comprises:

aligning a viewport with the video frames in accordance with the estimated jitter.

21. The method of claim 20, further comprising:

displaying the video frames, during display non-corresponding areas of the video frames being cropped.

22. An apparatus for estimating jitter between two video frames in a digital video sequence, comprising:

a frame comparator comparing a pixel subset of one video frame to candidate pixel subsets within a search area in the other video frame to determine a preliminary matching pixel subset from the candidate pixel subsets;

a match position estimator determining the location of the preliminary matching pixel subset relative to said search area;

the frame comparator determining an actual matching pixel subset from the other video frame based on the location determined by the match position estimator; and

a jitter estimator calculating a global jitter vector based on the pixel subset and the actual matching pixel subset thereby to estimate the jitter;

wherein said jitter estimator;

checks the reliability of the actual matching pixel subset; and

only calculates the global jitter vector if the actual matching pixel subset is deemed to be reliable;

wherein reliability of the actual matching pixel subset is based on at least one of frame contrast measure and steepness measure; and

wherein said frame comparator;

determines if the preliminary matching pixel subset is adjacent a border of the search area;

if not, designates the preliminary matching pixel subset as the actual matching pixel subset;

otherwise, alters the search area; and

compares the pixel subset to candidate pixel subsets within the altered search area to determine the actual matching pixel subset;

wherein said altering comprises;

shifting the search area in the direction of the preliminary matching pixel subset.

23. The apparatus of claim 22, further comprising:

a jitter compensator transforming the video frames based on the estimated jitter.

24. The apparatus of claim 23, wherein said jitter estimator further calculates a rotational jitter vector.

25. A non-transitory computer readable medium embodying a computer program for estimating jitter between two video frames in a digital video sequence, the computer program, when executed, performs a method comprising:

comparing a pixel subset of one video frame to candidate pixel subsets within a search area in the other video frame to determine a preliminary matching pixel subset from the candidate pixel subsets;

determining an actual matching pixel subset from the other video frame based on the location of the preliminary matching pixel subset relative to said search area; and

calculating a global jitter vector based on the pixel subset and the actual matching pixel subset thereby to estimate the jitter;

checking the reliability of the actual matching pixel subset; and

only calculating the global jitter vector if the actual matching pixel subset is deemed to be reliable;

wherein reliability of the actual matching pixel subset is based on at least one of frame contrast measure and steepness measure; and

wherein said determining comprises;

determining if the preliminary matching pixel subset is adjacent a border of the search area;

if not, designating the preliminary matching pixel subset as the actual matching pixel subset;

otherwise, altering the search area; and

comparing the pixel subset to candidate pixel subsets within the altered search area to determine the actual matching pixel subset;

wherein said altering comprises;

shifting the search area in the direction of the preliminary matching pixel subset.

26. The non-transitory computer readable medium according to claim 25, further comprising:

transforming the video frames based on the estimated jitter.