Method of generating motion vectors of images of a video sequence

• US 20100166300A1
• Filed: 12/31/2008
• Published: 07/01/2010
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Castorina, Alfio, Capra, Alessandro, Bruna, Arcangelo Ranieri, Spampinato, Giuseppe
• Granted Patent: US 8,107,750 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A digital image processing system and method for removing motion effects from images of a video sequence, and generating corresponding motion compensated images.

Claims

1. A method of generating signals representing absolute motion vectors of an image of a video sequence, the method comprising:

defining horizontal and vertical weighing functions with values over pixels belonging to a first portion of the image being larger than values over pixels outside the first portion of the image;

computing horizontal and vertical characteristic curves of the image and of a preceding image in the video sequence;

determining a horizontal global displacement between the two images that minimizes a weighted average, calculated using the horizontal weighing function, of an absolute value of a difference between taps of the horizontal characteristic curve of the current image and of a translated replica of the horizontal characteristic curve of the preceding image in the sequence;

determining a vertical global displacement between the two images that minimizes a weighted average, calculated using the vertical weighing function, of an absolute value of a difference between taps of the vertical characteristic curve of the current image and of a translated replica of the vertical characteristic curve of the preceding image in the sequence;

aligning the image with the preceding image in the sequence and calculating relative motion vectors between the aligned images; and

generating the signals representing absolute motion vectors between the images as a vector sum of the relative motion vectors and the horizontal and vertical global displacement.

2. The method of claim 1 wherein the images are YUV or RGB images and the characteristic curves are calculated considering only luminance values of pixels.

3. The method of claim 1 wherein the images are YUV or RGB images and the characteristic curves are calculated considering only green pixels.

4. The method of claim 1 wherein the images are Bayer images and the characteristic curves are calculated considering only green pixels.

5. The method of claim 1 wherein the horizontal and vertical weighing functions are null outside the first portion and, within the first portion, are respectively equal to a horizontal and vertical size of the current image.

6. A method of filtering an image of an input video sequence, the method comprising:

defining horizontal and vertical weighing functions with values over pixels belonging to a first portion of the image being larger than values over pixels outside the first portion of said image;

computing horizontal and vertical characteristic curves of the image and of a preceding image in the sequence;

determining a horizontal global displacement between the two images that minimizes a weighted average, calculated using the horizontal weighing function, of an absolute value of a difference between taps of the horizontal characteristic curve of the current image and of a translated replica of the horizontal characteristic curve of the preceding image in the sequence;

determining a vertical global displacement between the two images that minimizes a weighted average, calculated using the vertical weighing function, of an absolute value of a difference between taps of the vertical characteristic curve of the current image and of a translated replica of the vertical characteristic curve of the preceding imagein the sequence; and

generating a filtered output image by filtering out the horizontal and vertical global displacements from the current image.

7. A method of filtering an image of an input video sequence, the method comprising:

computing horizontal and vertical characteristic curves of the image and of a preceding image in the sequence, then generating corresponding high-pass replicas thereof using a high-pass filter;

calculating a horizontal global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the horizontal high-pass replica characteristic curve of the current image and of a translated replica of the high-pass replica horizontal characteristic curve of the preceding image in the sequence;

calculating a vertical global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the vertical high-pass replica characteristic curve of the current image and of a translated replica of the vertical high-pass replica characteristic curve of the preceding image in the sequence; and

generating a filtered output image by filtering out the horizontal and vertical global displacements from the current image.

8. The method of claim 7, further comprising:

high-pass filtering motion vectors with a second order Butterworth IIR high-pass filter, direct form II, with a cut-off frequency of about 0.40 Hz.

9. The method of claim 7, further comprising:

high-pass filtering motion vectors with a second order Butterworth IIR high-pass filter, direct form II, with a cut-off frequency of about 0.40 Hz divided by half a frame rate of the video sequence.

10. A method of filtering an image of an input video sequence, comprising:

computing horizontal and vertical characteristic curves of the image and of a preceding image in the sequence, then generating corresponding band-pass replicas thereof using a Butterworth IIR band-pass filter;

calculating a horizontal global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the horizontal band-pass replica characteristic curve of the current image and of a translated replica of the horizontal band-pass replica characteristic curve of the proceding image in the sequence;

calculating a vertical global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the vertical band-pass replica characteristic curve of the current image and of a translated replica of the vertical band-pass replica characteristic curve of the preceding image in the sequence; and

generating a filtered output image by filtering out the horizontal and vertical global displacements from the current image.

11. The method of claim 10 wherein the Butterworth IIR band-pass filter has cut-off frequencies of about 0.01 Hz and 0.20 Hz.

12. A method of filtering an image of an input video sequence, comprising the steps of:

computing horizontal and vertical characteristic curves of the image and of a preceding image in the sequence, then generating corresponding high-pass replicas thereof;

calculating a horizontal global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the horizontal high-pass replica characteristic curve of the current image and of a translated replica of the horizontal high-pass replica characteristic curve of the proceding image in the sequence;

calculating a vertical global displacement between the two images that minimizes an average of an absolute value of a difference between taps of the vertical high-pass replica characteristic curve of the current image and of a translated replica of the vertical high-pass replica characteristic curve of the preceding image in the sequence; and

generating a filtered output image by filtering out the horizontal and vertical global displacements from the current image.

13. The method of claim 12 wherein a FIR [−

1 −

8 −

28 −

55 −

62 −

28 28 62 55 28 8 1] is employed.

14. The method of claim 12 wherein the images are YUV or RGB images and the characteristic curves are calculated considering only luminance values of pixels.

15. The method of claim 12 wherein the images are YUV or RGB images and the characteristic curves are calculated considering only green pixels.

16. The method of claim 12 wherein the images are Bayer images and the characteristic curves are calculated considering only green pixels.

17. A system to process digital images, comprising:

a Bayer sensor configured to generate Bayer pattern images of captured scenes;

a curve generator configured to generate characteristic curves of the Bayer pattern images;

a memory storing the Bayer pattern images and the respective characteristic curves; and

a digital signal processor configured to generate absolute global motion vectors based on the Bayer images and the respective characteristic curves.

18. The system of claim 17 wherein the digital signal processor is configured to:

determine a horizontal global displacement between a current image and a preceding image that minimizes a weighted average, calculated using a horizontal weighing function, of an absolute value of a difference between taps of a horizontal characteristic curve of the current image and of a translated replica of a horizontal characteristic curve of the preceding image in the sequence;

determine a vertical global displacement between the two images that minimizes a weighted average, calculated using a vertical weighing function, of an absolute value of a difference between taps of a vertical characteristic curve of the current image and of a translated replica of a vertical characteristic curve of the preceding image in the sequence;

align the current image with the preceding image in the sequence and calculate relative motion vectors between the aligned images; and

generate signals representing absolute motion vectors between the images as a vector sum of the relative motion vectors and the horizontal and vertical global displacement.

19. The system of claim 17 wherein the digital signal processor is configured to:

determine a horizontal global displacement between a current image and a preceding image that minimizes a weighted average, calculated using a horizontal weighing function, of an absolute value of a difference between taps of a horizontal characteristic curve of the current image and of a translated replica of a horizontal characteristic curve of the preceding image in the sequence;

determine a vertical global displacement between the two images that minimizes a weighted average, calculated using a vertical weighing function, of an absolute value of a difference between taps of a horizontal characteristic curve of the current image and of a translated replica of a characteristic curve of the preceding image in the sequence;

align the current image with the preceding image in the sequence and calculate relative motion vectors between the aligned images; and

generate signals representing absolute motion vectors between the images as a vector sum of the relative motion vectors and the horizontal and vertical global displacement.

20. A computer readable memory medium comprising contents that cause a computing device to implement a method of processing a digital image, the method including:

defining horizontal and vertical weighing functions with values over pixels belonging to a first portion of the image being larger than values over pixels outside the first portion of the image;

computing horizontal and vertical characteristic curves of the image and of a preceding image in the video sequence;

determining a horizontal global displacement between the two images that minimizes a weighted average, calculated using the horizontal weighing function, of an absolute value of a difference between taps of the horizontal characteristic curve of the current image and of a translated replica of the horizontal characteristic curve of the preceding image in the sequence;

determining a vertical global displacement between the two images that minimizes a weighted average, calculated using the vertical weighing function, of an absolute value of a difference between taps of the vertical characteristic curve of the current image and of a translated replica of the vertical characteristic curve of the preceding image in the sequence;

aligning the image with the preceding image in the sequence and calculating relative motion vectors between the aligned images; and

generating the signals representing absolute motion vectors between the images as a vector sum of the relative motion vectors and the horizontal and vertical global displacement.

21. A method of generating signals representing absolute motion vectors of an image of a video sequence, the method comprising:

defining horizontal and vertical weighing functions with values over pixels belonging to a first portion of the image being larger than values over pixels outside the first portion of the image;

computing horizontal and vertical characteristic curves of the image and of a preceding image in the video sequence;

determining a horizontal global displacement between the two images that minimizes a weighted average, calculated using the horizontal weighing function, of an absolute value of a difference between taps of the horizontal characteristic curve of the current image and of a translated replica of the characteristic curve of the preceding image in the sequence;

determining a vertical global displacement between the two images that minimizes a weighted average, calculated using the vertical weighing function, of an absolute value of a difference between taps of the horizontal characteristic curve of the current image and of a translated replica of the characteristic curve of the preceding image in the sequence;

aligning the image with the preceding image in the sequence and calculating relative motion vectors between the aligned images; and

generating the signals representing absolute motion vectors between the images as a vector sum of the relative motion vectors and the horizontal and vertical global displacement.

22. The method of claim 21 wherein the images are YUV or RGB images and the characteristic curves are calculated considering only luminance values of pixels.

23. The method of claim 21 wherein the images are Bayer images and the characteristic curves are calculated considering only green pixels.

24. The method of claim 21 wherein the horizontal and vertical weighing functions are null outside the first portion and, within the first portion, are respectively equal to a horizontal and vertical size of the current image.