Systems and methods for a motion compensated picture rate converter

• US 20070297513A1
• Filed: 05/14/2007
• Published: 12/27/2007
• Est. Priority Date: 06/27/2006
• Status: Active Grant

• Current Assignee: Synaptics Incorporated, Synaptics, LLC (Synaptics Incorporated)
• Original Assignee: Marvell International Limited (Marvell Technology Group Limited)
• Inventors: Balram, Nikhil, Pathak, Bharat, Biswas, Mainak
• Granted Patent: US 8,340,185 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

Disclosed herein are systems and methods for estimating global and local motions between a pair of temporally adjacent frames of an input signal and for applying these motion vectors to produce at least one interpolated, motion-compensated frame between the adjacent frames. In particular, the systems and methods comprise designs for a motion compensated frame rate converter including a global affine motion estimation engine, a global translation motion estimation engine, a segmentation mask generator, an object edge strength map generator and a local motion estimation engine. Combinations of these features are implemented in a motion compensated picture rate converter to accurately and efficiently provide motion estimation and compensation for a sequence of frames.

Claims

1. A method for applying motion estimation, the method comprising:

receiving an input signal having at least a current frame and a reference frame;

estimating global motion between the reference frame and the current frame to generate at least one affine motion parameter; and

estimating local motion between the reference frame and the current frame to generate at least one motion vector, wherein

the at least one affine motion parameter estimates the global motion of a first region between the reference frame and the current frame, and the at least one motion vector estimates a local motion in a second region between the reference frame and the current frame.

2. The method of claim 1, further comprising:

generating at least one coarse affine parameter; and

refining the at least one coarse affine parameter to generate the at least one affine motion parameter.

3. The method claim 1, wherein the first region is a background region of the current frame, and the at least one affine motion parameter is used to compensate motion of the first region.

4. The method of claim 1, further comprising:

identifying neighboring areas adjacent to the second region of the reference frame; and

generating the at least one motion vector of the second region of the current frame using motion vectors for the neighboring areas computed for the reference frame, wherein

the at least one motion vector is a local motion vector.

5. The method of claim 1, further comprising:

identifying neighboring areas adjacent to the second region on the reference frame;

computing affine motion parameters corresponding to the neighboring areas; and

generating the at least one motion vector of the second region using the computed affine motion parameters, wherein

the at least one motion vector is a modified global motion vector.

6. The method of claim 1, wherein the second region is determined by at least one of a segmentation mask and an object edge strength map applied to the current frame.

7. The method of claim 1, wherein one of a modified global motion vector and a local motion vector is used to compensate motion in the second region based on a smaller one of errors produced from applying the modified global motion vector and the local motion vector.

8. A method for performing global motion estimation, the method comprising:

receiving an input signal having at least a current frame and a reference frame;

phase-correlating the current frame and the reference frame; and

computing at least one affine parameter for estimating a global translation motion between the current frame and the reference frame based on the phase correlation.

9. The method of claim 8, wherein phase-correlating the current frame and the reference frame comprises:

decimating respective ones of the current and reference frames by a decimation factor;

Fourier transforming the decimated current and reference frames;

subtracting phases correlating to the transformed current frame from phases correlating to the transformed reference frame to generate a phase difference array; and

inverse Fourier transforming an exponential of the phase difference array to generate a correlation surface.

10. The method of claim 9, wherein a maximum value of the correlation surface and a location of the maximum value on the correlation surface are used to compute the at least one affine parameter.

11. The method of claim 9, wherein the decimation factor is one of 1, 4, and 9.

12. The method of claim 8, further comprising:

updating the reference frame using the at least one affine parameter;

obtaining a difference between the updated reference frame and the current frame; and

refining the at least one affine parameter to minimize the difference.

13. A method for generating at least one motion vector for an interest area of a current frame in relation to a reference frame, the method comprising:

providing at least one affine parameter for estimating a global motion between the current frame and the reference frame;

generating a segmentation mask based on the at least one affine parameter for identifying foreground and background regions of the current frame;

generating an object edge strength map for identifying regions of significant edge strengths in the current frame; and

computing the at least one motion vector for the interest area based on the foreground, background and significant edge strength regions in relation to the interest area.

14. The method of claim 13, wherein computing the at least one motion vector comprises computing at least one of a global motion vector, a modified global motion vector and a local motion vector.

15. The method of claim 13, wherein generating the segmentation mask comprises:

updating the reference frame using the at least one affine parameter;

obtaining a difference frame between the updated reference frame and the current frame; and

comparing each region of the difference frame to a first threshold for classifying the region into one of the foreground and the background regions.

16. The method of claim 15, further comprising:

determining an object having at least two connected regions in the segmentation mask;

quantifying an area occupied by the object; and

comparing the quantified area to a second threshold to re-classify each of the at least two connected regions into one of the foreground and background regions.

17. The method of claim 13, wherein generating an object edge strength map comprises:

generating a plurality of eigenvalues correlating to vertical and horizontal directions of sub-areas in the current frame; and

determining a maximum of the eigenvalues, wherein each of the sub-areas having eigenvalues approximately within a range defined by the maximum is adapted to be a region of significant edge strength.

18. The method of claim 13, further comprising applying at least one of a median filtering, an edge adaptive filtering and a Gaussian filtering to the motion vector corresponding to the interest area, wherein the interest area is selected from a combination of the segmentation mask and the object edge strength map applied to the current frame.

19. The method of claim 13, further comprising using the at least one motion vector to produce a motion-compensated frame interpolated between the current frame and the reference frame.

20. A method for performing motion compensated frame rate conversion, the method comprising:

receiving an input signal having at least a current frame and a reference frame;

processing the current frame and the reference frame;

providing at least one of an affine global motion parameter and a motion vector that characterize respective global and local motions between the processed current frame and the processed reference frame; and

enabling motion compensated interpolation, wherein the motion compensated interpolation generates at least one interpolated frame between the processed current frame and the processed reference frame using at least one of the affine global motion parameter and the motion vector.

21. The method of claim 20, further comprising post processing an output signal of the motion compensated interpolation, wherein

the output signal has a frame rate that is faster than a native frame rate of the input signal.

22. The method of claim 20, further comprising post processing the processed current frame and the processed reference frame to generate a signal for undergoing the motion compensated interpolation, wherein

the signal is adapted to have a frame rate about equal to a native frame rate of the input signal.

23. The method of claim 20, wherein processing the current frame and the reference frame comprises performing at least one of noise reduction and de-interlacing of the current frame and the reference frame.

24. A motion compensated frame rate converter comprising:

circuitry for receiving an input signal having at least a current frame and a reference frame;

a global affine motion estimation module for estimating global motion between the reference frame and the current frame to generate at least one affine motion parameter; and

a local motion correction module for estimating local motion between the reference frame and the current frame to generate at least one motion vector, wherein

the at least one affine motion parameter estimates the global motion of a first region between the reference frame and the current frame, and the at least one motion vector estimates a local motion in a second region between the reference frame and the current frame.

25. The motion compensated frame rate converter of claim 24, wherein the global affine motion estimation module comprising:

a global translation estimate and affine prediction module for generating at least one coarse affine parameter; and

an affine-parameter refinement module for refining the at least one coarse affine parameter to generate the at least one affine motion parameter.

26. The motion compensated frame rate converter of claim 24, wherein the affine-parameter refinement module uses a RANSAC-based refinement scheme.

27. The motion compensated frame rate converter of claim 24, wherein the first region is a background region of the current frame, and the at least one affine motion parameter is used to compensate motion in the first region.

28. The motion compensated frame rate converter of claim 24, wherein the local motion correction module comprises:

circuitry for identifying neighboring areas adjacent to the second region of the reference frame; and

a local motion estimation module for generating the at least one motion vector of the second region of the current frame using motion vectors for the neighboring areas computed for the reference frame, wherein

the at least one motion vector is a local motion vector.

29. The motion compensated frame rate converter of claim 24, wherein the local motion correction module comprises:

circuitry for identifying neighboring areas adjacent to the second region of the reference frame; and

a modified-global motion estimation module for generating the at least one motion vector of the second region using affine motion parameters computed for the neighboring areas, wherein

the at least one motion vector is a modified global motion vector.

30. The motion compensated frame rate converter of claim 24, wherein the second region is determined based on applying at least one of a segmentation mask and an object edge strength map to the current frame.

31. The motion compensated frame rate converter of claim 24, wherein one of a modified global motion vector and a local motion vector is used to compensate motion in the second region based on a smaller one of errors produced from applying the modified global motion vector and the local motion vector.

32. The motion compensated frame rate converter of claim 24, wherein the motion compensated frame rate converter is provided inside of a motion compensated picture rate converter for processing an input signal having a sequence of frames.

33. A global translation estimation module comprising:

circuitry for receiving an input signal having at least a current frame and a reference frame;

a phase correlation module for phase correlating the current frame and the reference frame; and

circuitry for computing at least one affine parameter that estimates a global translation motion between the current frame and the reference frame based on the phase correlation.

34. The global translation estimation module of claim 33 comprises:

circuitry for decimating respective ones of the current and reference frames by a decimation factor;

circuitry for Fourier transforming the decimated current and reference frames;

circuitry for subtracting phases correlating to the transformed current frame from phases correlating to the transformed reference frame to generate a phase difference array; and

circuitry for inverse Fourier transforming an exponential of the phase difference array to generate a correlation surface.

35. The global translation estimation module of claim 34, further comprising circuitry for determining a maximum value of the correlation surface and a location of the maximum value on the correlation surface, wherein the maximum value and the location of the maximum value are used to compute the at least one affine parameter.

36. The global translation estimation module of claim 34, wherein the decimation factor is one of 1, 4, and 9.

37. The global translation estimation module of claim 33, wherein the at least one affine parameter is provided to an affine-parameter refinement module coupled to the global translation estimation module, and the affine-parameter refinement module comprises:

circuitry for updating the reference frame based on the at least one affine parameter,

circuitry for obtaining a difference between the updated reference frame and the current frame, and

circuitry for refining the at least one affine parameter to minimize the difference.

38. A local motion correction module, comprising:

a segmentation mask for identifying foreground and background regions of a current frame;

an object edge strength map for identifying regions of significant edge strengths in the current frame; and

circuitry for computing at least one motion vector for an interest area based on the foreground, background and significant edge strength regions in relation to the interest area.

39. The local motion correction module of claim 38, wherein circuitry for computing the at least one motion vector comprises circuitry for computing at least one of a modified global motion vector and a local motion vector for the interest area.

40. The local motion correction module of claim 38, wherein the segmentation mask is generated by circuitry comprises:

circuitry for updating the reference frame using at least one affine parameter that characterizes a global motion between the current frame and the reference frame;

circuitry for obtaining a difference frame between the updated reference frame and the current frame; and

circuitry for comparing each region of the difference frame to a first threshold for classifying the region into one of the foreground and the background regions.

41. The local motion correction module of claim 40, wherein the segmentation mask is further generated by circuitry comprises:

circuitry for determining an object having at least two connected regions in the segmentation mask;

circuitry for quantifying an area occupied by the object; and

circuitry for comparing the quantified area to a second threshold to re-classify each of the at least two connected regions into one of the foreground and background regions.

42. The local motion correction module of claim 38, wherein the object edge strength map is generated by circuitry comprises:

circuitry for generating a plurality of eigenvalues correlating to vertical and horizontal directions of sub-areas in the current frame; and

circuitry for determining a maximum of the eigenvalues, wherein each of the sub-areas having eigenvalues approximately within a range defined by the maximum is adapted to be a region of significant edge strength.

43. The local motion correction module of claim 38, wherein at least one of a median filter, an edge adaptive filter and a Gaussian filter is used to filter the motion vector corresponding to the interest area, wherein the interest area is selected from a combination of the segmentation mask and the object edge strength map applied to the current frame.

44. The local motion correction module of claim 38, wherein the at least one motion vector is used to produce a motion-compensated frame interpolated between the current frame and the reference frame.

45. A motion compensated picture rate converter, comprising:

circuitry for receiving an input signal having at least a reference frame and a current frame;

a processing module for processing the reference frame and the current frame; and

a motion compensated frame rate converter including;

circuitry for providing at least one of an affine global motion parameter and a motion vector that characterize respective global and local motions between the processed current frame and the processed reference frame, and

motion compensated interpolation circuitry for generating at least one interpolated frame between the processed current frame and the processed reference frame using at least one of the affine global motion parameter and the motion vector.

46. The motion compensated picture rate converter of claim 45, further comprising a post processing module for processing an output signal of the motion compensated frame rate converter, wherein

the output signal has a frame rate that is higher than a native frame rate of the input signal.

47. The motion compensated picture rate converter of claim 45, further comprising a post processing module that processes the processed current frame and the processed reference frame to generate a signal for supplying to the motion compensated frame rate converter, wherein

the signal is adapted to have a frame rate about equal to a native frame rate of the input signal.

48. The motion compensated picture rate converter of claim 45, wherein the processing module comprises circuitry for performing at least one of noise reduction and de-interlacing of the current frame and the reference frame.

49. A motion compensated frame rate converter comprising:

means for receiving an input signal having at least a reference frame and a current frame;

means for estimating global motion between the reference frame and the current frame to generate at least one affine motion parameter; and

means for estimating local motion between the reference frame and the current frame to generate at least one motion vector, wherein

the at least one affine motion parameter estimates the global motion of a first region between the reference frame and the current frame, and the at least one motion vector estimates a local motion in a second region between the reference frame and the current frame.

50. The motion compensated frame rate converter of claim 49, wherein means for estimating global motion between the reference frame and the current frame further comprising:

means for generating at least one coarse affine parameter; and

means for refining the at least one coarse affine parameter to generate the at least one affine motion parameter.

51. The motion compensated frame rate converter of claim 50, wherein means for refining the at least one coarse affine parameter comprises a RANSAC-based refinement scheme.

52. The motion compensated frame rate converter of claim 49, wherein the first region is a background region of the current frame, and the at least one affine motion parameter is used to compensate motion in the first region.

53. The motion compensated frame rate converter of claim 49, wherein means for estimating local motion between the reference frame and the current frame comprises:

means for identifying neighboring areas adjacent to the second region of the reference frame; and

means for generating the at least one motion vector of the second region of the current frame using motion vectors for the neighboring areas computed for the reference frame, wherein

the at least one motion vector is a local motion vector.

54. The motion compensated frame rate converter of claim 49, wherein means for estimating local motion between the reference frame and the current frame comprises:

means for identifying neighboring areas adjacent to the second region of the reference frame; and

means for generating the at least one motion vector of the second region using affine motion parameters of the neighboring areas, wherein

the at least one motion vector is a modified global motion vector.

55. The motion compensated frame rate converter of claim 49, further comprising means for determining the second region based on means for applying at least one of a segmentation mask and an object edge strength map to the current frame.

56. The motion compensated frame rate converter of claim 49, further comprising means for compensating the motion in the second region using one of a modified global motion vector and a local motion vector based on a smaller one of errors produced from applying the modified global motion vector and the local motion vector.

57. A global translation estimation module comprising:

means for receiving an input signal having at least a current frame and a reference frame;

means for phase correlating the current frame and the reference frame; and

means for computing at least one affine parameter for estimating a global translation motion between the current frame and the reference frame based on the phase correlation.

58. The global translation estimation module of claim 57, wherein means for phase correlating the current frame and the reference frame further comprising:

means for decimating respective ones of the current and reference frames by a decimation factor;

means for Fourier transforming the decimated current and reference frames;

means for subtracting phases correlating to the transformed current frame from phases correlating to the transformed reference frame to generate a phase difference array; and

means for inverse Fourier transforming an exponential of the phase difference array to generate a correlation surface.

59. The global translation estimation module of claim 58, further comprising means for determining a maximum value of the correlation surface and a location of the maximum value on the correlation surface, wherein the maximum vale and the location of the maximum value are used to compute the at least one affine parameter.

60. The global translation estimation module of claim 59, wherein the decimation factor is one of 1, 4, and 9.

61. The global translation estimation module of claim 59, wherein the at least one affine parameter is provided to an affine-parameter refinement module coupled to the global translation estimation module, wherein the affine-parameter refinement module comprises:

means for updating the reference frame based on the at least one affine parameter,

means for obtaining a difference between the updated reference frame and the current frame, and

means for refining the at least one affine parameter to minimize the difference.

62. A local motion correction module, comprising:

means for producing a segmentation mask for identifying foreground and the background regions of a current frame;

means for producing an object edge strength map for identifying regions of significant edge strengths in the current frame; and

means for computing at least one motion vector for an interest area based on the foreground, background and significant edge strength regions in relation to the interest area.

63. The local motion correction module of claim 62, wherein means for computing the at least one motion vector comprises means for computing at least one of a modified global motion vector and a local motion vector for the interest area.

64. The local motion correction module of claim 62, wherein means for generating the segmentation mask includes:

means for updating the reference frame using at least one affine parameter that characterizes a global motion between the current frame and the reference frame;

means for obtaining a difference frame between the updated reference frame and the current frame; and

means for comparing each region of the difference frame to a first threshold for classifying the region into one of the foreground and background regions.

65. The local motion correction module of claim 64, wherein means for producing the segmentation mask further includes:

means for determining an object having at least two connected regions in the segmentation mask;

means for quantifying an area occupied by the object; and

means for comparing the quantified area to a second threshold to re-classify each of the at least two connected regions into one of the foreground and background regions.

66. The local motion correction module of claim 62, wherein means for producing the object edge strength includes:

means for generating a plurality of eigenvalues correlating to vertical and horizontal directions of sub-areas in the current frame; and

means for determining a maximum of the eigenvalues, wherein each of the sub-areas having eigenvalues approximately within a range defined by the maximum is a region of significant edge strength.

67. The local motion correction module of claim 62, wherein at least one of a median filter meanings, an edge adaptive filter meanings and a Gaussian filter meanings is used to filter the motion vector corresponding to the interest area, wherein the interest area is selected from a combination of the segmentation mask and the object edge strength map applied to the current frame.

68. The local motion correction module of claim 62, wherein the at least one motion vector is used to produce a motion-compensated frame interpolated between the current frame and the reference frame.

69. A motion compensated picture rate converter, comprising:

means for receiving an input signal having at least a current frame and a reference frame;

meanings for processing the current frame and the reference frame;

meanings for providing at least one of an affine global motion parameter and a motion vector characterizing respective global and local motions between the processed current frame and the processed reference frame; and

meanings for enabling motion compensated interpolation, wherein the motion compensated interpolation generates at least one interpolated frame between the processed current frame and the processed reference frame using at least one of the affine global motion parameter and the motion vector.

70. The motion compensated picture rate converter of claim 69, further comprising means for post-processing an output signal from the means for enabling motion compensated interpolation, wherein

the output signal has a frame rate that is faster than a native frame rate of the input signal.

71. The motion compensated picture rate converter of claim 69, further comprising means for post-processing the processed current frame and the processed reference frame to generate a signal for supplying to the means for enabling motion compensated interpolation, wherein

the signal is adapted to have a frame rate about equal to a native frame rate of the input signal.

72. The motion compensated picture rate converter of claim 69, wherein the means for processing the current frame and the reference frame comprises means for performing at least one of noise reduction and de-interlacing of the current frame and the reference frame.