System and method for motion compensation and frame rate conversion

• US 6,442,203 B1
• Filed: 11/05/1999
• Issued: 08/27/2002
• Est. Priority Date: 11/05/1999
• Status: Expired due to Fees

• Current Assignee: Dolby Laboratories Incorporated, Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Demografx Inc. (Dolby Laboratories Incorporated)
• Inventors: Demos, Gary E.
• Primary Examiner(s): Le, Vu
• Application Number:
  US09/435,277
• Time in Patent Office: 1,026 Days
• Field of Search: 375/240.16, 375/240.17, 375/240.19, 375/240.29, 375/240.27, 348/406.1, 348/407.1, 348/408.1, 348/416.1, 348/421.1, 348/422.1, 348/425.3, 348/431.1, 348/441, 348/451, 348/452, 348/699, 348/700, 382/236, 382/238, 386/109, 386/111
• US Class Current: 375/240.16
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 7/012 Conversion between an interlaced and a progressive signal for graphics images G09G2310/0229
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A system and method for de-interlacing, motion compensation and frame rate conversion of digitally encoded video signals. The invention utilizes multiple motion vectors per pixel of the final image. In a preferred embodiment, this is accomplished by increasing the size of the image, with the amount of size increase depending upon the degree of sub-pixel accuracy desired. In a preferred embodiment, image size is doubled. Thus, four motion vectors are generated for each pixel. Each motion vector is found by searching independently for the best match with previous and subsequent frames. In one aspect, the invention includes a method of temporal processing of motion picture image frames each comprising a plurality of pixels, including the steps of comparing each pixel of a current frame to at least one previous or subsequent image frame; determining at least one motion vector corresponding to each such pixel relative to such at least one previous or subsequent image frame; and saving the determined motion vectors. The invention also includes the steps of applying motion vectors corresponding to multiple image frames to define a new pixel for each pixel of the current frame, and outputting all of such new pixels as a constructed image frame. The invention includes system and computer program implementations.

Claims

1. A method of temporal processing of motion picture image frames each comprising a plurality of pixels, including the steps of:

(a) determining at least one motion vector corresponding to each pixel of a current frame relative to at least one previous or subsequent image frame by performing a hierarchical motion vector search including at least;

(1) a first search between a matching region associated with each pixel of the current frame and a sequence of search regions at a first resolution within such at least one previous or subsequent image frame; and

(2) a second search between the matching region and a sequence of search regions at a second resolution within such at least one previous or subsequent image frame;

(b) saving the determined motion vectors.

2. The method of claim 1, further including the steps of:

(a) for each pixel of the current frame, applying at least one such corresponding motion vector to define a new pixel;

(b) outputting all of such new pixels as a constructed image frame.

3. The method of claim 2, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

4. The method of claim 2, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

5. The method of claim 1, wherein the step of determining at least one motion vector includes determining a plurality of motion vectors corresponding to each pixel of the current frame.

6. The method of claim 1, wherein the step of determining at least one motion vector includes applying at least one match criteria for each such matching region and sequence of search regions.

7. The method of claim 6, wherein the step of applying at least one match criteria includes applying a DC minimum area difference match criteria.

8. The method of claim 6, wherein the step of applying at least one match criteria includes applying an AC maximum area correlation match criteria.

9. The method of claim 1, wherein the step of determining at least one motion vector includes applying a weighting function such that motion vectors are biased towards shorter lengths.

10. The method of claim 1, further including the step of computing a confidence factor for each motion vector.

11. The method of claim 10, wherein the confidence factor is based upon a DC minimum area difference function.

12. The method of claim 10, wherein the confidence factor is based upon an AC maximum area correlation function.

13. The method of claim 10, further including the steps of:

(a) defining at least one threshold value; and

(b) screening out motion vectors having a confidence factor below such threshold value.

14. The method of claim 10, wherein the confidence factor is based upon a local gradient among motion vectors.

15. The method of claim 1, wherein motion vectors are stored as picture image values.

16. The method of claim 1, further including the steps of:

(a) for each pixel of the current frame, applying motion vectors corresponding to multiple image frames to define a new pixel;

(b) outputting all of such new pixels as a constructed image frame.

17. The method of claim 16, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

18. The method of claim 16, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

19. The method of claim 16, wherein the temporal processing results in a set of de-interlaced constructed image frames.

20. The method of claim 16, further including the step of computing a confidence factor for each motion vector.

21. The method of claim 20, further including the step of weighting use of each motion vector by a corresponding confidence factor.

22. The method of claim 20, further including the step of weighting each motion vector by an overall confidence factor and a regional confidence factor.

23. The method of claim 20, further including the step of determining a statistical variance for a set of confidence factors as a further measure of confidence.

24. The method of claim 16, further including the step of applying the motion vectors to add motion blur to the constructed image.

25. The method of claim 16, further including the step of applying a filter function to each new pixel touched by the applied motion vectors.

26. The method of claim 25, wherein the filter function is a selected from a set consisting of a cone filter, Gaussian filter, or truncated sinc filter.

27. The method of claim 25, further including the step of normalizing each pixel that has been filtered by a total combined weight function.

28. The method of claim 1, wherein the first resolution is a reduced resolution relative to the second resolution.

29. The method of claim 1, wherein the second resolution is a higher resolution relative to the first resolution.

30. The method of claim 1, further including a third search between the matching region and a sequence of search regions at a third resolution within such at least one previous or subsequent image frame.

31. The method of claim 30, wherein the first resolution is a reduced resolution relative to the second resolution, and the third resolution is a higher resolution relative to the second resolution.

32. The method of claim 1, further including the step of using motion vectors determined during the first search as guide vectors in determining corresponding motion vectors during the second search.

33. A system of temporal processing of motion picture image frames each comprising a plurality of pixels, including:

(a) means for determining at least one motion vector corresponding to each pixel of a current frame relative to at least one previous or subsequent image frame by performing a hierarchical motion vector search including at least;

(1) a first search between a matching region associated with each pixel of the current frame and a sequence of search regions at a first resolution within such at least one previous or subsequent image frame; and

(2) a second search between a matching region and a sequence of search regions at a second resolution within such at least one previous or subsequent image frame;

(b) means for saving the determined motion vectors.

34. The system of claim 33, further including:

(a) means for applying at least one such corresponding motion vector to define a new pixel for each pixel of the current frame;

(b) means for outputting all of such new pixels as a constructed image frame.

35. The system of claim 34, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

36. The system of claim 34, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

37. The system of claim 33, wherein the means for determining at least one motion vector includes means for determining a plurality of motion vectors corresponding to each pixel of the current frame.

38. The system of claim 33, wherein the means for determining at least one motion vector includes means for applying at least one match criteria for each such matching region and sequence of search regions.

39. The system of claim 38, wherein the means for applying at least one match criteria includes means for applying a DC minimum area difference match criteria.

40. The system of claim 38, wherein the means for applying at least one match criteria includes means for applying an AC maximum area correlation match criteria.

41. The system of claim 33, wherein the means for determining at least one motion vector includes means for applying a weighting function such that motion vectors are biased towards shorter lengths.

42. The system of claim 33, further including means for computing a confidence factor for each motion vector.

43. The system of claim 42, wherein the confidence factor is based upon a DC minimum area difference function.

44. The system of claim 42, wherein the confidence factor is based upon an AC maximum area correlation function.

45. The system of claim 42, wherein the confidence factor is based upon a local gradient among motion vectors.

46. The system of claim 42, further including:

(a) means for defining at least one threshold value; and

(b) means for screening out motion vectors having a confidence factor below such threshold value.

47. The computer program of claim 42, wherein the confidence factor is based upon a DC minimum area difference function.

48. The computer program of claim 42, wherein the confidence factor is based upon an AC maximum area correlation function.

49. The computer program of claim 42, wherein the confidence factor is based upon a local gradient among motion vectors.

50. The system of claim 33, wherein motion vectors are stored as picture image values.

51. The system of claim 33, further including:

(a) means for applying motion vectors corresponding to multiple image frames to define a new pixel for each pixel of the current frame;

(b) means for outputting all of such new pixels as a constructed image frame.

52. The system of claim 51, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

53. The system of claim 51, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

54. The system of claim 51, wherein the temporal processing results in a set of de-interlaced constructed image frames.

55. The system of claim 51, further including means for computing a confidence factor for each motion vector.

56. The system of claim 55, further including means for weighting use of each motion vector by a corresponding confidence factor.

57. The system of claim 55, wherein the filter function is a selected from a set consisting of a cone filter, Gaussian filter, or truncated sinc filter.

58. The system of claim 55, further including means for normalizing each pixel that has been filtered by a total combined weight function.

59. The system of claim 55, further including means for weighting each motion vector by an overall confidence factor and a regional confidence factor.

60. The system of claim 55, further including means for determining a statistical variance for a set of confidence factors as a further measure of confidence.

61. The system of claim 51, further including means for applying the motion vectors to add motion blur to the constructed image.

62. The system of claim 51, further including means for applying a filter function to each new pixel touched by the applied motion vectors.

63. The system of claim 33, wherein the first resolution is a reduced resolution relative to the second resolution.

64. The system of claim 33, wherein the second resolution is a higher resolution relative to the first resolution.

65. The system of claim 33, further including means for performing a third search between the matching region and a sequence of search regions at a third resolution within such at least one previous or subsequent image frame.

66. The system of claim 65, wherein the first resolution is a reduced resolution relative to the second resolution, and the third resolution is a higher resolution relative to the second resolution.

67. The system of claim 33, further including means for using motion vectors determined during the first search as guide vectors in determining corresponding motion vectors during the second search.

68. A computer program, stored on a computer-readable medium, for temporal processing of motion picture image frames each comprising a plurality of pixels, ) the computer program including instructions for causing a computer to:

(a) determine at least one motion vector corresponding to each pixel of a current frame relative to at least one previous or subsequent image frame by performing a hierarchical motion vector search including at least;

(1) a first search between a matching region associated with such pixel of the current frame and a sequence of search regions at a first resolution within such at least one previous or subsequent image frame; and

(2) a second search between a matching region and a sequence of search regions at a second resolution within such at least one previous or subsequent image frame;

(b) save the determined motion vectors.

69. The computer program of claim 68, further including instructions for causing the computer to:

(a) for each pixel of the current frame, apply at least one such corresponding motion vector to define a new pixel;

(b) output all of such new pixels as a constructed image frame.

70. The computer program of claim 69, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

71. The computer program of claim 69, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

72. The computer program of claim 68, wherein the instructions causing the computer to determine at least one motion vector include instructions for causing the computer to determine a plurality of motion vectors corresponding to each pixel of the current frame.

73. The computer program of claim 68, wherein the instructions causing the computer to determine at least one motion vector include instructions for causing the computer to apply at least one match criteria for each such matching region and sequence of search regions.

74. The computer program of claim 73, wherein the instructions causing the computer to apply at least one match criteria include instructions for causing the computer to apply a DC minimum area difference match criteria.

75. The computer program of claim 73, wherein the instructions causing the computer to apply at least one match criteria include instructions for causing the computer to apply an AC maximum area correlation match criteria.

76. The computer program of claim 68, wherein the instructions causing the computer to determine at least one motion vector include instructions for causing the computer to apply a weighting function such that motion vectors are biased towards shorter lengths.

77. The computer program of claim 68, further including instructions for causing the computer to compute a confidence factor for each motion vector.

78. The computer program of claim 77, further including instructions for causing the computer to:

(a) define at least one threshold value; and

(b) screen out motion vectors having a confidence factor below such threshold value.

79. The computer program of claim 68, wherein motion vectors are stored as picture image values.

80. The computer program of claim 68, further including instructions for causing the computer to:

(a) apply motion vectors corresponding to multiple image frames to define a new pixel for each pixel of the current frame;

(b) output all of such new pixels as a constructed image frame.

81. The computer program of claim 80, wherein the temporal processing results in a set of such constructed image frames having a frame rate different from the frame rate of the motion picture image frames.

82. The computer program of claim 80, wherein the temporal processing results in a set of such constructed image frames having motion compensated reduced noise compared to the motion picture image frames.

83. The computer program of claim 80, wherein the temporal processing results in a set of de-interlaced constructed image frames.

84. The computer program of claim 80, further including instructions for causing the computer to compute a confidence factor for each motion vector.

85. The computer program of claim 84, further including instructions for causing the computer to weight use of each motion vector by a corresponding confidence factor.

86. The computer program of claim 84, wherein the filter function is a selected from a set consisting of a cone filter, Gaussian filter, or truncated sinc filter.

87. The computer program of claim 84, further including instructions for causing the computer to normalize each pixel that has been filtered by a total combined weight function.

88. The computer program of claim 84, further including instructions for causing the computer to weight each motion vector by an overall confidence factor and a regional confidence factor.

89. The computer program of claim 84, further including instructions for causing the computer to determine a statistical variance for a set of confidence factors as a further measure of confidence.

90. The computer program of claim 80, further including instructions for causing the computer to apply the motion vectors to add motion blur to the constructed image.

91. The computer program of claim 80, further including instructions for causing the computer to apply a filter function to each new pixel touched by the applied motion vectors.

92. The computer program of claim 68, wherein the first resolution is a reduced resolution relative to the second resolution.

93. The computer program of claim 68, wherein the second resolution is a higher resolution relative to the first resolution.

94. The computer program of claim 68, further including instructions for causing the computer to perform a third search between the matching region and a sequence of search regions at a third resolution within such at least one previous or subsequent image frame.

95. The computer program of claim 94, wherein the first resolution is a reduced resolution relative to the second resolution, and the third resolution is a higher resolution relative to the second resolution.

96. The computer program of claim 68, further including instructions for causing the computer to use motion vectors determined during the first search as guide vectors in determining corresponding motion vectors during the second search.