Apparatus, and associated method, for forming a compressed motion vector field utilizing predictive motion coding

• US 20040165664A1
• Filed: 02/13/2004
• Published: 08/26/2004
• Est. Priority Date: 08/11/1999
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Karczewicz, Marta, Lainema, Jani, Dobrin, Bogdan-Paul
• Granted Patent: US 7,149,251 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/147 according to rate distortion criteria rate-distortion as a criterion for motion estimation H04N19/567
• CPC Class Codes: H04N 19/19 using optimisation based on Lagrange multipliers
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/517 by encoding
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/567 Motion estimation based on rate distortion criteria

Abstract

Apparatus, and an associated method, motion compensates coding of video sequences. Motion compensated prediction is utilized in the representation of motion vector fields. Reduced numbers of bits are required to represent the motion vector field while maintaining a low prediction error, thereby facilitating improved communication of, and recreation of, video frames forming a video sequence.

Claims

1. In a method of operating on a video sequence, said video sequence being formed of at least a current video frame and a reference video frame, the current video frame comprising at least one first neighboring segment and a second neighboring segment, an improvement of a method for motion compensated prediction of the current video frame comprising:

retrieving a previously stored first motion field model, said first motion field model being a model of a first motion vector field describing the displacements of pixels in the first neighboring segment with respect to pixels in the reference video frame;

determining a second motion vector field describing displacements of pixels in the second neighboring segment of the current video frame with respect to pixels in the reference video frame;

modeling said second motion vector field using a motion model to form a second motion field model;

approximating said second motion field model on the basis of said first motion field model to form a prediction field model;

comparing said second motion field model with said prediction field model and forming a refinement field model, said refinement field model representing the difference between said second motion field model and said prediction field model;

constructing an alternative model representation of said second motion field model by making a summation of said prediction field model and said refinement field model;

calculating a first cost function wherein said first cost function includes a measure of a first image distortion incurred and a measure of a first amount of data required when using said second motion field model;

calculating a second cost function wherein said second cost function includes a measure of a second image distortion incurred and a measure of a second amount of data required when using said alternative model representation of said second motion field;

comparing said first and second cost functions and determining which of said first and second cost functions has a smaller absolute value;

choosing that alternate one of said second motion field model and said alternative model representation of said second motion vector field associated with said smaller absolute value to indicate a chosen motion field model and storing said chosen motion field model.

2. A method according to claim 1 further including:

encoding information about said chosen motion field model.

3. A method according to claim 2 further including:

transmitting said coded information to a decoder for decoding.

4. A method according to claim 2 further including:

storing said coded information in a storage means.

5. A method according to claim 1 wherein each of said first motion field model, said second motion field model, said second motion field model, said prediction field model and said refinement field model is formed as a sum of motion field basis functions, each of said motion field basis functions being multiplied by a motion coefficient.

6. A method according to claim 5 wherein said motion field basis functions are orthogonal functions.

7. A method according to claim 6 wherein each of said first motion field model, said second motion field model, said prediction field model and said refinement field model is an affine motion field model.

8. A method according to claim 1 wherein said at least one first neighboring segment and said second neighboring segment are quadrilateral.

9. A method according to claim 1 further including:

dividing said at least one first neighboring segment into a plurality of sub-segments and using a motion field model of at least one of said sub-segments to form said prediction field model.

10. A method according to claim 1 wherein said prediction field model is formed by projecting the motion field model of said at least one neighboring segment.

11. A method according to claim 1 wherein said prediction field model is formed by averaging approximations of said second motion vector field determined from more than one first neighboring segment.

12. A method according to claim 1 wherein said prediction field model is formed by averaging approximations of said second field model determined from more than one first neighboring segment.

13. A method according to claim 1 wherein said step of calculating said first cost function is performed using a Lagrangian criterion.

14. A method according to claim 13 wherein said Lagrangian criterion has the form L=D+lambda×

B where D is the distortion incurred when encoding a given set of motion coefficients, B is number of bits required to represent the motion coefficients and lambda is a multiplying Lagrangian parameter.

15. A method according to claim 1 wherein said prediction motion field and said refinement motion field are represented using a common set of basis functions.

16. A method according to claim 1 further including:

defining a first threshold value;

identifying a motion coefficient of said refinement field model with the smallest value of all motion coefficients of said refinement field model;

determining a third cost function incurred by setting said smallest motion coefficient to zero;

forming an approximation of said refinement field by setting said smallest valued motion coefficient to zero, in a situation in which said third image distortion does not exceed said first threshold value.

17. A method according to claim 1 wherein if said chosen motion field model is said second motion field model, said method further includes:

setting all motion coefficients of said prediction field model to zero;

setting all motion coefficients of said refinement field model equal to said motion coefficients of said second motion field model.

18. A method according to claim 17 wherein said encoding of information takes place in a manner depending on the chosen field model.

19. A method according to claim 18 wherein if said chosen field model is said second motion field model, said encoding of information includes the step of encoding said refinement field model.

20. A method according to claim 18 wherein if said chosen field model is said alternative model representation, said encoding of information includes the steps of:

encoding said prediction field model;

encoding said refinement field model.

21. A method according to claim 20 wherein said encoding of said refinement field model includes the steps of:

indicating, by said a motion coefficient indicator to one alternate of a first and a second value, that said encoded information includes said motion coefficients of said refinement field model;

indicating, by setting a motion coefficient pattern indicator, which of said motion coefficients have non-zero values;

encoding said non-zero motion coefficient values.

22. A method according to claim 21 wherein each of said non-zero motion coefficient values is encoded by indicating an amplitude value and a sign.

23. A method according to claim 20 wherein encoding of said prediction field model includes the steps of:

indicating, by setting a motion coefficient indicator to one alternate of a first and a second value, that said encoded information does not include motion coefficient values;

indicating, by setting a direction discrimination indicator, the direction with respect to said second neighboring segment of said at least one first neighboring segment from which said alternative model representation is constructed.

24. A method according to claim 23 wherein encoding of said prediction field model includes the further step of:

indicating, by setting a sub-segment discrimination indicator, a sub-segment of said at least one first neighboring segment from which said alternative model representation is constructed.

25. In a method of operating on a video sequence, said video sequence being formed of at least a current video frame and a reference video frame, the current video frame comprising at least one first neighboring segment and a second neighboring segment, an improvement of a method for motion compensated prediction of the current video frame including:

retrieving at least one previously stored first motion field model, said at least one first motion field model being a model of a first motion vector field describing the displacements of pixels in the at least one first neighboring segment with respect to pixels in the reference video frame

determining a second motion vector field describing displacements of pixels in the second neighboring segment of the current video frame with respect to pixels in the reference video frame;

modeling said second motion vector field using a motion model to form a second motion field model

approximating said second motion field model on the basis of said at least one first motion field model to form a prediction field model.

26. In a video device for operating upon a video sequence formed at least of a current video frame, the current video frame having at least a first neighboring segment and a second neighboring segment, an improvement of apparatus for forming an approximation of a motion vector field of the second neighboring segment, said apparatus comprising:

a motion vector field builder coupled to receive indications representative of a first affine motion model forming an approximation of a first motion vector field representative of the first neighboring segment and to receive indications of the second neighboring segment, said motion vector field builder for forming a second affine motion model responsive to the indications representative of the first affine motion model, the second affine motion model forming the approximation of the motion vector field of the second neighboring segment.

27. The apparatus of claim 26 wherein the video sequence is further formed of a reference video frame and wherein said motion vector field builder is further coupled to receive indications of the reference video frame and wherein the second affine motion model is responsive to an alternate one of the indications representative of the first affine model and a selected portion of the reference video frame.

28. The apparatus of claim 27 wherein said motion vector field builder further calculates a second motion vector field, the second motion vector field calculated responsive to the selected portion of the reference video frame,

29. The apparatus of claim 28 wherein said motion vector field builder further determines differences between the second motion vector field and the second affine motion model, differences therebetween forming a refinement field model.

30. The apparatus of claim 29 wheein said motion vector field builder further constructs an alternative-representation model of the second motion vector field, the alternative-representation model of the second motion vector field formed of a combination of the refinement field model and of the second affine motion model.

31. The apparatus of claim 30 wherein said motion vector field builder further determines a cost function, the cost function at least in part a representation of image distortion and of data requirements related to at least a selected one of the second motion vector field and the second affine motion model.

32. The apparatus of claim 31 wherein said motion vector field builder further utilizes a selected one of the second motion vector field and the second affine motion model, selection made responsive to the cost function.

33. The apparatus of claim 27 wherein said motion vector field builder further selects which alternate one of the indications representative of the first affine model and the selected portion of the reference video frame responsive to which the second affine motion model is formed.

34. The apparatus of claim 26 wherein the first affine motion model has first affine motion coefficients associated therewith and wherein said motion vector field builder further projects values of the first affine motion model to form the second affine motion model.

35. The apparatus of claim 26 wherein the current video frame further has a third neighboring segment, the third neighboring segment adjacent to both the first neighboring segment and to the second neighboring segment, wherein said motion vector field builder is further coupled to receive indications of the second neighboring segment, and wherein said motion vector field builder is further for forming a third affine motion model responsive to a selected alternate one of the first affine motion model and the second affine motion model.

36. The apparatus of claim 35 wherein said motion vector field builder further selects which of the alternate one of the first affine motion model and the second affine motion model responsive to which the third affine motion model is formed.

37. The apparatus of claim 26 wherein the video device forms a video sequence generator having an encoder and wherein said motion vector field builder forms a portion of the encoder.

38. The apparatus of claim 26 wherein the video device forms a video sequence receiver having a decoder and wherein said motion field builder forms a portion of the decoder.

39. In a method for operating upon a video sequence formed of at least a current video frame, the current video frame having at least a first neighboring segment and a second neighboring segment, an improvement of a method for forming an approximation of a motion vector field of the second neighboring segment, said method comprising:

forming a first motion vector field representative of the first neighboring segment;

modeling the first motion vector field with a first affine motion model; and

forming a second affine motion vector model responsive to the first motion vector field modeled during said operation of modeling, the second affine motion model forming the approximation of the motion vector field of the second neighboring segment.

40. The method of claim 39 wherein the current video frame further comprises a third neighboring segment, the third neighboring segment adjacent to both the first neighboring segment and to the second neighboring segment, said method further for forming an approximation of a motion vector field of the third neighboring segment, said method comprising the further operation of:

forming a third affine motion model responsive to an alternate one of the first affine motion model and the second affine motion model.

41. The method of claim 39 wherein the video sequence is further formed of a reference video frame and wherein the second affine motion vector field formed during said operation of forming the second affine motion model is responsive to an alternate one of the first motion vector field and a portion of the reference frame.

42. The method of claim 41 further comprising the additional operation of selecting which of the first motion vector field and the reference frame responsive to which the second affine motion model is formed.

43. In a video device for operating upon a video sequence formed at least of a current video frame and a reference video frame, the current video frame having at least a first neighboring segment and a second neighboring segment, an improvement of apparatus of forming an approximation of a motion vector field, said apparatus comprising:

a motion vector field builder coupled to receive indications representative of a selected one of the first neighboring segment and the second neighboring segment and indications representative of portions of the reference video frame, said motion vector field builder for determining a mapping between the selected one of the first and second neighboring segments and a selected portion of the reference video frame and for approximating the mapping with an affine motion model, the affine motion model forming the approximation of the motion vector field.

44. The apparatus of claim 43 wherein the selected one of the first neighboring segment and the second neighboring segment comprises the first neighboring segment and wherein the affine motion model forming the approximation of the motion vector field comprises a first affine motion model, the first affine motion model representative of the first neighboring segment.

45. The apparatus of claim 43 wherein the selected one of the first neighboring segment and the second neighboring segment further comprises the second neighboring segment, said motion vector field builder further for determining a mapping between the second neighboring segment and an alternate one of the selected portion of the reference video frame and the first neighboring segment, and wherein the affine motion model forming the approximation of the motion vector field further comprises a second affine motion model, the second affine motion model representative of the second neighboring segment.

46. The apparatus of claim 43 wherein the video device comprises a video sequence generator having an encoder and wherein said motion vector field builder comprises a portion of the encoder.

47. The apparatus of claim 43 wherein the video device comprises a video sequence receiver having a decoder and wherein said motion vector field builder comprises a portion of the decoder.

48. In a method of decoding a video sequence, said video sequence being formed of at least a current frame and a reference frame, the current frame comprising at least a first neighboring segment and a second neighboring segment, an improvement of a method for decoding said current video frame comprising the steps of:

receiving an indication of an information type;

receiving segment reconstruction information for said second neighboring segment;

selecting a segment reconstruction mode responsive to said indication;

reconstructing said second neighboring segment according to said selected segment reconstruction mode.

49. A method according to claim 48, wherein said selected segment reconstruction mode is one of a set of segment reconstruction modes comprising:

a first segment reconstruction mode wherein said segment reconstruction information comprises an indication of a first neighboring segment to be used in said step of reconstructing said neighboring segment;

a second segment reconstruction mode wherein said segment reconstruction information comprises motion coefficient information.

50. A method according to claim 49, wherein said set of segment reconstruction modes further comprises:

a third segment reconstruction mode wherein said segment reconstruction information comprises an indication of pixel values from said reference frame;

a fourth segment reconstruction mode wherein said segment reconstruction information comprises an indication of pixel values from said current frame.

51. A method according to claim 49, wherein said indication of a first neighboring segment comprises information about the position of said first neighboring segment with respect to said second neighboring segment.

52. A method according to claim 51, wherein said indication of a first neighboring segment further comprises information about a sub-segment within said first neighboring segment.

53. A method according to claim 49 wherein said motion coefficient information comprises an indication of at least one non-zero motion coefficient value.

54. A method according to claim 53, wherein said indication of at least one non-zero motion coefficient value comprises a non-zero coefficient pattern indication and at least one non-zero coefficient value.

55. A method according to claim 49, wherein said first segment reconstruction mode comprises using a prediction motion field model derived from a first motion field model representing said first neighboring segment.

56. A method according to claim 55, wherein said prediction motion field model is constructed by projecting said first motion field model from said first neighboring segment into said second neighboring segment.

57. A method according to claim 49, wherein said second segment reconstruction mode comprises using a refinement motion field model.

58. A method according to claim 57, wherein said refinement motion field model is represented by an indication of at least one motion coefficient value.

59. A method according to claim 57, wherein said refinement motion field model represents a difference between a second motion field model and said prediction motion field-model, wherein said second motion field model is a representation of said second segment derived from said reference frame.

60. A method according to claim 57, wherein said refinement motion field model is a representation of said second segment derived from said reference frame.

61. In a method of encoding a video sequence, said video sequence being formed of at least a current video frame and a reference video frame, the current video frame comprising at least a first neighboring segment and a second neighboring segment, an improvement of a method for motion compensated prediction of the current video frame comprising:

defining a set of coding modes for said second neighboring segment;

calculating a set of cost functions, each one of said cost functions being associated with one of said set of coding modes;

choosing that one of said set of cost functions with a smallest absolute value;

defining that one of said set of coding modes associated with said smallest absolute value as a chosen coding mode for said second neighboring segment;

encoding information about said second neighboring segment according to said chosen coding mode.

62. A method according to claim 61 further comprising:

transmitting said encoded information to a decoder for decoding.

63. A method according to claim 61 further comprising:

storing said coded information in a storage means.

64. A method according to claim 61 wherein said set of coding modes comprises:

a first coding mode wherein a motion field model from said first neighboring segment is projected into said second neighboring segment to form a prediction motion field model and said second neighboring segment is represented by said prediction motion field model;

a second coding mode wherein said second neighboring segment is represented by a motion field model derived from said reference frame;

a third coding mode wherein a motion field model from said first neighboring segment is projected into said second neighboring segment to form a projection field model and said second neighboring segment is represented by said prediction motion field model and a refinement motion field model.

65. A method according to claim 64 wherein said set of coding modes further comprises:

a fourth coding mode wherein said second neighboring segment is encoded using pixel values from said reference frame;

a fifth coding mode wherein said second neighboring segment is encoded using pixel values from said current frame.

66. A method according to claim 64 wherein said refinement motion field model represents a difference between said motion field model derived from said reference frame and said prediction motion field model.

67. A method according to claim 64 wherein said prediction motion field model, said refinement motion field model and said motion field model derived from said reference frame comprise a set of basis functions, each one of said basis functions being multiplied by a motion coefficient value.

68. A method according to claim 67 wherein said basis functions are orthogonal functions.

69. A method according to claim 68 wherein said prediction motion field model, said refinement motion field model and said motion field model derived from said reference frame are affine motion field models.

70. A method according to claim 61 wherein each one of said set of cost functions comprises a measure of an image distortion incurred and a measure of an amount of data required when using a given one of said coding modes.

71. A method according to claim 70 wherein each one of said set of cost functions is calculated using a Lagrangian criterion.

72. A method according to claim 71 wherein said Lagrangian criterion has the form L=D+lambda×

B where D is a measure of the distortion incurred when encoding a given set of motion coefficients, B is the number of bits required to represent the motion coefficients and lambda is a Lagrangian parameter.

73. A method according to claim 67 wherein said prediction motion field and said refinement motion field are represented using a common set of basis functions.

74. A method according to claim 67 wherein said refinement motion field model is approximated by removing a motion coefficient.

75. A method according to claim 64 wherein said current frame comprises a plurality of first neighboring segments, said method further comprising:

forming a plurality of prediction motion field models, one for each of said plurality of first neighboring segments;

forming a plurality of refinement motion field models, each corresponding to a given one of said plurality of prediction motion field models.

76. A method according to claim 75 wherein said prediction motion field model is formed on the basis of more than one first neighboring segment.

77. A method according to claim 76 wherein said prediction motion field model is formed by averaging projections of motion field models from more than one first neighboring segment.

78. A method according to claim 64 wherein said method further comprises dividing said first neighboring segment into a plurality of sub-segments and using a motion field model of at least one of said sub-segments to form said prediction field motion model.

79. A method according to claim 61 wherein said encoding of information takes place in a manner depending on the chosen field model.

80. A method according to claim 79 wherein if said chosen coding mode is said second coding mode, said method further comprises setting all motion coefficients of said refinement motion field model equal to said motion coefficients of said motion field model derived from said reference frame.

81. A method according to claim 80 wherein said encoding of information comprises the step of encoding said refinement motion field model.

82. A method according to claim 79 wherein if said chosen coding mode is said first coding mode, said encoding of information comprises the step of encoding said prediction motion field model.

83. A method according to claim 79 wherein if said chosen coding mode is said third coding mode, said encoding of information comprises the steps of:

encoding said prediction motion field model;

encoding said refinement motion field model.

84. A method according to claim 81 wherein said encoding of said refinement motion field model comprises the steps of:

indicating by setting a motion coefficient indicator to one alternate of a first and a second value that said encoded information includes said motion coefficients of said refinement field model;

indicating by setting a motion coefficient indicator, which of said motion coefficients of said refinement field model have non-zero values;

encoding said non-zero values.

85. A method according to claim 83 wherein said encoding of said refinement motion field model comprises the steps of:

indicating by setting a motion coefficient indicator to one alternate of a first and a second value that said encoded information includes said motion coefficients of said refinement field model;

indicating by setting a motion coefficient pattern indicator, which of said motion coefficients of said refinement field model have non-zero values;

encoding said non-zero values.

86. A method according to claim 84 wherein each of said non-zero coefficient values is encoded by indicating an amplitude and a sign.

87. A method according to claim 82 wherein encoding of said prediction motion field model comprises indicating, by setting a motion coefficient indicator to one alternate of a first and a second value, that said encoded information does not include motion coefficient values.

88. A method according to claim 87 wherein encoding of said prediction motion field model further comprises indicating a direction identifying the relative position with respect to said second neighboring segment of said first neighboring segment from which said prediction motion field model is formed.

89. A method according to claim 88 wherein encoding of said prediction motion field model further comprises indicating by setting a sub-segment discrimination indicator, a sub-segment of said first neighboring segment from which said prediction motion field model is formed.

90. A method according to claim 83 wherein encoding of said prediction motion field model comprises indicating, by setting a motion coefficient indicator to one alternate of a first and a second value, that said encoded information does not include motion coefficient values.

91. A method according to claim 90 wherein encoding of said prediction motion field model further comprises indicating a direction identifying the relative position with respect to said second neighboring segment of said first neighboring segment from which said prediction motion field model is formed.

92. A method according to claim 91 wherein encoding of said prediction motion field model further comprises indicating by setting a sub-segment discrimination indicator, a sub-segment of said first neighboring segment from which said prediction motion field model is formed.