Method and apparatus of temporal error concealment for P-frame

• US 20060215761A1
• Filed: 03/09/2006
• Published: 09/28/2006
• Est. Priority Date: 03/10/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Shi, Fang, Raveendran, Vijayalakshmi R.
• Granted Patent: US 8,693,540 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04L 1/0057 Block codes H04L1/0061, H04L1/0064 take precedence
• CPC Class Codes: H04L 1/0066 Parallel concatenated codes
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/895 in combination with error concealment
• CPC Class Codes: H04N 21/64792 Controlling the complexity of the content stream, e.g. by dropping packets intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16

Abstract

A method and apparatus of processing a multimedia frame comprising a first section received in error are disclosed. The method comprises obtaining temporal prediction data corresponding to a first reference frame. The method further comprises determining whether the first reference frame is reliable for estimating the first section. The method further comprises estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

Claims

1. A method of processing a multimedia frame comprising a first section received in error, comprising:

obtaining temporal prediction data corresponding to a first reference frame;

determining whether the first reference frame is reliable for estimating the first section; and

estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

2. The method of claim 1, wherein the one or more sections comprises one or more neighboring sections to the first section.

3. The method of claim 1, wherein the temporal prediction data comprises motion vectors corresponding to portions of the first reference frame.

4. The method of claim 3, wherein the determining further comprises:

determining whether each pair of motion vectors including a motion vector of the multimedia frame and a co-located motion vector within the first reference frame are similar, if the pair is available; and

determining that the first reference frame is reliable if the ratio of the number of similar pairs over the number of available pairs is over a threshold value,

wherein a pair of motion vectors are available if both motion vectors are received correctly.

5. The method of claim 4, wherein whether each pair is similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

6. The method of claim 5, wherein each pair is determined to be not similar only if the magnitudes of both motion vectors are large and the directions of the motion vectors are opposite.

7. The method of claim 1, wherein each section is a macroblock.

8. The method of claim 3, further comprising scaling the motion vectors corresponding to portions of one or more reference frames from which the multimedia frame is to be predicted to motion vectors corresponding to portions of the first reference frame such that the first reference frame is adjacent to the multimedia frame.

9. The method of claim 8, further comprising smoothing at least one of the scaled motion vectors.

10. The method of claim 9, wherein the smoothing comprises median filtering.

11. The method of claim 3, wherein each frame comprises a plurality of sections and each section comprises one or more base units of the same size, the method further comprising:

merging motion vectors corresponding to portions of variable sizes of each base unit to form one base motion vector corresponding to the base unit,

wherein each base units is N by M blocks of pixels, and further wherein N and M are integers.

12. The method of claim 11, wherein merging motion vectors comprises at least one of simple averaging, weighted averaging and median filtering.

13. The method of claim 11, further comprising smoothing at least one of the merged motion vectors.

14. The method of claim 13, wherein the smoothing comprises median filtering.

15. The method of claim 3, wherein the estimating further comprises:

determining an anchor section within the first reference frame;

determining a first motion vector window based on available motion vectors from a second motion vector window of the multimedia frame centered on the first section and from a third motion vector window of the first reference frame centered on the anchor section; and

calculating the median of the available motion vectors within the first motion vector window.

16. The method of claim 15, further comprising updating the first motion vector window such that each motion vector is marked as unavailable if the motion vector is not similar to the motion vector of the anchor section.

17. The method of claim 16, wherein whether two motion vectors are not similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

18. The method of claim 17, wherein two motion vectors are determined to be not similar only if the magnitudes of both motion vectors are large and the directions of both motion vectors are opposite.

19. The method of claim 15, wherein the anchor section is selected to be the section at the corresponding location to the erroneous first section of the multimedia frame.

20. The method of claim 15, wherein the anchor section is selected such that the anchor section corresponds to a section of the first reference frame having the largest overlap with an area of the first reference frame, wherein the area is pointed to by an anchor motion vector of the predicted frame, wherein the anchor motion vector is calculated based on motion vectors of available neighboring sections within the multimedia frame.

21. The method of claim 1, further comprising estimating the first section based on one or more sections other than the first second of the multimedia frame and not based on the first reference frame, if it is determined that the first reference frame is not reliable.

22. The method of claim 1, wherein the method can be applied in both a causal decoding process and a non-causal decoding process.

23. A processor configured to perform a method, the method comprising:

obtaining temporal prediction data corresponding to a first reference frame;

determining whether the first reference frame is reliable for estimating the first section; and

estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

24. The processor of claim 23, wherein the one or more sections comprises one or more neighboring sections to the first section.

25. The processor of claim 23, wherein the temporal prediction data comprises motion vectors corresponding to portions of the first reference frame.

26. The processor of claim 25, wherein the determining further comprises:

determining whether each pair of motion vectors including a motion vector of the multimedia frame and a co-located motion vector within the first reference frame are similar, if the pair is available; and

determining that the first reference frame is reliable if the ratio of the number of similar pairs over the number of available pairs is over a threshold value,

wherein a pair of motion vectors are available if both motion vectors are received correctly.

27. The processor of claim 26, wherein whether each pair is similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

28. The processor of claim 27, wherein each pair is determined to be not similar only if the magnitudes of both motion vectors are large and the directions of the motion vectors are opposite.

29. The processor of claim 23, wherein each section is a macroblock.

30. The processor of claim 25, the method further comprising scaling the motion vectors corresponding to portions of one or more reference frames from which the multimedia frame is to be predicted to motion vectors corresponding to portions of the first reference frame such that the first reference frame is adjacent to the multimedia frame.

31. The processor of claim 30, the method further comprising smoothing at least one of the scaled motion vectors.

32. The processor of claim 31, wherein the smoothing comprises median filtering.

33. The processor of claim 25, wherein each frame comprises a plurality of sections and each section comprises one or more base units of the same size, the method further comprising:

merging motion vectors corresponding to portions of variable sizes of each base unit to form one base motion vector corresponding to the base unit,

wherein each base units is N by M blocks of pixels, and further wherein N and M are integers.

34. The processor of claim 33, wherein merging motion vectors comprises at least one of simple averaging, weighted averaging and median filtering.

35. The processor of claim 33, the method further comprising smoothing at least one of the merged motion vectors.

36. The processor of claim 35, wherein the smoothing comprises median filtering.

37. The processor of claim 25, wherein the estimating further comprises:

determining an anchor section within the first reference frame;

determining a first motion vector window based on available motion vectors from a second motion vector window of the multimedia frame centered on the first section and from a third motion vector window of the first reference frame centered on the anchor section; and

calculating the median of the available motion vectors within the first motion vector window.

38. The processor of claim 37, the method further comprising updating the first motion vector window such that each motion vector is marked as unavailable if the motion vector is not similar to the motion vector of the anchor section.

39. The processor of claim 38, wherein whether two motion vectors are not similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

40. The processor of claim 39, wherein two motion vectors are determined to be not similar only if the magnitudes of both motion vectors are large and the directions of both motion vectors are opposite.

41. The processor of claim 37, wherein the anchor section is selected to be the section at the corresponding location to the erroneous first section of the multimedia frame.

42. The processor of claim 37, wherein the anchor section is selected such that the anchor section corresponds to a section of the first reference frame having the largest overlap with an area of the first reference frame, wherein the area is pointed to by an anchor motion vector of the predicted frame, wherein the anchor motion vector is calculated based on motion vectors of available neighboring sections within the multimedia frame.

43. The processor of claim 23, the method further comprising estimating the first section based on one or more sections other than the first second of the multimedia frame and not based on the first reference frame, if it is determined that the first reference frame is not reliable.

44. The processor of claim 23, wherein the method can be applied in both a causal decoding process and a non-causal decoding process.

45. An apparatus for processing a multimedia frame comprising a first section received in error, comprising:

means for obtaining temporal prediction data corresponding to a first reference frame;

means for determining whether the first reference frame is reliable for estimating the first section; and

means for estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

46. The apparatus of claim 45, wherein the one or more sections comprises one or more neighboring sections to the first section.

47. The apparatus of claim 45, wherein the temporal prediction data comprises motion vectors corresponding to portions of the first reference frame.

48. The apparatus of claim 47, wherein the means for determining further comprises:

means for determining whether each pair of motion vectors including a motion vector of the multimedia frame and a co-located motion vector within the first reference frame are similar, if the pair is available; and

means for determining that the first reference frame is reliable if the ratio of the number of similar pairs over the number of available pairs is over a threshold value,

wherein a pair of motion vectors are available if both motion vectors are received correctly.

49. The apparatus of claim 48, wherein whether each pair is similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

50. The apparatus of claim 49, wherein each pair is determined to be not similar only if the magnitudes of both motion vectors are large and the directions of the motion vectors are opposite.

51. The apparatus of claim 45, wherein each section is a macroblock.

52. The apparatus of claim 47, further comprising means for scaling the motion vectors corresponding to portions of one or more reference frames from which the multimedia frame is to be predicted to motion vectors corresponding to portions of the first reference frame such that the first reference frame is adjacent to the multimedia frame.

53. The apparatus of claim 52, further comprising means for smoothing at least one of the scaled motion vectors.

54. The apparatus of claim 53, wherein the means for smoothing comprises means for median filtering.

55. The apparatus of claim 47, wherein each frame comprises a plurality of sections and each section comprises one or more base units of the same size, further comprising:

means for merging motion vectors corresponding to portions of variable sizes of each base unit to form one base motion vector corresponding to the base unit,

wherein each base units is N by M blocks of pixels, and further wherein N and M are integers.

56. The apparatus of claim 55, wherein the means for merging motion vectors comprises means for at least one of simple averaging, weighted averaging and median filtering.

57. The apparatus of claim 55, further comprising means for smoothing at least one of the merged motion vectors.

58. The apparatus of claim 57, wherein the means for smoothing comprises means for median filtering.

59. The apparatus of claim 47, wherein the means for estimating further comprises:

means for determining an anchor section within the first reference frame;

means for determining a first motion vector window based on available motion vectors from a second motion vector window of the multimedia frame centered on the first section and from a third motion vector window of the first reference frame centered on the anchor section; and

means for calculating the median of the available motion vectors within the first motion vector window.

60. The apparatus of claim 59, further comprising means for updating the first motion vector window such that each motion vector is marked as unavailable if the motion vector is not similar to the motion vector of the anchor section.

61. The apparatus of claim 60, wherein whether two motion vectors are not similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

62. The apparatus of claim 61, wherein two motion vectors are determined to be not similar only if the magnitudes of both motion vectors are large and the directions of both motion vectors are opposite.

63. The apparatus of claim 59, wherein the anchor section is selected to be the section at the corresponding location to the erroneous first section of the multimedia frame.

64. The apparatus of claim 59, wherein the anchor section is selected such that the anchor section corresponds to a section of the first reference frame having the largest overlap with an area of the first reference frame, wherein the area is pointed to by an anchor motion vector of the predicted frame, wherein the anchor motion vector is calculated based on motion vectors of available neighboring sections within the multimedia frame.

65. The apparatus of claim 45, further comprising means for estimating the first section based on one or more sections other than the first second of the multimedia frame and not based on the first reference frame, if it is determined that the first reference frame is not reliable.

66. The apparatus of claim 45, wherein the apparatus can be used in both a causal decoding process and a non-causal decoding process.

67. A computer-readable medium embodying means for causing a computer to execute a method, the method comprising:

obtaining temporal prediction data corresponding to a first reference frame;

determining whether the first reference frame is reliable for estimating the first section; and

estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

68. The computer-readable medium of claim 67, wherein the one or more sections comprises one or more neighboring sections to the first section.

69. The computer-readable medium of claim 67, wherein the temporal prediction data comprises motion vectors corresponding to portions of the first reference frame.

70. The computer-readable medium of claim 69, wherein the determining further comprises:

determining whether each pair of motion vectors including a motion vector of the multimedia frame and a co-located motion vector within the first reference frame are similar, if the pair is available; and

determining that the first reference frame is reliable if the ratio of the number of similar pairs over the number of available pairs is over a threshold value,

wherein a pair of motion vectors are available if both motion vectors are received correctly.

71. The computer-readable medium of claim 70, wherein whether each pair is similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

72. The computer-readable medium of claim 71, wherein each pair is determined to be not similar only if the magnitudes of both motion vectors are large and the directions of the motion vectors are opposite.

73. The computer-readable medium of claim 67, wherein each section is a macroblock.

74. The computer-readable medium of claim 69, the method further comprising scaling the motion vectors corresponding to portions of one or more reference frames from which the multimedia frame is to be predicted to motion vectors corresponding to portions of the first reference frame such that the first reference frame is adjacent to the multimedia frame.

75. The computer-readable medium of claim 74, the method further comprising smoothing at least one of the scaled motion vectors.

76. The computer-readable medium of claim 75, wherein the smoothing comprises median filtering.

77. The computer-readable medium of claim 69, wherein each frame comprises a plurality of sections and each section comprises one or more base units of the same size, the method further comprising:

merging motion vectors corresponding to portions of variable sizes of each base unit to form one base motion vector corresponding to the base unit,

wherein each base units is N by M blocks of pixels, and further wherein N and M are integers.

78. The computer-readable medium of claim 77, wherein merging motion vectors comprises at least one of simple averaging, weighted averaging and median filtering.

79. The computer-readable medium of claim 77, the method further comprising smoothing at least one of the merged motion vectors.

80. The computer-readable medium of claim 79, wherein the smoothing comprises median filtering.

81. The computer-readable medium of claim 69, wherein the estimating further comprises:

determining an anchor section within the first reference frame;

determining a first motion vector window based on available motion vectors from a second motion vector window of the multimedia frame centered on the first section and from a third motion vector window of the first reference frame centered on the anchor section; and

calculating the median of the available motion vectors within the first motion vector window.

82. The computer-readable medium of claim 81, the method further comprising updating the first motion vector window such that each motion vector is marked as unavailable if the motion vector is not similar to the motion vector of the anchor section.

83. The computer-readable medium of claim 82, wherein whether two motion vectors are not similar is determined based on at least one of the following:

the magnitudes of both motion vectors and the directions of both motion vectors.

84. The computer-readable medium of claim 83, wherein two motion vectors are determined to be not similar only if the magnitudes of both motion vectors are large and the directions of both motion vectors are opposite.

85. The computer-readable medium of claim 81, wherein the anchor section is selected to be the section at the corresponding location to the erroneous first section of the multimedia frame.

86. The computer-readable medium of claim 81, wherein the anchor section is selected such that the anchor section corresponds to a section of the first reference frame having the largest overlap with an area of the first reference frame, wherein the area is pointed to by an anchor motion vector of the predicted frame, wherein the anchor motion vector is calculated based on motion vectors of available neighboring sections within the multimedia frame.

87. The computer-readable medium of claim 67, the method further comprising estimating the first section based on one or more sections other than the first second of the multimedia frame and not based on the first reference frame, if it is determined that the first reference frame is not reliable.

88. The computer-readable medium of claim 67, wherein the method can be applied in both a causal decoding process and a non-causal decoding process.

89. An apparatus for processing a multimedia frame comprising a first section received in error, comprising:

a temporal prediction data generator configured to obtain temporal prediction data corresponding to a first reference frame;

a frame motion change detector configured to determine whether the first reference frame is reliable for estimating the first section; and

a motion vector estimator configured to estimating the first section received in error based on one or more sections other than the first section of the multimedia frame and the first reference frame, if it is determined that the first reference frame is reliable.

90. The device of claim 89, wherein the one or more sections comprises one or more neighboring sections to the first section.