Multi-View Video Coding Using Scalable Video Coding

• US 20100165077A1
• Filed: 10/17/2006
• Published: 07/01/2010
• Est. Priority Date: 10/19/2005
• Status: Active Grant

• Current Assignee: InterDigital Madison Patent Holdings (InterDigital, Inc.)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Gomila, Cristina, Yin, Peng, Su, Yeping
• Granted Patent: US 9,131,247 B2
• Time in Patent Office: Days
• US Class Current: 348/42
• CPC Class Codes: H04N 19/31 in the temporal domain
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/615 using motion compensated temporal filtering [MCTF]
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/91 Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Abstract

There are provided methods and apparatus for stereoscopic video coding using scalable video coding. A scalable video encoder includes an encoder (100) for encoding at least two views corresponding to multi-view video content by, encoding a particular view of the at least two views as a base layer, and encoding each of at least one other view of the at least two views as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view. The at least two views are encoded based on a selection from among at least two of temporal, spatial, and signal to noise ratio scalability techniques.

Claims

1. An apparatus comprising:

an encoder for encoding at least two views corresponding to multi-view video content by, encoding a particular view of the at least two views as a base layer, and encoding each of at least one other view of the at least two views as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views are encoded based on a selection from among at least two of temporal, spatial, and signal to noise ratio scalability techniques.

2. The apparatus of claim 1, wherein the base layer and the enhancement layer are included a video bitstream, and said encoder adds an indicator in a message signaling the video bitstream, the indicator for indicating that the video bitstream includes multi-view video signals.

3. The apparatus of claim 2, wherein the message is a Supplemental Enhancement Information (SEI) message.

4. The apparatus of claim 2, wherein the message is sent out of band.

5. The apparatus of claim 2, wherein the indicator is provided as metadata.

6. The apparatus of claim 2, wherein the indicator specifies one of the particular view or the at least one other view as being encoded in one of the base layer or the enhancement layer.

7. The apparatus of claim 1, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

8. The apparatus of claim 1, wherein said encoder constrains a spatial scaling factor to be equal to or less than one, when the spatial scalability technique is selected.

9. The apparatus of claim 8, wherein said encoder encodes at least some blocks in the base layer and the enhancement layer using inter-layer prediction, by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, performing a many-to-one block mapping from the base layer to the enhancement layer, converting any of the at least some blocks in the base layer having an intra prediction mode to an inter-prediction mode, and forcing motion vectors and discrete cosine transform coefficients in the intra-prediction mode to be zero.

10. The apparatus of claim 9, wherein said encoder maps a motion vector for a block in the enhancement layer to a scaled motion vector of a corner of a corresponding mapped block from a set of mapped blocks in the base layer, and assigns a reference index for the block in the enhancement layer to that of the corner of the corresponding mapped block in the base layer, the block in the enhancement layer and the set of mapped blocks in the base layer included in the at least some blocks, the block in the enhancement layer being co-located with respect to the set of mapped blocks in the base layer.

11. The apparatus of claim 8, wherein said encoder encodes at least some blocks in the base layer and the enhancement layer by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, and performing averaging for texture downsampling of residues corresponding to the at least some blocks.

12. The apparatus of claim 1, wherein the at least two views are encoded by enabling weighted prediction for cross view prediction between the particular view and the at least one other view.

13. The apparatus of claim 12, wherein the weighted prediction is enabled when coding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

14. The apparatus of claim 1, wherein the at least two views correspond to a pair of stereoscopic views, with one view of the pair being encoded as the base layer and another view of the pair being encoded as the enhancement layer using a prediction from the base layer.

15. A method for scalable video encoding,

comprising

encoding at least two views corresponding to multi-view video content by, encoding a particular view of the at least two views as a base layer, and encoding each of at least one other view of the at least two views as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views are encoded based on a selection from among at least two of temporal, spatial, and signal to noise ratio scalability techniques.

16. The method of claim 15, wherein the base layer and the enhancement layer are included a video bitstream, and said encoding step comprises adding an indicator in a message signaling the video bitstream, the indicator for indicating that the video bitstream includes multi-view video signals.

17. The method of claim 16, wherein the message is a Supplemental Enhancement Information (SEI) message.

18. The method of claim 16, wherein the message is sent out of band.

19. The method of claim 16, wherein the indicator is provided as metadata.

20. The method of claim 16, wherein the indicator specifies one of the particular view or the at least one other view as being encoded in one of the base layer or the enhancement layer.

21. The method of claim 15, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

22. The method of claim 15, wherein said encoding step comprises constraining a spatial scaling factor to be equal to or less than one, when the spatial scalability technique is selected.

23. The method of claim 22, wherein said encoding step encodes at least some blocks in the base layer and the enhancement layer using inter-layer prediction, by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, performing a many-to-one block mapping from the base layer to the enhancement layer, converting any of the at least some blocks in the base layer having an intra prediction mode to an inter-prediction mode, and forcing motion vectors and discrete cosine transform coefficients in the intra-prediction mode to be zero.

24. The method of claim 23, wherein said encoding step maps a motion vector for a block in the enhancement layer to a scaled motion vector of a corner of a corresponding mapped block from a set of mapped blocks in the base layer, and assigns a reference index for the block in the enhancement layer to that that of the corner of the corresponding mapped block in the base layer, the block in the enhancement layer and the set of mapped blocks in the base layer included in the at least some blocks, the block in the enhancement layer being co-located with respect to the set of mapped blocks in the base layer.

25. The method of claim 22, wherein said encoding step encodes at least some blocks in the base layer and the enhancement layer by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, and performing averaging for texture downsampling of residues corresponding to the at least some blocks.

26. The method of claim 15, wherein the at least two views are encoded by enabling weighted prediction for cross view prediction between the particular view and the at least one other view.

27. The method of claim 26, wherein the weighted prediction is enabled when coding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

28. The method of claim 15, wherein the at least two views correspond to a pair of stereoscopic views, with one view of the pair being encoded as the base layer and another view of the pair being encoded as the enhancement layer using a prediction from the base layer.

30. The apparatus of claim 28, wherein the base layer and the enhancement layer are included a video bitstream, and said decoder reads an indicator in a message signaling the video bitstream, the indicator for indicating that the video bitstream includes multi-view video signals.

34. The apparatus of claim 30, wherein the indicator specifies one of the particular view or the at least one other view as being encoded in one of the base layer or the enhancement layer.

29. An apparatus comprising

a decoder for decoding at least two views corresponding to multi-view video content by, decoding a particular view of the at least two views as a base layer, and decoding each of at least one other view of the at least two views as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views are decoded based on a selection from among at least two of temporal, spatial, and signal to noise ratio scalability techniques.

31. The apparatus of claim 29, wherein the message is a Supplemental Enhancement Information (SEI) message.

32. The apparatus of claim 29, wherein the message is received out of band.

33. The apparatus of claim 29, wherein the indicator is provided as metadata.

35. The apparatus of claim 29, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector I-1.264 standard.

36. The apparatus of claim 29, wherein the at least two views are decoded by enabling weighted prediction for cross view prediction between the particular view and the at least one other view.

37. The apparatus of claim 36, wherein the weighted prediction is enabled when decoding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set set of mapped macroblocks in the base layer.

38. The apparatus of claim 29, wherein the at least two views correspond to a pair of stereoscopic views, with one view of the pair being decoded as the base layer and another view of the pair being decoded as the enhancement layer using a prediction from the base layer.

39. A method for scalable video decoding,

comprising

decoding at least two views corresponding to multi-view video content by, decoding a particular view of the at least two views as a base layer, and decoding each of at least one other view of the at least two views as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views are decoded based on a selection from among at least two of temporal, spatial, and signal to noise ratio scalability techniques.

40. The method of claim 39, wherein the base layer and the enhancement layer are included a video bitstream, and said decoder reads an indicator in a message signaling the video bitstream, the indicator for indicating that the video bitstream includes multi-view video signals.

41. The method of claim 40, wherein the message is a Supplemental Enhancement Information (SEI) message.

42. The method of claim 40, wherein the message is received out of band.

43. The method of claim 40, wherein the indicator is provided as metadata.

44. The method of claim 40, wherein the indicator specifies one of the particular view or the at least one other view as being encoded in one of the base layer or the enhancement layer.

45. The method of claim 39, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

46. The method of claim 39, wherein the at least two views are decoded by enabling weighted prediction for cross view prediction between the first stereoscopic view and the second stereoscopic view.

47. The method of claim 46, wherein the weighted prediction is enabled when decoding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

48. The scalable video encoder of claim 39, wherein the at least two views correspond to a pair of stereoscopic views, with one view of the pair being encoded as the base layer and another view of the pair being encoded as the enhancement layer using a prediction from the base layer.

49. A video signal structure for scalable video encoding,

comprising:

a particular view of at least two views encoded as a base layer; and

at least one other view of the at least two views encoded as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views correspond to multi-view video content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

50. A storage media having scalable video signal data encoded thereupon, comprising:

a particular view of at least two views encoded as a base layer; and

at least one other view of the at least two views encoded as an enhancement layer using a prediction from a lower layer corresponding to at least one of the particular view and the at least one other view,

wherein the at least two views correspond to multi-view video content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

51. An apparatus comprising:

an encoder for encoding a first stereoscopic view as a base layer, and encoding at least one of a depth map and a disparity map as an enhancement layer using a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

52. The apparatus of claim 51, wherein the base layer and the enhancement layer are included a video bitstream, and said encoder adds additional three-dimensional information relating to the particular stereoscopic content in a message signaling the video bitstream.

53. The apparatus of claim 52, wherein the message is a Supplemental Enhancement Information (SEI) message.

54. The apparatus of claim 52, wherein the message is sent out of band.

55. The apparatus of claim 52, wherein the additional three-dimensional information is provided as metadata.

56. The apparatus of claim 52, wherein said encoder reconstructs a non-coded view of the particular stereoscopic content using reconstructed video from the base layer, at least one of a reconstructed depth map and a reconstructed disparity map, and the additional three-dimensional information in the message, and encodes an additional enhancement layer based on the reconstructed non-coded view.

57. The apparatus of claim 51, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

58. The apparatus of claim 51, wherein said encoder constrains a spatial scaling factor to be equal to or less than one, when the spatial scalability technique is selected.

59. The apparatus of claim 58, wherein said encoder encodes at least some blocks in the base layer and the enhancement layer using inter-layer prediction, by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, performing a many-to-one block mapping from the base layer to the enhancement layer, converting any of the at least some blocks in the base layer having an intra prediction mode to an inter-prediction mode, and forcing motion vectors and discrete cosine transform coefficients in the intra-prediction mode to be zero.

60. The apparatus of claim 59, wherein said encoder maps a motion vector for a block in the enhancement layer to a scaled motion vector of a corner of a corresponding mapped block from a set of mapped blocks in the base layer, and assigns a reference index for the block in the enhancement layer to that of the corner of the corresponding mapped block in the base layer, the block in the enhancement layer and the set of mapped blocks in the base layer included in the at least some blocks, the block in the enhancement layer being co-located with respect to the set of mapped blocks in the base layer.

61. The apparatus of claim 58, wherein said encoder encodes at least some blocks in the base layer and the enhancement layer by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, and performing averaging for texture downsampling of residues corresponding to the at least some blocks.

62. The apparatus of claim 51, wherein the pair of stereoscopic views is encoded by enabling weighted prediction for cross view prediction between the first stereoscopic view and the second stereoscopic view.

63. The apparatus of claim 62, wherein the weighted prediction is enabled when coding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

64. A method for scalable video encoding,

comprising:

encoding a first stereoscopic view as a base layer; and

encoding at least one of a depth map and a disparity map as an enhancement layer using a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

65. The method of claim 64, wherein the base layer and the enhancement layer are included a video bitstream, and said encoder adds additional three-dimensional information relating to the particular stereoscopic content in a message signaling the video bitstream.

66. The method of claim 65, wherein the message is a Supplemental Enhancement Information (SEI) message.

67. The method of claim 65, wherein the message is sent out of band.

68. The method of claim 65, wherein the additional three-dimensional information is provided as metadata.

69. The method of claim 65, further comprising:

reconstructing a non-coded view of the particular stereoscopic content using reconstructed video from the base layer, at least one of a reconstructed depth map and a reconstructed disparity map, and the additional three-dimensional information in the message; and

encoding an additional enhancement layer based on the reconstructed non-coded view.

70. The method of claim 64, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

71. The method of claim 64, wherein said encoder constrains a spatial scaling factor to be equal to or less than one, when the spatial scalable video encoding technique is selected.

72. The method of claim 71, wherein said encoding step encodes at least some blocks in the base layer and the enhancement layer using inter-layer prediction, by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, performing a many-to-one block mapping from the base layer to the enhancement layer, converting any of the at least some blocks in the base layer having an intra prediction mode to an inter-prediction mode, and forcing motion vectors and discrete cosine transform coefficients in the intra-prediction mode to be zero.

73. The method of claim 72, wherein said encoding step maps a motion vector for a block in the enhancement layer to a scaled motion vector of a corner of a corresponding mapped block from a set of mapped blocks in the base layer, and assigns a reference index for the block in the enhancement layer to that of the corner of the corresponding mapped block in the base layer, the block in the enhancement layer and the set of mapped blocks in the base layer included in the at least some blocks, the block in the enhancement layer being co-located with respect to the set of mapped blocks in the base layer.

74. The method of claim 72, wherein said encoding step encodes at least some blocks in the base layer and the enhancement layer by setting the spatial scaling factor equal to 2⁽⁻

n), n being an integer greater than zero, and performing averaging for texture downsampling of residues corresponding to the at least some blocks.

75. The method of claim 64, wherein the pair of stereoscopic views is encoded by enabling weighted prediction for cross view prediction between the first stereoscopic view and the second stereoscopic view.

76. The method of claim 75, wherein the weighted prediction is enabled when coding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

77. An apparatus comprising:

a decoder for decoding a first stereoscopic view from a base layer, and decoding at least one of a depth map and a disparity map from an enhancement layer using a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are decoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

78. The apparatus of claim 77, wherein the base layer and the enhancement layer are included a video bitstream, and said decoder reads additional three-dimensional information relating to the particular stereoscopic content from a message signaling the video bitstream.

79. The apparatus of claim 78, wherein the message is a Supplemental Enhancement Information (SEI) message.

80. The apparatus of claim 78, wherein the message is sent out of band.

81. The apparatus of claim 78, wherein the additional three-dimensional information is provided as metadata.

82. The apparatus of claim 78, wherein said decoder reconstructs a non-coded view of the particular stereoscopic content as the base layer using reconstructed video from the base layer, at least one of a reconstructed depth map and a reconstructed disparity map, and the additional three-dimensional information in the message, and decodes an additional enhancement layer based on the reconstructed non-coded view to form a refined non-coded view of the particular stereoscopic content.

83. The apparatus of claim 77, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector 11.264 standard.

84. The apparatus of claim 77, wherein the pair of stereoscopic views is decoded by enabling weighted prediction for cross view prediction between the first stereoscopic view and the second stereoscopic view.

85. The apparatus of claim 84, wherein the weighted prediction is enabled when decoding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

86. A method for scalable video decoding,

comprising:

decoding a first stereoscopic view from a base layer; and

decoding at least one of a depth map and a disparity map from an enhancement layer using a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are decoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

87. The method of claim 86, wherein the base layer and the enhancement layer are included a video bitstream, and method further comprises reading additional three-dimensional information relating to the particular stereoscopic content from a message signaling the video bitstream.

88. The method of claim 87, wherein the message is a Supplemental Enhancement Information (SEI) message.

89. The method of claim 87, wherein the message is sent out of band.

90. The method of claim 87, wherein the additional three-dimensional information is provided as metadata.

91. The method of claim 87, further comprising:

reconstructing a non-coded view of the particular stereoscopic content as the base layer using reconstructed video from the base layer, at least one of a reconstructed depth map and a reconstructed disparity map, and the additional three-dimensional information in the message; and

decoding an additional enhancement layer based on the reconstructed non-coded view to form a refined non-coded view of the particular stereoscopic content.

92. The method of claim 86, wherein the temporal, spatial, and signal to noise ratio scalability techniques are performed in compliance with the Scalable Video Coding Extension of the International Organization for Standardization/International Electrotechnical Commission Moving Picture Experts Group-4 Part 10 Advanced Video Coding standard/International Telecommunication Union, Telecommunication Sector H.264 standard.

93. The method of claim 86, wherein the pair of stereoscopic views is decoded by enabling weighted prediction for cross view prediction between the first stereoscopic view and the second stereoscopic view.

94. The method of claim 93, wherein the weighted prediction is enabled when decoding a difference between a macroblock in the enhancement layer and a weighted macroblock scaled from a set of mapped macroblocks in the base layer, the macroblock in the enhancement layer being co-located with respect to the set of mapped macroblocks in the base layer.

95. A video signal structure for scalable video encoding,

comprising:

a first stereoscopic view encoded as a base layer; and

at least one of a depth map and a disparity map encoded as enhancement layer to allow a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.

96. A storage media having scalable video signal data encoded thereupon, comprising:

a first stereoscopic view encoded as a base layer; and

at least one of a depth map and a disparity map encoded as enhancement layer to allow a prediction from the first stereoscopic view,

wherein the first stereoscopic view and the at least one of the depth map and the disparity map each correspond to a particular stereoscopic content and are encoded based on a selection from among at least two of temporal, spatial, and SNR scalability techniques.