Multi-view video coding and decoding methods and apparatuses, coder, and decoder

• US 9,467,676 B2
• Filed: 05/23/2014
• Issued: 10/11/2016
• Est. Priority Date: 11/23/2011
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Yang, Mingyuan, Fu, Jiali, Jaeger, Fabian, Wien, Mathias
• Primary Examiner(s): WILLIAMS, JEFFERY A
• Application Number:
  US14/285,962
• Publication Number:
  US 20140254690A1
• Time in Patent Office: 872 Days
• Field of Search: 375/240.26
• US Class Current: 1/1
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Embodiments of the present invention provide multi-view video coding and coding methods and corresponding apparatuses. The multi-view video coding method includes: minimizing an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset; calculating disparity information between the front view image and the currently coded view image by using the optimal warping offset, a camera parameter of a view, and depth image information of the front view image; and calculating the warped view image of the front view image by using the disparity information and the front view image, and predicting a current view image by using the warped view image as a prediction signal.

Claims

1. A multi-view video coding method, comprising:

minimizing an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset;

obtaining disparity information between the front view image and the currently coded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image; and

determining the warped view image of the front view image according to the disparity information and the front view image, and performing predictive coding on the currently coded view image by using the warped view image as a prediction signal.

2. The multi-view video coding method according to claim 1, wherein the error between the currently coded view image and the warped view image of the front view image comprises:

a sum of absolute differences SAD, or a mean square error MSE, or a sum of squares for error SSE.

3. The multi-view video coding method according to claim 1, wherein the optimal warping offset is an optimal warping offset of both a warped video image of the front view image and a warped depth image, or the optimal warping offset comprises respective optimal warping offsets of the warped video image of the front view image and a warped depth image.

4. The multi-view video coding method according to claim 1, wherein minimizing an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset comprises:

setting a preset offset range, correcting the disparity information between the front view image and the currently coded view image by separately using all values in the preset offset range as a warping offset, obtaining the warped view image of the front view image by using the disparity information, calculating the error between the currently coded view image and the warped view image of the front view image, selecting, from the preset offset range, a value making the error between the currently coded view image and the warped view image of the front view image minimum, and determining the value as the optimal warping offset, wherein all the values in the preset offset range comprise an integral value or a fractional value, and the fractional value is a fractional value with one decimal place.

5. The multi-view video coding method according to claim 1, wherein minimizing an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset comprises:

a. setting a warping offset value offset_i;

b. determining the disparity information between the front view image and the currently coded view image according to the warping offset value offset_i, the camera parameter of the view, and the depth image information of the front view image;

c. determining the warped view image of the front view image according to the disparity information and the front view image;

d. calculating an error between the warped view image and an original image of the currently coded view image;

e. determining whether the error between the warped view image and the original image of the currently coded view image is less than a current minimum error value;

if yes, performing step f;

otherwise, performing step g;

f. setting offset_opt=offset_i and MSE(offset_opt)=MSE(offset_i) and

g. changing the warping offset value offset_i within a preset offset range that is preset, returning to step b, and a cycle ends when all values in the preset offset range are traversed.

6. The method according to claim 1, wherein obtaining disparity information between the front view image and the currently coded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image comprises:

calculating an initial disparity signal according to the camera parameter of the view and the depth image information of the front view image;

obtaining an offset correction signal according to the optimal warping offset and the depth image information of the front view image; and

obtaining the disparity information between the front view image and the currently coded view image according to the initial disparity signal and the offset correction signal.

7. The method according to claim 1, wherein the disparity information between the front view image and the currently coded view image is obtained by using the following formula:

8. The multi-view video coding method according to claim 1, wherein after minimizing an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset, the method further comprises:

writing the optimal warping offset into a code stream.

9. The multi-view video coding method according to claim 1, wherein after obtaining disparity information between the front view image and the currently coded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image, comprises:

calculating the warped view image of the front view image according to the disparity information and the front view image, and performing predictive coding on the currently coded view image by using the warped view image as a prediction signal, so as to obtain a first prediction result;

performing predictive coding on the currently coded view image by using another predictive coding mode of the currently coded view image, so as to obtain a second prediction result; and

performing predictive coding on a current coding unit by selecting, according to the first prediction result and the second prediction result and by using an optimal decision, a best mode from a predictive mode of the warped view image and another predictive mode of the current view image, and writing a mode indicating symbol into a code stream, wherein the mode indicating symbol is used for selecting the best mode from the predictive mode of the warped view image and the another predictive mode of the current view image by using the optimal decision.

10. The multi-view video coding method according to claim 1, wherein calculating the warped view image of the front view image by using the disparity information and the front view image, and predicting the currently coded view image by using the warped view image as a prediction signal comprises:

calculating the warped view image of the front view image by using the disparity information and the front view image;

obtaining an occupancy mask of the currently coded view image, wherein the occupancy mask is used for describing whether a pixel of the front view image can be warped into the currently coded view image;

obtaining a mask identifier of each coding unit of the currently coded view image according to the occupancy mask;

calculating an occupation ratio of a current coding unit according to the mask identifier of each coding unit of the currently coded view image; and

determining whether the occupation ratio of the current coding unit is greater than a preset threshold;

if yes, performing predictive coding on the currently coded view image by using the warped view image as a prediction signal of the current coding unit; and

if not, performing predictive coding on the current coding unit by selecting, by using an optimal decision, a best mode from a predictive mode of the warped view image and another predictive mode of a current view image.

11. The multi-view video coding method according to claim 10, wherein when the occupation ratio of the current coding unit is less than or equal to the preset threshold, the method further comprises:

identifying, by using a mode indicating symbol, the best mode selected, by using the optimal decision, from the predictive mode of the warped view image and the another predictive mode of the current view image, and writing the mode indicating symbol into a code stream.

12. The multi-view video coding method according to claim 9, wherein the optimal decision is a rate-distortion decision.

13. The multi-view video coding method according to claim 10, wherein the occupation ratio p(CU) of the current coding unit is calculated by using the following formula:

14. A multi-view video decoding method, comprising:

parsing a code stream of a current view image to obtain an optimal warping offset;

determining disparity information between a front view image and a currently decoded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image;

calculating a warped view image of the front view image according to the disparity information and the front view image; and

using the warped view image as a reference image for reconstructing the currently decoded view image.

15. The multi-view video decoding method according to claim 14, wherein the optimal warping offset is an optimal warping offset of both a warped video image of the front view image and a warped depth image, or the optimal warping offset comprises respective optimal warping offsets of the warped video image of the front view image and the warped depth image.

16. The multi-view video decoding method according to claim 14, wherein obtaining disparity information between the front view image and the currently coded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image comprises:

calculating an initial disparity signal according to the camera parameter of the view and the depth image information of the front view image;

obtaining an offset correction signal according to the optimal warping offset and the depth image information of the front view image; and

obtaining the disparity information between the front view image and the currently coded view image according to the initial disparity signal and the offset correction signal.

17. The multi-view video decoding method according to claim 14, wherein when the disparity information between the front view image and the currently decoded view image is determined according to the optimal warping offset, the camera parameter of the view, and the depth image information of the front view image, the disparity information between the front view image and the currently decoded view image is calculated by using the following formula:

18. The multi-view video decoding method according to claim 14, wherein calculating the warped view image of the front view image by using the disparity information and the front view image, and predicting the currently decoded view image by using the warped view image as a prediction signal comprises:

when a code stream of a current decoding unit is decoded, obtaining a mode indicating symbol by decoding, and determining, according to the mode indicating symbol, whether to perform predictive decoding on the current decoding unit by using the warped view image as a prediction signal of the current view image, or to perform predictive decoding on a decoding unit of the current view image by using another prediction signal.

19. The multi-view video decoding method according to claim 14, wherein calculating the warped view image of the front view image by using the disparity information and the front view image, and predicting the currently decoded view image by using the warped view image as a prediction signal comprises:

calculating the warped view image of the front view image by using the disparity information and the front view image;

obtaining an occupancy mask of the currently decoded view image, wherein the occupancy mask is used for describing whether a pixel of the front view image can be warped into the currently decoded view image;

obtaining a mask identifier of each decoding unit of the currently decoded view image according to the occupancy mask;

calculating an occupation ratio of a current decoding unit according to the mask identifier of each decoding unit of the currently decoded view image; and

determining whether the occupation ratio of the current decoding unit is greater than a preset threshold;

if yes, performing predictive decoding on the currently decoded view image by using the warped view image as a prediction signal of the current decoding unit;

otherwise, decoding a mode indicating symbol of the current decoding unit, and determining, according to the mode indicating symbol, whether to perform predictive decoding on a current decoding unit by using a best mode selected from a predictive mode of the warped view image and another predictive mode of the currently decoded view image.

20. A multi-view video coding apparatus, comprising:

an optimal warping offset obtaining unit, configured to minimize an error between a currently coded view image and a warped view image of a front view image to obtain an optimal warping offset;

a first disparity information calculating unit, configured to obtain disparity information between the front view image and the currently coded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image;

a first warped view image calculating unit, configured to determine the warped view image of the front view image according to the disparity information and the front view image; and

a first coding prediction unit, configured to perform predictive coding on the currently coded view image by using the warped view image as a prediction signal.

21. The multi-view video coding apparatus according to claim 20, wherein the optimal warping offset obtaining unit comprises:

an initial setting module, configured to set a warping offset value offset_i;

a disparity calculating module, configured to determine the disparity information between the front view image and the currently coded view image according to the warping offset value offset_i, the camera parameter of the view, and the depth image information of the front view image;

a warped view image calculating module, configured to determine the warped view image of the front view image according to the disparity information and the front view image;

an error calculating module, configured to calculate an error between the warped view image and an original image of the currently coded view image;

an error determining module, configured to determine whether the error between the warped view image and the original image of the currently coded view image is less than a current minimum error value; and

an offset value change module, configured to set offset_opt=offset_i and MSE(offset_opt)=MSE(offset_i) and change the warping offset value offset_i within a preset offset range, which is preset, when the error determining module determines that the error between the warped view image and the original image of the currently coded view image is less than the current minimum error value;

otherwise, change the warping offset value offset_i within the preset offset range that is preset.

22. The multi-view video coding apparatus according to claim 20, wherein the apparatus further comprises:

a first mode prediction unit, configured to calculate the warped view image of the front view image according to the disparity information and the front view image, and perform predictive coding on the currently coded view image by using the warped view image as a prediction signal, so as to obtain a first prediction result;

a second mode prediction unit, configured to perform predictive coding on the currently coded view image by using another predictive coding mode of the currently coded view image, so as to obtain a second prediction result; and

a mode selection unit, configured to perform predictive coding on a current coding unit by selecting, according to the first prediction result and the second prediction result and by using an optimal decision, a best mode from a predictive mode of the warped view image and another predictive mode of the currently coded view image, and write a mode indicating symbol into a code stream, wherein the mode indicating symbol is used for identifying the best mode that is selected, by using the optimal decision, from the predictive mode of the warped view image and the another predictive mode of the currently coded view image.

23. The multi-view video coding apparatus according to claim 20, wherein the apparatus further comprises:

a first occupancy mask obtaining unit, configured to obtain an occupancy mask of the currently coded view image, wherein the occupancy mask is used for describing whether a pixel of the front view image can be warped into the currently coded view image;

a first mask identifier obtaining unit, configured to obtain a mask identifier of each coding unit of the currently coded view image according to the occupancy mask;

a first occupation ratio calculating unit, configured to calculate an occupation ratio of a current coding unit according to the mask identifier of each coding unit of the currently coded view image; and

a second coding prediction unit, configured to determine whether the occupation ratio of the current coding unit is greater than a preset threshold;

if yes, perform predictive coding on the currently coded view image by using the warped view image as a prediction signal of the current coding unit; and

if not, perform predictive coding on the current coding unit by selecting, by using an optimal decision, a best mode from a predictive mode of the warped view image and another coding prediction mode of the currently coded view image.

24. A multi-view video decoding apparatus, comprising:

a code stream parsing unit, configured to parse a code stream of a current view image to obtain an optimal warping offset;

a second disparity information calculating unit, configured to determine disparity information between a front view image and a currently decoded view image according to the optimal warping offset, a camera parameter of a view, and depth image information of the front view image;

a second warped view image calculating unit, configured to calculate a warped view image of the front view image according to the disparity information and the front view image; and

a first decoding prediction unit, configured to use the warped view image as a reference image for reconstructing the currently decoded view image.

25. The multi-view video decoding apparatus according to claim 24, further comprising:

a mode indicating symbol decoding unit, configured to obtain a mode indicating symbol by decoding when a current decoding unit is decoded; and

a second decoding prediction unit, configured to determine, according to the mode indicating symbol, whether to perform predictive decoding on the current decoding unit by using the warped view image as a prediction signal of the current view image, or to perform predictive decoding on a decoding unit of the current view image by using another prediction signal.

26. The multi-view video decoding apparatus according to claim 24, further comprising:

a second occupancy mask obtaining unit, configured to obtain an occupancy mask of the currently decoded view image, wherein the occupancy mask is used for describing whether a pixel of the front view image can be warped into the currently decoded view image;

a second mask identifier obtaining unit, configured to obtain a mask identifier of each decoding unit of the currently decoded view image according to the occupancy mask;

a second occupation ratio calculating unit, configured to calculate an occupation ratio of a current decoding unit according to the mask identifier of each decoding unit of the currently decoded view image; and

a third decoding prediction unit, configured to determine whether the occupation ratio of the current decoding unit is greater than a preset threshold;

if yes, perform predictive decoding on the current view image by using the warped view image as a prediction signal of the current decoding unit;

otherwise, decode a mode indicating symbol of the current decoding unit, and determine, according to the mode indicating symbol, whether to decode the current decoding unit by using a predictive mode of the warped view image or to perform predictive decoding on the current decoding unit by using another predictive mode of the currently decoded view image.