MULTIVIEW VIDEO ENCODING METHOD, MULTIVIEW VIDEO DECODING METHOD, MULTIVIEW VIDEO ENCODING APPARATUS, MULTIVIEW VIDEO DECODING APPARATUS, AND PROGRAM

• US 20120314776A1
• Filed: 02/21/2011
• Published: 12/13/2012
• Est. Priority Date: 02/24/2010
• Status: Abandoned Application

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Shimizu, Shinya, Kimata, Hideaki, Matsuura, Norihiko
• Application Number:
  US13/579,675
• Publication Number:
  US 20120314776A1
• Time in Patent Office: Days
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A highly efficient encoding technique is realized even for a multiview video involved in local mismatches in illumination and color between cameras. A view synthesized picture corresponding to an encoding target frame is synthesized from an already encoded reference view frame taken at a reference view different from an encoding target view simultaneously with the encoding target frame at the encoding target view of a multiview video. For each processing unit region having a predetermined size, a reference region on an already encoded reference frame at the encoding target view corresponding to the view synthesized picture is searched for. A correction parameter for correcting a mismatch between cameras is estimated from the view synthesized picture for the processing unit region and the reference frame for the reference region. The view synthesized picture for the processing unit region is corrected using the estimated correction parameter. A video at the encoding target view is subjected to predictive encoding using the corrected view synthesized picture.

Claims

1-16. -16. (canceled)

17. A multiview video encoding method for encoding a multiview video, the method comprising:

a view synthesized picture generation step of synthesizing, from an already encoded reference view frame taken at a reference view different from an encoding target view of the multiview video simultaneously with an encoding target frame at the encoding target view, a view synthesized picture corresponding to the encoding target frame at the encoding target view;

a reference region estimation step of searching for a reference region on an already encoded reference frame at the encoding target view corresponding to the view synthesized picture for each processing unit region having a predetermined size;

a correction parameter estimation step of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame for the reference region;

a view synthesized picture correction step of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture encoding step of performing predictive encoding of a video at the encoding target view using the corrected view synthesized picture.

19. The multiview video encoding method according to claim 17, further comprising a degree of reliability setting step of setting a degree of reliability indicating certainty of the view synthesized picture for each pixel of the view synthesized picture,

wherein the reference region estimation step assigns a weight to a matching cost of each pixel when the reference region on the reference frame corresponding to the view synthesized picture is searched for, based on the degree of reliability.

21. The multiview video encoding method according to claim 19, wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on the degree of reliability.

23. The multiview video encoding method according to claim 19, further comprising an estimation accuracy setting step of setting estimation accuracy indicating whether or not the reference region has been accurately estimated for each pixel of the view synthesized picture,

wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on any one or both of the estimation accuracy and the degree of reliability.

18. A multiview video encoding method for performing predictive encoding, when a video at an encoding target view of a multiview video is encoded, using an already encoded reference view frame taken at a reference view different from the encoding target view simultaneously with an encoding target frame at the encoding target view and an already encoded reference frame at the encoding target view, the method comprising:

a view synthesized picture generation step of synthesizing, from the reference view frame, a view synthesized picture for the encoding target frame at the encoding target view and a view synthesized picture for the reference frame;

a reference region estimation step of searching for a reference region on the view synthesized picture for the reference frame corresponding to the view synthesized picture for the encoding target frame for each processing unit region having a predetermined size;

a correction parameter estimation step of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame at the same position as that of the reference region;

a view synthesized picture correction step of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture encoding step of performing the predictive encoding of the video at the encoding target view using the corrected view synthesized picture.

20. The multiview video encoding method according to claim 18, further comprising a degree of reliability setting step of setting a degree of reliability indicating certainty of the view synthesized picture for each pixel of the view synthesized picture,

wherein the reference region estimation step assigns a weight to a matching cost of each pixel when the reference region on the reference frame corresponding to the view synthesized picture is searched for, based on the degree of reliability.

22. The multiview video encoding method according to claim 20, wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on the degree of reliability.

24. The multiview video encoding method according to claim 20, further comprising an estimation accuracy setting step of setting estimation accuracy indicating whether or not the reference region has been accurately estimated for each pixel of the view synthesized picture,

wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on any one or both of the estimation accuracy and the degree of reliability.

25. A multiview video decoding method for decoding a multiview video, the method comprising:

a view synthesized picture generation step of synthesizing, from a reference view frame taken at a reference view different from a decoding target view of the multiview video simultaneously with a decoding target frame at the decoding target view, a view synthesized picture corresponding to the decoding target frame at the decoding target view;

a reference region estimation step of searching for a reference region on an already decoded reference frame at the decoding target view corresponding to the view synthesized picture for each processing unit region having a predetermined size;

a correction parameter estimation step of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame for the reference region;

a view synthesized picture correction step of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture decoding step of decoding a decoding target frame subjected to predictive encoding at the decoding target view from encoded data of a video at the decoding target view using the corrected view synthesized picture as a prediction signal.

27. The multiview video decoding method according to claim 25, further comprising a degree of reliability setting step of setting a degree of reliability indicating certainty of the view synthesized picture for each pixel of the view synthesized picture,

wherein the reference region estimation step assigns a weight to a matching cost of each pixel when the reference region on the reference frame corresponding to the view synthesized picture is searched for, based on the degree of reliability.

29. The multiview video decoding method according to claim 27, wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on the degree of reliability.

31. The multiview video decoding method according to claim 27, further comprising an estimation accuracy setting step of setting estimation accuracy indicating whether or not the reference region has been accurately estimated for each pixel of the view synthesized picture,

wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on any one or both of the estimation accuracy and the degree of reliability.

26. A multiview video decoding method for decoding a multiview video, when a video at a decoding target view of the multiview video is decoded, using an already decoded reference view frame taken at a reference view different from the decoding target view simultaneously with a decoding target frame at the decoding target view and an already decoded reference frame at the decoding target view, the method comprising:

a view synthesized picture generation step of synthesizing, from the reference view frame, a view synthesized picture for the decoding target frame at the decoding target view and a view synthesized picture for the reference frame;

a reference region estimation step of searching for a reference region on the view synthesized picture for the reference frame corresponding to the view synthesized picture for the decoding target frame for each processing unit region having a predetermined size;

a correction parameter estimation step of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame at the same position as that of the reference region;

a view synthesized picture correction step of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture decoding step of decoding a decoding target frame subjected to predictive encoding at the decoding target view from encoded data of a video at the decoding target view using the corrected view synthesized picture as a prediction signal.

28. The multiview video decoding method according to claim 26, further comprising a degree of reliability setting step of setting a degree of reliability indicating certainty of the view synthesized picture for each pixel of the view synthesized picture,

wherein the reference region estimation step assigns a weight to a matching cost of each pixel when the reference region on the reference frame corresponding to the view synthesized picture is searched for, based on the degree of reliability.

30. The multiview video decoding method according to claim 28, wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on the degree of reliability.

32. The multiview video decoding method according to claim 28, further comprising an estimation accuracy setting step of setting estimation accuracy indicating whether or not the reference region has been accurately estimated for each pixel of the view synthesized picture,

wherein the correction parameter estimation step assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on any one or both of the estimation accuracy and the degree of reliability.

33. A multiview video encoding apparatus for encoding a multiview video, the apparatus comprising:

a view synthesized picture generation unit which synthesizes, from an already encoded reference view frame taken at a reference view different from an encoding target view of the multiview video simultaneously with an encoding target frame at the encoding target view, a view synthesized picture corresponding to the encoding target frame at the encoding target view;

a reference region estimation unit which searches for a reference region on an already encoded reference frame at the encoding target view corresponding to the view synthesized picture synthesized by the view synthesized picture generation unit for each processing unit region having a predetermined size;

a correction parameter estimation unit which estimates a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing—

unit region and the reference frame for the reference region searched for by the reference region estimation unit;

a view synthesized picture correction unit which corrects the view synthesized picture for the processing unit region using the correction parameter estimated by the correction parameter estimation unit; and

a picture encoding unit which performs predictive encoding of a video at the encoding target view using the view synthesized picture corrected by the view synthesized picture correction unit.

34. The multiview video encoding apparatus according to claim 33, further comprising a degree of reliability setting unit which sets a degree of reliability indicating certainty of the view synthesized picture for each pixel of the view synthesized picture synthesized by the view synthesized picture generation unit,

wherein the reference region estimation unit assigns a weight to a matching cost of each pixel when the reference region on the reference frame corresponding to the view synthesized picture is searched for, based on the degree of reliability set by the degree of reliability setting unit.

35. The multiview video encoding apparatus according to claim 34, wherein the correction parameter estimation unit assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on the degree of reliability set by the degree of reliability setting unit.

36. The multiview video encoding apparatus according to claim 34, further comprising an estimation accuracy setting unit which sets estimation accuracy indicating whether or not the reference region has been accurately estimated for each pixel of the view synthesized picture synthesized by the view synthesized picture generation unit,

wherein the correction parameter estimation unit assigns a weight to a matching cost of each pixel when the correction parameter is estimated, based on any one or both of the estimation accuracy set by the estimation accuracy setting unit and the degree of reliability set by the degree of reliability setting unit.

37. A multiview video decoding apparatus for decoding a multiview video, the apparatus comprising:

a view synthesized picture generation unit which synthesizes, from a reference view frame taken at a reference view different from a decoding target view of the multiview video simultaneously with a decoding target frame at the decoding target view, a view synthesized picture corresponding to the decoding target frame at the decoding target view;

a reference region estimation unit which searches for a reference region on an already decoded reference frame at the decoding target view corresponding to the view synthesized picture synthesized by the view synthesized picture generation unit for each processing unit region having a predetermined size;

a correction parameter estimation unit which estimates a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame for the reference region searched for by the reference region estimation unit;

a view synthesized picture correction unit which corrects the view synthesized picture for the processing unit region using the correction parameter estimated by the correction parameter estimation unit; and

a picture decoding unit which decodes a decoding target frame subjected to predictive encoding at the decoding target view from encoded data of a video at the decoding target view using the view synthesized picture corrected by the view synthesized picture correction unit as a prediction signal.

38. A program for causing a computer of a multiview video encoding apparatus for encoding a multiview video to execute:

a view synthesized picture generation function of synthesizing, from an already encoded reference view frame taken at a reference view different from an encoding target view of the multiview video simultaneously with an encoding target frame at the encoding target view, a view synthesized picture corresponding to the encoding target frame at the encoding target view;

a reference region estimation function of searching for a reference region on an already encoded reference frame at the encoding target view corresponding to the view synthesized picture for each processing unit region having a predetermined size;

a correction parameter estimation function of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame for the reference region;

a view synthesized picture correction function of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture encoding function of performing predictive encoding of a video at the encoding target view using the corrected view synthesized picture.

39. A program for causing a computer of a multiview video decoding apparatus for decoding a multiview video to execute:

a view synthesized picture generation function of synthesizing, from a reference view frame taken at a reference view different from a decoding target view of the multiview video simultaneously with a decoding target frame at the decoding target view, a view synthesized picture corresponding to the decoding target frame at the decoding target view;

a reference region estimation function of searching for a reference region on an already decoded reference frame at the decoding target view corresponding to the view synthesized picture for each processing unit region having a predetermined size;

a correction parameter estimation function of estimating a correction parameter for correcting a mismatch between cameras from the view synthesized picture for the processing unit region and the reference frame for the reference region;

a view synthesized picture correction function of correcting the view synthesized picture for the processing unit region using the estimated correction parameter; and

a picture decoding function of decoding a decoding target frame subjected to predictive encoding at the decoding target view from encoded data of a video at the decoding target view using the corrected view synthesized picture as a prediction signal.