Synthetic Reference Picture Generation

• US 20130287289A1
• Filed: 04/25/2012
• Published: 10/31/2013
• Est. Priority Date: 04/25/2012
• Status: Abandoned Application

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Tian, Dong, Graziosi, Danillo Bracco, Vetro, Anthony
• Application Number:
  US13/455,904
• Publication Number:
  US 20130287289A1
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

A synthetic image block in a synthetic picture is generated for a viewpoint based on a texture image and a depth image. A subset of samples from the texture image are warped to the synthetic image block. Disoccluded samples are marked, and the disoccluded samples in the synthetic image block are filled based on samples in a constrained area. The method and system enables both picture level and block level processing for synthetic reference picture generation. The method can be used for power limited devices, and can also refine the synthetic reference picture quality at a block level to achieve coding gains.

Claims

1. A method for generating a synthetic image block in a synthetic picture for a viewpoint based on a texture image and a depth image, comprising the steps of:

warping a subset of samples from the texture image to the synthetic image block;

marking disoccluded samples; and

filling the disoccluded samples in the synthetic image block based on samples in a constrained area, wherein the steps are performed in a codec.

2. The method of claim 1, wherein the depth image corresponds to a viewpoint, and forward warping is performed.

3. The method of claim 1, wherein the depth image corresponds to the viewpoint to be decoded, and backward warping is performed.

4. The method of claim 2, wherein a subset of samples in the texture image is an overlapped image block, further comprising:

determining a maximum disparity D_max;

accessing a location of a current block to be decoded, denoted by a top-left and bottom-right location (X_tl, Y_tl) and (X_br, Y_br);

determining a location of an overlapped block in a reference texture image by applying the maximum disparity D_max, which is (X_tl−

D_max, Y_tl, and (X_br+D_max, Y_br).

5. The method of claim 1, wherein the constrained area for hole filling is the same as a warped block for intra block hole filling.

6. The method of claim 5, wherein the constrained area further comprises the neighboring blocks that are decoded in a current picture being decoded for inter block hole filling.

7. The method of claim 6, further comprising:

performing horizontal prediction from a neighboring block on the left in decoded picture to fill the hole samples.

8. The method of claim 6, further comprising:

performing vertical prediction from a neighboring block on the top in a decoded picture to fill the hole samples.

9. The method of claim 6, further comprising:

performing diagonal prediction from a neighboring block on the top right in a decoded picture to fill the hole samples.

10. The method of claim 6, further comprising:

performing inverse diagonal prediction from a neighboring block on the top left in a decoded picture to fill the hole samples.

11. The method of claim 5, further comprising:

replacing a synthetic block in a synthetic reference picture with a corresponding decoded block to refining the synthetic reference picture.

12. The method of claim 11, further comprising:

performing horizontal prediction from a neighboring block on the left in the synthetic picture to fill the hole samples.

13. The method of claim 11, further comprising:

performing vertical prediction from a neighboring block on the top in the synthetic picture to fill the hole samples.

14. The method of claim 11, further comprising:

performing diagonal prediction from a neighboring block on the top right in the synthetic picture to fill the hole samples.

15. The method of claim 11, further comprising:

performing inverse diagonal prediction from a neighboring block on the top left in the synthetic picture to fill the hole samples.

16. The method of claim 2, wherein a subset of samples in the texture image is an overlapped image block, further comprising:

determining a horizontal maximum disparity D_max, and a vertical maximum disparity D_{max, vertical};

accessing a location of a current block to be decoded, wherein the location is denoted by a top-left and bottom-right location (X_tl, Y_tl) and (X_br, Y_br); and

determining a location of an overlapped block in a reference texture image by applying the maximum disparity D_max, which is (X_tl−

D_max, Y_tl+D_{max, vertical}) and (X_br+D_max, Y_br+D_{max, vertical}).

17. A codec for generating a synthetic image block in a synthetic picture for a viewpoint based on a texture image and a depth image, comprising:

means for warping a subset of samples from the texture image to the synthetic image block;

means for marking disoccluded samples; and

means filling the disoccluded samples in the synthetic image block based on samples in a constrained area, wherein the steps are performed in a coder.

18. A codec using synthetic blocks in a synthetic picture for a viewpoint, comprising:

means for updating a first synthetic block in the synthetic picture with a reconstructed block;

means for updating hole filling for a second synthetic block in the synthetic picture by referencing the first synthetic block; and

means for using the synthetic picture with the updated first and second synthetic blocks as a reference picture to code a next block, wherein the blocks are based on a texture image and a depth image.