Concealed regions complementing system of free viewpoint video images

• US 8,189,036 B2
• Filed: 12/03/2009
• Issued: 05/29/2012
• Est. Priority Date: 08/19/2003
• Status: Active Grant

• Current Assignee: KDDI Corporation
• Original Assignee: KDDI Corporation
• Inventors: Matsumura, Atsushi, Naito, Sei, Kawada, Ryoichi, Koike, Atsushi, Matsumoto, Shuichi
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): Holder, Anner
• Application Number:
  US12/630,191
• Publication Number:
  US 20100079577A1
• Time in Patent Office: 908 Days
• Field of Search: 348/51, 348/42, 345/629
• US Class Current: 348/51
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation

Abstract

A sender produces a background video image 4 from a reference video image 1 and a depth map 3 and synthesizes the background video image 4 in a background buffer 5. A projection converting matrix 6 for synthesizing is calculated. A receiver receives the reference video image 1, the depth map 3 and the projection converting matrix 6. The receiver produces a free viewpoint video image 11 and a background video image 12. The background buffer 13 is dynamically renewed. The free viewpoint video image 11 and the image in the background buffer 13 are synthesized with each other, and the background video image in the concealed regions in the free viewpoint video image are complemented to obtain the complemented image 14.

Claims

1. A concealed regions complementing system of free viewpoint video images in which when the free viewpoint video images are output, the pixels in background regions concealed by foreground regions in a reference video image are complemented, said system comprising background regions extracting means for extracting, from an input video image, the background regions as a single-layered background video image and multi-layered background video images divided in accordance with depth,

a single-layered background buffer for storing the single-layered background video image, and

multi-layered background buffers for storing the multi-layered background video images, wherein

when the free viewpoint video images are output, the pixels in the background regions concealed by foreground regions in a reference video image are complemented using the pixels of the multi-layered background video images stored in the multi-layered background buffers, and the pixels which are not complemented by this operation are complemented using the pixels of the single-layered background video image stored in the single-layered background buffer.

2. The concealed regions complementing system of the free viewpoint video image according to claim 1, wherein the single-layered background video image stored in the single-layered background buffer and the multi-layered background video images stored in the multi-layered background buffers are produced and renewed for each frame.

3. The concealed regions complementing system of the free viewpoint video image according to claim 2, wherein corresponding points between the background picture stored in the single-layered background buffer and the multi-layered background buffers and a newly input frame video image are searched, and projection converting matrix is calculated, and using this projection converting matrix, the background video images to be stored in the single-layered background buffer and the multi-layered background buffers are renewed.

4. The concealed regions complementing system of the free viewpoint video image according to claim 1, wherein corresponding points between the background video images stored in the single-layered background buffer and the multi-layered background buffers and the free viewpoint video images are searched, the projection converting matrix is calculated, and using this projection converting matrix, the pixels in the concealed background regions are complemented.