Multi-layer UV map based texture rendering for free-running FVV applications

• US 10,453,244 B2
• Filed: 10/02/2017
• Issued: 10/22/2019
• Est. Priority Date: 10/07/2016
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hirota, Yoichi
• Primary Examiner(s): Broome, Said
• Application Number:
  US15/723,021
• Publication Number:
  US 20180101978A1
• Time in Patent Office: 750 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

A scalable texture data representation which enables spatio-temporal texture synthesis for occlusion by using a temporally consistent texture UV map, view-dependent texture mapping for seamless/specular color by using an additional multi-layer texture UV map, and view-dependent texture alignment for natural mapping by using an extra flow UV map, is described herein.

Claims

1. A method programmed in a memory of a device to render a texture on an object in a free viewpoint video content comprising:

implementing inpainting in a UV map domain;

utilizing a multi-layer texture UV map to combine texture segments; and

using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time.

2. The method of claim 1 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

3. The method of claim 1 wherein using the flow UV map includes generating and storing global alignment information of the texture.

4. The method of claim 1 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

5. The method of claim 1 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

6. The method of claim 1 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

7. The method of claim 1 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.

8. An apparatus comprising:

a non-transitory memory for storing an application, the application for;

implementing inpainting in a UV map domain;

utilizing a multi-layer texture UV map to combine texture segments; and

using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time; and

a processor coupled to the memory, the processor configured for processing the application.

9. The apparatus of claim 8 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

10. The apparatus of claim 8 wherein using the flow UV map includes generating and storing global alignment information of the texture.

11. The apparatus of claim 8 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

12. The apparatus of claim 8 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

13. The apparatus of claim 8 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

14. The apparatus of claim 8 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.

15. A system comprising:

a plurality of camera devices each configured for capturing free viewpoint video content;

a first computing device configured for;

implementing inpainting in a UV map domain;

utilizing a multi-layer texture UV map to combine texture segments; and

using a flow UV map to apply texture to a shape to prevent an incorrect shape, wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map produce a temporally consistent texture for the object in the free viewpoint video content, wherein implementing inpainting in the UV map domain includes filling in any occluded aspects of the texture for the object using the texture captured at a different time; and

a second computing device configured for;

displaying the free viewpoint video content on the second computing device.

16. The system of claim 15 wherein utilizing the multi-layer texture UV map includes employing a weighted sum to combine the texture segments.

17. The system of claim 15 wherein using the flow UV map includes generating and storing global alignment information of the texture.

18. The system of claim 15 wherein implementing inpainting, utilizing the multi-layer texture UV map and using the flow UV map are included in processing the free viewpoint video content which further includes geometry processing.

19. The system of claim 15 further comprising transmitting, rendering and displaying the free viewpoint video content including the object with temporally consistent texture.

20. The system of claim 15 further comprising acquiring the free viewpoint video content using at most 10 video cameras.

21. The system of claim 15 further comprising acquiring the free viewpoint video content using video cameras with at least 30 degrees of separation on every axis from other cameras.