VIEW FRUSTUM CULLING FOR FREE VIEWPOINT VIDEO (FVV)

• US 20130321593A1
• Filed: 08/29/2012
• Published: 12/05/2013
• Est. Priority Date: 05/31/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kirk, Adam G., Gillett, Donald Marcus, Sweeney, Patrick, Fishman, Neil, Eraker, David
• Application Number:
  US13/598,536
• Publication Number:
  US 20130321593A1
• Time in Patent Office: Days
• US Class Current: 348/51
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

The view frustum culling technique described herein allows Free Viewpoint Video (FVV) or other 3D spatial video rendering at a client by sending only the 3D geometry and texture (e.g., RGB) data necessary for a specific viewpoint or view frustum from a server to the rendering client. The synthetic viewpoint is then rendered by the client by using the received geometry and texture data for the specific viewpoint or view frustum. In some embodiments of the view frustum culling technique, the client has both some texture data and 3D geometric data stored locally if there is sufficient local processing power. Additionally, in some embodiments, additional spatial and temporal data can be sent to the client to support changes in the view frustum by providing additional geometry and texture data that will likely be immediately used if the viewpoint is changed either spatially or temporally.

Claims

1. A computer-implemented process for receiving spatial three dimensional video, comprising:

using a client computing device for;

receiving only texture data and geometric data for a given view frustum of a spatial three dimensional video from a server at a client;

rendering the given viewpoint of the spatial three dimensional video at the client using the downloaded texture and geometric data for the given view frustum.

2. The computer-implemented process of claim 1 wherein the client specifies the given view frustum to the server before the texture data and geometric data are downloaded to the client.

3. The computer-implemented process of claim 1 wherein the client receives texture data and geometric data computed by the server based on a viewpoint received from the client.

4. The computer-implemented process of claim 1, further comprising:

checking if texture data or geometric data has been previously downloaded to the client; and

not downloading the texture data or the geometric data which has previously downloaded to the client again.

5. The computer-implemented process of claim 1 wherein additional texture data and corresponding geometric data at the edges of the view frustum is received at the client.

6. The computer-implemented process of claim 1 wherein the client'"'"'s viewpoint is predicted based on the client'"'"'s rate of viewpoint change.

7. The computer-implemented process of claim 6 wherein the view frustum is expanded based on the client'"'"'s predicted viewpoint.

8. The computer-implemented process of claim 6 wherein a lower level of detail of geometric data is received at the client for viewpoints that the client has a lower probability of reaching.

9. The computer-implemented process of claim 1 wherein a lower level of detail of texture data and geometric data is sent for objects in the distance of the client'"'"'s view frustum.

10. The computer-implemented process of claim 1 wherein the geometric data is stored as a spatial representation of all viewpoints possible.

11. The computer-implemented process of claim 1 wherein the spatial representation of all viewpoints possible is defined by three dimensional cells.

12. The computer-implemented process of claim 11 wherein the server stores the cells and wherein the client requests specific cells corresponding to a desired view point to be rendered.

13. The computer-implemented process of claim 11 wherein the server computes the cells to send to the client based on a viewpoint the client wishes to render.

14. The computer-implemented process of claim 11 wherein the three dimensional cells are in a compressed format.

15. A computer-implemented process for receiving free viewpoint video, comprising:

using a client computing device for;

installing a free viewpoint video player on a local client;

selecting a free viewpoint video stored on a server;

receiving a message from the server that tells the client to instantiate the free viewpoint video player with controls appropriate to the selected free viewpoint video type;

instantiating the free viewpoint video player with controls appropriate to the selected free viewpoint video type;

requesting a desired viewpoint of the selected free viewpoint video from the server;

receiving only the necessary geometric and texture data to render the desired viewpoint of the selected viewpoint video; and

combining the received geometric and texture data to render the desired viewpoint of the free viewpoint video

16. The computer-implemented process of claim 15 further comprising:

the client checking for user viewpoint navigation input; and

if there is any user viewpoint navigation input, the client sending the navigation input to the server.

17. The computer-implemented process of claim 16 wherein the server uses the client'"'"'s navigation input to determine which 3D geometry and texture data to next send to the client.

20. The system of claim 17 wherein the downloaded texture data and geometric data is slightly greater than required to render the given viewpoint.

18. A system for providing free viewpoint video, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

download only texture data and geometric data relevant to a given viewpoint of a free viewpoint video at a client;

render the given viewpoint of the free viewpoint video at the client using only the downloaded texture and geometric data for the given viewpoint.

19. The system of claim 18 wherein the downloaded texture data and the downloaded geometric data is downloaded from more than one server in a computing cloud.