CLOUD BASED FREE VIEWPOINT VIDEO STREAMING
First Claim
1. A computer-implemented process for generating a free viewpoint video (FVV) of a scene, comprising:
- using one or more computing devices to perform the following process actions;
capturing the scene using an arrangement of sensors, said arrangement comprises a plurality of sensors that generate a plurality of streams of sensor data each of which represents the scene from a different geometric perspective;
inputting and calibrating the streams of sensor data;
generating scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time;
receiving a current synthetic viewpoint of the scene from a client computing device via a data communication network, said current synthetic viewpoint having been selected by an end user of the client computing device;
generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene; and
transmitting each frame generated to the client computing device via the data communication network for display to the end user of the client computing device.
2 Assignments
0 Petitions
Accused Products
Abstract
Cloud based FVV streaming technique embodiments presented herein generally employ a cloud based FVV pipeline to create, render and transmit FVV frames depicting a captured scene as would be viewed from a current synthetic viewpoint selected by an end user and received from a client computing device. The FVV frames use a similar level of bandwidth as a conventional streaming movie would consume. To change viewpoints, a new viewpoint is sent from the client to the cloud, and a new streaming movie is initiated from the new viewpoint. Frames associated with that viewpoint are created, rendered and transmitted to the client until a new viewpoint request is received.
50 Citations
20 Claims
-
1. A computer-implemented process for generating a free viewpoint video (FVV) of a scene, comprising:
-
using one or more computing devices to perform the following process actions; capturing the scene using an arrangement of sensors, said arrangement comprises a plurality of sensors that generate a plurality of streams of sensor data each of which represents the scene from a different geometric perspective; inputting and calibrating the streams of sensor data; generating scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time; receiving a current synthetic viewpoint of the scene from a client computing device via a data communication network, said current synthetic viewpoint having been selected by an end user of the client computing device; generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene; and transmitting each frame generated to the client computing device via the data communication network for display to the end user of the client computing device. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A system for generating a free viewpoint video (FVV) of a scene, comprising:
-
an arrangement of sensors used to capture video of the scene, or audio, or both, each of said sensors producing a stream of sensor data representing the scene from a different geometric perspective; at least one general purpose computing device; and a computer program comprising program modules executed by the computing device or devices, wherein the computing device or devices are directed by the program modules of the computer program to, input and calibrate the streams of sensor data produced by the arrangement of sensors, generate scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time, store the scene proxies as they are generated, generate a sequence of frames using the stored scene proxies, each frame of which depicts at least a portion of the scene as viewed from a current synthetic viewpoint of the scene, and transmit each frame generated to a client computing device via a data communication network for display to the end user of the client computing device. - View Dependent Claims (14, 15, 16)
-
-
17. A computer-implemented process for playing a free viewpoint video (FVV) of a scene, comprising:
-
using a client computing device to perform the following process actions; inputting a request from an end user to display a FVV selection user interface screen that allows the end user to select a FVV available for playing; displaying the FVV selection user interface screen on a display device; inputting an end user FVV selection; transmitting the end user FVV selection to a server via a data communication network; receiving an instruction from the server via the data communication network that instructs the client computing device to instantiate end user controls appropriate for the type of FVV selected; displaying a FVV control user interface on said display device; monitoring for end user inputs via the FVV control user interface; whenever an end user viewpoint navigation input is received via the FVV control user interface, transmitting the input to the server via the data communication network; receiving FVV frames from the server via the data communication network, each FVV frame depicting at least a portion of the scene as it would be viewed from either an initial viewpoint if the end user has not yet input a viewpoint navigation input via the FVV control user interface or the last viewpoint the end user input using a viewpoint navigation input via the FVV control user interface; and displaying each FVV frame on said display device as it is received. - View Dependent Claims (18, 19, 20)
-
Specification