CLOUD BASED FREE VIEWPOINT VIDEO STREAMING

• US 20130321586A1
• Filed: 08/17/2012
• Published: 12/05/2013
• Est. Priority Date: 05/31/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sweeney, Patrick, Gillett, Don, Fishman, Neil, Mitra, Kanchan, Mital, Amit, Harnett, David, Eshet, Yaron, Winder, Simon, Eraker, David, Kirk, Adam
• Application Number:
  US13/588,917
• Publication Number:
  US 20130321586A1
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

Cloud based FVV streaming technique embodiments presented herein generally employ a cloud based FVV pipeline to create, render and transmit FVV frames depicting a captured scene as would be viewed from a current synthetic viewpoint selected by an end user and received from a client computing device. The FVV frames use a similar level of bandwidth as a conventional streaming movie would consume. To change viewpoints, a new viewpoint is sent from the client to the cloud, and a new streaming movie is initiated from the new viewpoint. Frames associated with that viewpoint are created, rendered and transmitted to the client until a new viewpoint request is received.

Claims

1. A computer-implemented process for generating a free viewpoint video (FVV) of a scene, comprising:

using one or more computing devices to perform the following process actions;

capturing the scene using an arrangement of sensors, said arrangement comprises a plurality of sensors that generate a plurality of streams of sensor data each of which represents the scene from a different geometric perspective;

inputting and calibrating the streams of sensor data;

generating scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time;

receiving a current synthetic viewpoint of the scene from a client computing device via a data communication network, said current synthetic viewpoint having been selected by an end user of the client computing device;

generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the current synthetic viewpoint of the scene; and

transmitting each frame generated to the client computing device via the data communication network for display to the end user of the client computing device.

2. The process of claim 1, further comprising, prior to performing the process action of receiving a current synthetic viewpoint of the scene from the client computing device, performing the actions of:

generating a sequence of frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from a prescribed default viewpoint of the scene; and

transmitting each frame generated to the client computing device via the data communication network for display to the end user of the client computing device.

3. The process of claim 1, wherein the process action of generating a scene proxy comprises the actions of:

monitoring and periodically analyzing current conditions related to at least one of capturing streams of sensor data, or calibrating the streams of sensor data, or generating scene proxies, or generating a sequence of frames, or transmitting each frame generated; and

each time the current conditions are analyzed,

using results of the periodic analysis to select one or more different 3D reconstruction methods which are matched to the current conditions,

using each selected 3D reconstruction method to generate a 3D reconstruction of at least part of the scene from the calibrated streams of sensor data, and

using the 3D reconstructions and the results of the periodic analysis to generate scene proxies.

4. The process of claim 1, wherein each scene proxy comprises one or more types of geometric proxy data, and wherein the process action of generating scene proxies, comprises an action of, for each scene proxy generated, matching the scene proxy to a set of current conditions related to at least one of capturing streams of sensor data or calibrating the streams of sensor data, so as to maximize the photo-realism of frames generated from the scene proxy.

5. The process of claim 1, wherein the process action of generating a sequence of frames, comprises the actions of,

monitoring and periodically analyzing current conditions related to at least one of capturing streams of sensor data, or calibrating the streams of sensor data, or generating scene proxies, or generating a sequence of frames, or transmitting each frame generated;

and

each time the current conditions are analyzed,

using results of the periodic analysis to select one or more different image-based rendering methods which are matched to the current conditions, and

using each selected image-based rendering method and the results of the period analysis to generate the sequence of frames.

6. The process of claim 1, further comprising an action of, for each frame transmitted to the client computing device, also transmitting at least some of the scene proxies used to generate the frame.

7. The process of claim 1, wherein the process action of generating a sequence of frames, comprises generating each frame so as to depict at least a portion of the scene as viewed from the current synthetic viewpoint of the scene that is larger than a display device associated with the client computing device is capable of displaying.

8. The process of claim 1, further comprising the actions of:

predicting one or more synthetic viewpoints of the scene that may be received from the client computing device in the future;

for each predicted synthetic viewpoint of the scene,

generating one or more frames using the scene proxies, each frame of which depicts at least a portion of the scene as viewed from the predicted synthetic viewpoint of the scene, and

storing each frame generated based on the predicted synthetic viewpoint of the scene; and

whenever a current synthetic viewpoint of the scene is received from the client computing device that matches one of the predicted synthetic viewpoints, prior to performing the process action of generating a sequence of frames using the scene proxies, transmitting each frame generated based on the matched predicted synthetic viewpoint of the scene to the client computing device via the data communication network for display to the end user of the client computing device.

9. The process of claim 1, further comprising:

prior to performing the process actions of generating and transmitting a sequence of frames, performing the process actions of,

storing each scene proxy generated, and

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to provide FVV frames from a specified temporal location in the FVV; and

wherein

the process action of generating a sequence of frames further comprises using the stored scene proxies to generate frames, the first of which corresponds to the last user-specified temporal location in the FVV.

10. The process of claim 1, further comprising:

performing prior to performing the process actions of generating and transmitting a sequence of frames, the process actions of,

storing each scene proxy generated, and

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to provide FVV frames in reverse order from a specified temporal location in the FVV thereby rewinding the FVV; and

wherein

the process action of generating a sequence of frames comprises using the stored scene proxies to generate frames in reverse order, the first of which corresponds to the last user-specified temporal location in the FVV.

11. The process of claim 1, further comprising the process actions of:

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to pause the FVV; and

suspending the generation and transmission of FVV frames to the client computing device.

12. The process of claim 11, further comprising the process actions of:

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to restart the paused FVV; and

restarting the generation and transmission of FVV frames to the client computing device.

13. A system for generating a free viewpoint video (FVV) of a scene, comprising:

an arrangement of sensors used to capture video of the scene, or audio, or both, each of said sensors producing a stream of sensor data representing the scene from a different geometric perspective;

at least one general purpose computing device; and

a computer program comprising program modules executed by the computing device or devices, wherein the computing device or devices are directed by the program modules of the computer program to,

input and calibrate the streams of sensor data produced by the arrangement of sensors,

generate scene proxies from the calibrated streams of sensor data, which geometrically describe the scene as a function of time,

store the scene proxies as they are generated,

generate a sequence of frames using the stored scene proxies, each frame of which depicts at least a portion of the scene as viewed from a current synthetic viewpoint of the scene, and

transmit each frame generated to a client computing device via a data communication network for display to the end user of the client computing device.

14. The system of claim 13, further comprising a program module for receiving current synthetic viewpoints of the scene from a client computing device via a data communication network, each of said current synthetic viewpoints having been selected by an end user of the client computing device, and wherein said program module for generating a sequence of frames using the stored scene proxies, employs each current synthetic viewpoint received to generate a sequence of frames, each frame of which depicts at least a portion of the scene as viewed from the last-received current synthetic viewpoint.

15. The system of claim 14, wherein prior to receiving a first current synthetic viewpoint of the scene from a client computing device, said program module for generating a sequence of frames using the stored scene proxies, employs a prescribed default synthetic viewpoint as the current synthetic viewpoint to generate a sequence of frames of the scene.

16. The system of claim 13, wherein prior to executing the program module for transmitting each frame generated, executing additional program modules for:

storing each frame generated; and

receiving a temporal navigation instruction from the client computing device via the data communication network, said temporal navigation instruction having been specified by the end user of the client computing device and representing an instruction to provide FVV frames from a specified temporal location in the FVV; and

wherein

the program module for transmitting each frame generated to the client computing device, comprises transmitting frames, the first of which corresponds to a frame assigned to the last user-specified temporal location in the FVV.

17. A computer-implemented process for playing a free viewpoint video (FVV) of a scene, comprising:

using a client computing device to perform the following process actions;

inputting a request from an end user to display a FVV selection user interface screen that allows the end user to select a FVV available for playing;

displaying the FVV selection user interface screen on a display device;

inputting an end user FVV selection;

transmitting the end user FVV selection to a server via a data communication network;

receiving an instruction from the server via the data communication network that instructs the client computing device to instantiate end user controls appropriate for the type of FVV selected;

displaying a FVV control user interface on said display device;

monitoring for end user inputs via the FVV control user interface;

whenever an end user viewpoint navigation input is received via the FVV control user interface, transmitting the input to the server via the data communication network;

receiving FVV frames from the server via the data communication network, each FVV frame depicting at least a portion of the scene as it would be viewed from either an initial viewpoint if the end user has not yet input a viewpoint navigation input via the FVV control user interface or the last viewpoint the end user input using a viewpoint navigation input via the FVV control user interface; and

displaying each FVV frame on said display device as it is received.

18. The process of claim 17, further comprising process actions of:

whenever an end user temporal navigation input is received via the FVV control user interface, transmitting the input to the server via the data communication network, said temporal navigation input representing an instruction to provide FVV frames from a user-specified temporal location in the FVV; and

receiving a sequence of FVV frames from the server via the data communication network, the first of which corresponds to the last user-specified temporal location in the FVV.

19. The process of claim 17, further comprising the process actions of:

receiving with each FVV frame received from the server, one or more scene proxies which were used by the server to generate the received FVV frame; and

whenever a same-frame end user viewpoint navigation input is received via the FVV control user interface which represents an instruction to view a scene depicted in the last-displayed FVV frame from a different viewpoint,

generating a new FVV frame using the scene proxy or proxies received with the last-displayed frame, said new FVV frame depicting the scene depicted in the last-displayed FVV frame from a viewpoint specified in the same-frame end user viewpoint navigation input, and

displaying the new FVV frame on said display device.

20. The process of claim 17, wherein the FVV frames received from the server depict at least a portion of the scene as viewed from the current synthetic viewpoint of the scene that is larger than said display device is capable of displaying, and only a portion of each FVV frame is displayed on said display device, the process further comprising a process actions of, whenever a same-frame end user viewpoint navigation input is received via the FVV control user interface which represents an instruction to view a portion of the scene depicted in the last-received FVV frame that was not shown in the last-displayed portion of the frame, displaying at least the portion of the scene depicted in the last-received FVV frame specified in the same-frame end user viewpoint navigation input on said display device.