STREAMING VIRTUAL REALITY VIDEO

• US 20190310472A1
• Filed: 11/03/2017
• Published: 10/10/2019
• Est. Priority Date: 11/04/2016
• Status: Active Grant

• Current Assignee: Koninklijke KPN NV, Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno
• Original Assignee: Koninklijke KPN NV
• Inventors: Schilt, Gerrit, Stokking, Hans Maarten, Brouwer, Anne-Marie, Van Der Waa, Jasper Sebastiaan, Hogervorst, Maarten Andreas
• Granted Patent: US 10,712,555 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/2665 Gathering content from different sources, e.g. Internet and satellite
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Methods and devices are provided for use in a streaming process (130) of a Virtual Reality [VR] video to a VR rendering device (120). The streaming process has a configurable setup, each configured setup providing respective different image data of a scene. The VR rendering device is arranged to render a current view of the scene based on a current head position. The streaming process is executed according to a first configured setup providing first image data needed to render the current view. EEG data is determined by measuring brain signals (115) and a head movement is predicted based on the EEG data. Target image data is determined as needed for rendering a future view based on the predicted head movement. A new configured setup for providing the target image data is determined. By adapting the streaming process based on the prediction the target image data can be provided in time and efficiently using available bandwidth.

Claims

1. A method for use in a streaming process of a Virtual Reality [VR] video to a VR rendering device, the streaming process having a configurable setup, in each configured setup the VR video being represented by one or more streams and each configured setup providing respective different image data of a scene, the VR rendering device being arranged to render, on the basis of the one or more streams, a current view of the scene based on a current head position, wherein the method comprises:

executing the streaming process according to a first configured setup providing first image data needed to render the current view;

determining EEG data by measuring brain signals;

predicting a head movement based on the EEG data;

determining target image data differing from the first image data and needed for rendering a future view based on the predicted head movement;

determining a second configured setup for providing the target image data; and

executing the streaming process according to the second configured setup.

2. The method according to claim 1, wherein predicting a head movement comprises predicting a movement direction based on the EEG data, and

determining the target image data is based on the predicted movement direction.

3. The method according to claim 1, the predicting a head movement further comprising at least one of:

predicting a head movement due to a change of a focus of attention position based on the EEG data;

predicting a yaw movement rotating the head around a vertical axis based on the EEG data;

predicting a roll movement rotating the head around a view axis in a view direction based on the EEG data;

predicting a pitch movement rotating the head around a lateral axis perpendicular to the view direction based on the EEG data;

predicting a head translation based on the EEG data;

predicting a head movement speed based on the EEG data;

predicting a head movement acceleration based on the EEG data;

predicting a head movement angle based on the EEG data.

4. The method according to claim 1, wherein

predicting a head movement comprises

predicting a movement onset based on the EEG data, and determining the second configured setup in dependence of the predicted movement onset;

or

predicting a movement ending based on the EEG data and

determining the second configured setup in dependence of the predicted movement ending.

5. The method according to claim 1, wherein the target image data comprises a target area where the future view is expected and a guard area, and wherein determining the target image data comprises determining the guard area which is spatially adjacent to the target area.

6. The method according to claim 5, wherein determining the guard area further comprises at least one of:

determining the guard area for accommodating a difference between the predicted head movement and an actual future head movement;

determining the guard area in dependence of an estimated accuracy of the predicted head movement;

adjusting the size of the guard area in dependence of the estimated accuracy of the predicted head movement, and reducing the size at increasing accuracies;

adjusting the quality of the guard area in dependence of the estimated accuracy of the predicted head movement, and increasing the quality at increasing accuracies;

determining the guard area spatially adjacent in all directions around the image data of the first configured setup based on a predicted movement onset while the movement direction is not yet predicted;

determining the guard area spatially adjacent in a subset of directions to the image data of the first configured setup corresponding to a predicted movement direction;

determining the guard area spatially adjacent in a left or right movement direction to the image data of the first configured setup when a horizontal head rotation and a corresponding movement direction are predicted;

determining the guard area spatially adjacent in a direction above or below of the image data of the first configured setup when a vertical head rotation and a corresponding movement direction are predicted.

7. The method according to claim 5, the determining the second configured setup further comprising at least one of:

having the second configured setup provide the image data for the guard area at a lower quality than the quality of the image data for the area corresponding to the current view;

having the second configured setup provide the image data for at least part of the target area at a lower quality than the image data of the current view according to the first configured setup when a head movement is predicted;

having the second configured setup provide at least part of the target area at a higher quality than the image data of the current view according to the first configured setup when a deceleration of head movement is predicted for the second configured setup.

8. The method according to claim 1, wherein the determining a second configured setup further comprises determining prediction data indicating at least one of

the predicted head movement;

a predicted head movement direction;

a predicted head movement speed;

a predicted head movement acceleration;

a predicted head movement angle;

a predicted head movement onset;

a predicted head movement ending;

the target image data;

the future view; and

the streaming process comprises obtaining the VR video from or via a VR server and processing the VR video at the VR server to provide the one or more streams and executing the second configured setup comprises transferring the prediction data to the VR server for enabling the VR server to adapt processing of the VR video to provide the one or more streams according to the prediction data;

or

the streaming process comprises obtaining the VR video from a camera oriented towards the scene at a VR source, and the determining a second configured setup further comprises determining at least one command based on the prediction data, and executing the second configured setup further comprises transferring the command to the VR source to enable controlling the camera.

9. The method according to claim 1,

wherein the streaming process comprises obtaining the VR video from multiple locations, respective image data from the respective locations together constituting the scene and each location providing at least one respective stream, and wherein

the determining the second configured setup further comprises including in the second configured setup a respective stream when the target image data comprises at least part of the respective image data;

or

the determining of a further configured setup comprises excluding from the further configured setup a respective stream when the target image data no longer comprises the respective image data.

10. The method according to claim 1,

wherein the streaming process comprises, in a video delivery chain, at least one of the sub-processes:

capturing image data,

stitching together image data from multiple sources,

encoding image data,

encoding different versions of the same content for different viewports, while in a respective version encoding image data for the respective viewport in high quality and encoding image data for other parts in lower quality,

packetizing image data into a stream,

transporting a stream via a network,

de-packetizing image data from a stream,

buffering image data, decoding image data,

displaying image data; and

executing the second configured setup further comprises controlling at least one of the sub-processes based on the predicted head movement.

11. The method according to claim 1, the predicting a head movement further comprising using a classification model to distinguish predictions of different head movements based on the EEG data;

the method further comprising at least one of

training the classification model using the EEG data in sample periods of a predetermined length preceding a predetermined head movement;

training the classification model based on head movement data derived from successive actual head orientations and corresponding predicted head movements;

providing from the model multiple graded outputs for multiple possible head movements and using the multiple graded outputs for determining the target image data.

12. The method according to claim 1, wherein determining the second configured setup comprises

obtaining spatial relation data which is indicative of a spatial relation between respective different image data of the scene as provided by respective streams;

and

identifying, by using the spatial relation data, the one or more streams which are needed to provide the target image data.

13. A transitory or non-transitory computer-readable medium comprising a computer program, the computer program comprising instructions for causing a processor system to perform the method according to claim 1.

14. A Virtual Reality [VR] rendering device for rendering a VR video using a streaming process having a configurable setup, in each configured setup the VR video being represented by one or more streams and each configured setup providing respective different image data of a scene,

wherein the VR rendering device comprises:

a receiver to receive the one or more streams according to a current configured setup; and

a display processor configured to render, on the basis of the one or more streams, a current view of the scene based on a current head position; and

a controller configured to;

execute the streaming process according to a first configured setup providing first image data needed to render the current view;

determine EEG data by measuring brain signals;

predict a head movement based on the EEG data;

determine target image data differing from the first image data and needed for rendering a future view based on the predicted head movement;

determine a second configured setup for providing the target image data; and

execute the streaming process according to the second configured setup.

15. The VR rendering device as claimed in claim 14, wherein the streaming process comprises processing, in the VR rendering device, at least one of the sub-processes:

retrieving the one or more streams from a local file in or near the VR rendering device,

stitching together image data from multiple streams,

de-packetizing image data from a stream,

buffering image data, decoding image data,

displaying image data; and

executing the second configured setup further comprises controlling at least one of the sub-processes based on the predicted head movement.

16. VR data structure for transferring VR data between the VR rendering device as claimed in claim 14 and a VR server, the VR data structure comprising prediction data indicating at least one of

the predicted head movement;

a predicted head movement direction;

a predicted head movement speed;

a predicted head movement acceleration;

a predicted head movement angle;

a predicted head movement onset;

a predicted head movement ending;

the target image data;

the second configured setup;

the future view;

which prediction data is based on a predicted head movement based on the EEG data.

17. VR server for providing or forwarding VR video to a VR rendering device as claimed in claim 14, wherein the VR server is arranged

to process the VR video to provide the one or more streams;

to receive, from the VR rendering device, a prediction data structure comprising prediction data indicating at least one of

the predicted head movement

a predicted head movement direction;

a predicted head movement speed;

a predicted head movement acceleration;

a predicted head movement angle;

a predicted head movement onset

a predicted head movement ending;

the target image data;

the second configured setup;

the future view;

which prediction data is based on a predicted head movement based on the EEG data; and

to adapt said processing of the VR video according to the prediction data.