SYSTEM AND METHOD FOR PREEMPTIVE AND ADAPTIVE 360 DEGREE IMMERSIVE VIDEO STREAMING

• US 20160198140A1
• Filed: 01/06/2015
• Published: 07/07/2016
• Est. Priority Date: 01/06/2015
• Status: Abandoned Application

• Current Assignee: 3doo, Inc.
• Original Assignee: 3doo, Inc.
• Inventors: Nadler, Ingo
• Application Number:
  US14/590,267
• Publication Number:
  US 20160198140A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

A method for delivering streaming 3D video to an electronic device is presented, the method including storing scene files including unwrapped hemispherical representations of scenes for left and right eye perspective views in first and second video files, respectively. The method includes transmitting the scene files of the left and right eye perspective views to the electronic having head tracking capabilities, 3D video streaming capabilities, and 3D viewing capabilities into the electronic device. The method also includes allowing the electronic device to request from the one or more servers the left and right eye perspective views including the scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, extracting and re-coding the requested left and right eye perspective views including the scene files, and enabling the electronic device to stream real-time 3D video and allowing 360 degree freedom of eye movement for the user.

Claims

1. A method for delivering streaming 3D video to an electronic device, the method comprising:

storing first scene files including unwrapped hemispherical representations of scenes for a left eye perspective view in a first video file located in one or more servers;

storing second scene files including unwrapped hemispherical representations of scenes for a right eye perspective view in a second video file located in the one or more servers;

transmitting the first and second scene files of the left and right eye perspective views, respectively, to the electronic device from the one or more servers, the electronic device having head tracking capabilities, 3D video streaming capabilities, and 3D viewing capabilities;

generating, via the electronic device, left and right eye perspective views of a user;

detecting, via the electronic device, a head position and a head movement of the user;

allowing the electronic device to request from the one or more servers the left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively;

extracting and re-encoding the requested left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively; and

enabling the electronic device to stream real-time 3D video with 360 degree freedom of eye motion for the user by switching between bandwidths based on the extracted and re-encoded left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes.

2. The method of claim 1, wherein the electronic device includes display hardware.

3. The method of claim 1, wherein the electronic device is a wearable electronic device.

4. The method of claim 1, wherein the electronic device is a gaming console.

5. The method of claim 1, wherein the electronic device is a mobile device.

6. The method of claim 1, wherein the electronic device is a 3D television.

7. The method of claim 1, wherein the electronic device includes a client application for predicting the eye motion of the user of the electronic device by calculating a probability graph.

8. The method of claim 7, wherein the probability graph is calculated by:

generating a first vector for each frame of the first and second video files of the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively;

selecting two consecutive frames and generating a second vector therefrom including a direction of motion of the eyes of the user;

storing the second vector in a time-coded file for each frame of the two consecutive frames; and

transmitting motion vector data to the client application of the electronic device of the user.

9. The method of claim 8, wherein if a disparity map of the first and second video files is available, a change in disparity between the two consecutive frames is included in calculating the probability graph.

10. The method of claim 8, wherein the motion vector data is used for the switching between the bandwidths to enable the 360 degree freedom of the eye motion for the user.

11. A method for delivering streaming 3D video to an electronic device, the method comprising:

storing first scene files including unwrapped hemispherical representations of scenes for a left eye perspective view in a first video file located in one or more servers;

storing second scene files including unwrapped hemispherical representations of scenes for a right eye perspective view in a second video file located in the one or more servers;

transmitting the first and second scene files of the left and right eye perspective views, respectively, to the electronic device from the one or more servers, the electronic device having head tracking capabilities, 3D video streaming capabilities, and 3D viewing capabilities;

generating, via the electronic device, left and right eye perspective views of a user;

detecting, via the electronic device, a head position and a head movement of the user;

allowing the electronic device to request from the one or more servers the left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively;

extracting the requested left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively;

merging the extracted left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively, into a stereoscopic side-by-side format;

re-encoding the merged left and right eye perspective views; and

enabling the electronic device to stream real-time 3D video with 360 degree freedom of eye motion for the user by switching between bandwidths based on the extracted and re-encoded left and right eye perspective views including the first and second scene files having the unwrapped hemispherical representations of scenes.

12. The method of claim 11, wherein the electronic device includes display hardware.

13. The method of claim 11, wherein the electronic device is one of a wearable electronic device, a gaming console, a mobile device, and a 3D television.

14. The method of claim 11, wherein the electronic device includes a client application for predicting the eye motion of the user of the electronic device by calculating a probability graph.

15. The method of claim 14, wherein the probability graph is calculated by:

generating a first vector for each frame of the first and second video files of the unwrapped hemispherical representations of scenes for the left and right eye perspective views, respectively;

selecting two consecutive frames and generating a second vector therefrom including a direction of motion of the eyes of the user;

storing the second vector in a time-coded file for each frame of the two consecutive frames; and

transmitting motion vector data to the client application of the electronic device of the user.

16. The method of claim 15, wherein if a disparity map of the first and second video files is available, a change in disparity between the two consecutive frames is included in calculating the probability graph.

17. A system for delivering streaming 3D video, the system comprising:

one or more servers for storing scene files including unwrapped hemispherical representations of scenes for a left eye perspective view and a right eye perspective view;

a network connected to the one or more servers;

an electronic device in communication with the network, the electronic device having head tracking capabilities, 3D video streaming capabilities, and 3D viewing capabilities, the electronic device configured to request from the one or more servers the left and right eye perspective views including the scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views;

a calculating module for calculating a probability graph for predicting eye motion of a user of the electronic device;

an extracting module and a re-encoding module for extracting and re-encoding the requested left and right eye perspective views including the scene files having the unwrapped hemispherical representations of scenes for the left and right eye perspective views;

wherein the electronic device streams real-time 3D video with 360 degree freedom of eye motion for the user by switching between bandwidths based on the probability graph calculated.

18. The system of claim 17, wherein the electronic device is one of a wearable electronic device, a gaming console, a mobile device, and a 3D television.

19. The system of claim 17, wherein the probability graph is calculated by:

generating a first vector for each frame of the scene files of the unwrapped hemispherical representations of scenes for the left and right eye perspective views;

selecting two consecutive frames and generating a second vector therefrom including a direction of motion of the eyes of the user;

storing the second vector in a time-coded file for each frame of the two consecutive frames; and

transmitting motion vector data to a client application of the electronic device of the user.

20. The system of claim 19,

wherein the unwrapped hemispherical representations of scenes for the left and right eye perspective views are stored in first and second video files, respectively;

and

wherein, if a disparity map of the first and second video files is available, a change in disparity between the two consecutive frames is included in calculating the probability graph.