Predictive bitrate selection for 360 video streaming

• US 10,547,704 B2
• Filed: 04/06/2017
• Issued: 01/28/2020
• Est. Priority Date: 04/06/2017
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Beran, Erik, Tokubo, Todd
• Primary Examiner(s): Dinh, Khanh Q
• Application Number:
  US15/481,324
• Publication Number:
  US 20180295205A1
• Time in Patent Office: 1,027 Days
• Field of Search: 709220, 709224, 709228
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/5681 Pre-fetching or pre-delivering data based on network characteristics
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/6373 for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Predictive pre-fetching of streams for 360 degree video is described. User view orientation metadata is obtained for a 360 degree video stream that includes data for a plurality of viewports. Data corresponding to one or more high-resolution frames for a particular one of the viewports is pre-fetched based on the user view orientation metadata and those frames are displayed. The high resolution frames are characterized by a higher resolution than for remaining viewports.

Claims

1. A method comprising:

a) obtaining view orientation metadata for a 360 degree video stream that includes data for a plurality of viewports;

b) pre-fetching data corresponding to one or more high-resolution frames for a particular viewport of the plurality of viewports determined by the user view orientation metadata;

c) displaying the one or more high-resolution frames, wherein the one or more high resolution frames are characterized by a higher resolution than for remaining viewports of the plurality of viewports

d) checking the view orientation metadata against an actual user view orientation; and

disabling pre-fetching when the user view orientation metadata does not coincide with the actual user view orientation during display.

2. The method of claim 1 wherein the view orientation metadata is an Author-defined pre-fetch vector.

3. The method of claim 1 wherein the view orientation metadata is a pre-fetch vector created to match production effects in a scene of the 360 degree video stream.

4. The method of claim 1 wherein the view orientation metadata is a predictive probability of a viewport frame change.

5. The method of claim 4 wherein the predictive probability is generated by a focus group.

6. The method of claim 4 wherein the predictive probability is generated using user generated frame location data.

7. The method of claim 4 wherein the predictive probability also includes an Author defined pre-fetch vector.

8. The method of claim 7 wherein the Author defined pre-fetch vector is a weight applied to the probability of a viewport frame change.

9. The method of claim 4 wherein said pre-fetching frames includes applying a first threshold to the user view orientation metadata to determine the particular viewport of the plurality of viewports.

10. The method of claim 4 wherein the predictive probability of a viewport frame change is a probability for a Markov model.

11. The method of claim 1 wherein the view orientation metadata is a movement vector.

12. The method of claim 9 wherein b) includes prefetching intermediate resolution frames when the predictive probability exceeds a second threshold.

13. The method of claim 1, wherein the view orientation metadata represents a probability of where a user is likely to look.

14. The method of claim 1, wherein the view orientation metadata represents a probability of where a user is not likely to look.

15. The method of claim 13 wherein disabling pre-fetching when the view orientation metadata does not coincide with the actual user view orientation during display includes applying a missed frame threshold to determine whether to disable prefetching.

16. The method of claim 1 wherein prefetching data corresponding to one or more high-resolution frames determined by the user view orientation metadata includes pre-fetching a frame that has a higher resolution section that the rest of the frame.

17. A non-transient computer readable medium containing program instructions for causing a computer to perform the method of:

a) obtaining a user view orientation metadata for a 360 degree video stream that includes data for a plurality of viewports;

b) pre-fetching data corresponding to one or more high-resolution frames for a particular viewport of the plurality of viewports determined by the user view orientation metadata;

c) displaying the one or more high-resolution frames, wherein the one or more high resolution frames are characterized by a higher resolution than for remaining viewports of the plurality of viewports

d) checking the view orientation metadata against an actual user view orientation; and

disabling pre-fetching when the user view orientation metadata does not coincide with the actual user view orientation during display.

18. A system, comprising:

a processor;

a display coupled to the processor;

a memory coupled to the processor having processor-executable instructions embodied therein, the instructions being configured to implement a method upon execution by the processor, the method comprising;

a) obtaining a view orientation metadata for a 360 degree video stream that includes data for a plurality of viewports;

b) pre-fetching data corresponding to one or more high-resolution frames for a particular viewport of the plurality of viewports determined by the user view orientation metadata;

c) displaying the one or more high-resolution frames with the display, wherein the one or more high resolution frames are characterized by a higher resolution than for remaining viewports of the plurality of viewports

d) checking the view orientation metadata against an actual user view orientation; and

disabling pre-fetching when the user view orientation metadata does not coincide with the actual user view orientation during display.

19. The system of claim 18 wherein the display is a 360 degree display configured to display the plurality of viewports simultaneously.