Method and apparatus for improving quality of experience and bandwidth in virtual reality streaming systems

• US 10,460,700 B1
• Filed: 02/06/2017
• Issued: 10/29/2019
• Est. Priority Date: 10/12/2015
• Status: Active Grant

• Current Assignee: Cinova Media
• Original Assignee: Cinova Media, Inc.
• Inventors: Mendhekar, Anurag, Seigneurbieux, Pierre, Sarkar, Subhrendu, Guardini, Domenico
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Robinson, Terrell M
• Application Number:
  US15/425,613
• Time in Patent Office: 995 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 3/4092 Image resolution transcoding, e.g. by using client-server architectures
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04L 27/2607 Cyclic extensions
• CPC Class Codes: H04L 41/0823 characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for optimising operational conditions of wireless networks H04W24/02
• CPC Class Codes: H04L 41/5067 Customer-centric QoS measurements
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

An apparatus and method for improving the quality of user experience and reducing bandwidth consumption in Virtual Reality systems that optimize bandwidth by restricting the transmitted stream to only what is in the field of view of a virtual reality headset are disclosed. The actual content streamed may vary from interactive games to pre-recorded or live video.

Claims

1. A virtual reality data apparatus, comprising:

a backend system coupled to each virtual reality device over a communication path that streams virtual reality data to each virtual reality device, each virtual reality device receiving a virtual reality data stream that has an in view portion of a frame of virtual reality data and one or more pieces of optimized virtual reality data, the one or more pieces of optimized virtual reality data that is an adaptive guard band that surrounds the in view portion of the frame, the adaptive guard band having a width that is not uniform around each side of the in view portion of the frame for reducing an amount of virtual reality data being streamed to each virtual reality device while addressing a head motion artifact latency issue and a scene change issue of streaming virtual reality data;

the backend system having an adaptive guard band component for receiving data about each virtual reality device, determining a motion of each virtual reality device and generating one or more pieces of optimized virtual reality data for each virtual reality device in response to the data about each virtual reality device and the detected motion of the virtual reality device; and

wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wider width in a diagonal direction if the determined motion of the particular virtual reality device is diagonal.

2. The apparatus of claim 1, wherein the backend system is further configured to determine a latency of the communication path between the backend system and each virtual reality device and wherein the adaptive guard band component is configured to generate the adaptive guard band for each virtual reality device in response to the detected latency of the communication path between the backend system and each virtual reality device.

3. The apparatus of claim 2, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a narrow width if the detected latency for the communication path between the particular virtual reality device and the backend system is low.

4. The apparatus of claim 2, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wide width if the detected latency for the communication path between the particular virtual reality device and the backend system is high.

5. The apparatus of claim 1, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wider width in a horizontal direction if the determined motion of the particular virtual reality device is horizontal.

6. The apparatus of claim 1, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wider width in a vertical direction if the determined motion of the particular virtual reality device is vertical.

7. The apparatus of claim 1, wherein the backend system is further configured to determine a motion of each virtual reality device and determine a latency of the communication path between the backend system and each virtual reality device and wherein the adaptive guard band component is configured to generate the adaptive guard band for each virtual reality device in response to the detected motion of the virtual reality device and the detected latency of the communication path between the backend system and each virtual reality device.

8. The apparatus of claim 1, wherein the backend system further comprises a scene change component that is configured to adjust a set of encoding parameters when a scene change is detected.

9. The apparatus of claim 8, wherein the scene change component is configured to determine an inter-view distance for a particular virtual reality device and detect the scene change when the inter-view distance for the particular virtual reality device exceeds a threshold.

10. A method, comprising:

receiving data about a plurality of virtual reality devices;

generating one or more pieces of optimized virtual reality data for each virtual reality device in response to the data about each virtual reality device, wherein the one or more pieces of optimized data is an adaptive guard band that surrounds the in view portion of the frame, the adaptive guard band having a width that is not uniform around each side of the in view portion of the frame;

determining a motion of each virtual reality device;

wherein generating the adaptive guard band further comprises generating the adaptive guard band for the virtual reality device in response to the detected motion of the virtual reality device and generating the adaptive guard band for a particular virtual reality device having a wider width in a diagonal direction if the determined motion of the particular virtual reality device is diagonal; and

generating a virtual reality stream for the virtual reality device, the virtual reality stream having an in view portion of a frame of virtual reality data and the one or more pieces of optimized virtual reality data, the one or more pieces of optimized virtual reality data reducing an amount of virtual reality data being streamed to each virtual reality device while addressing a head motion artifact latency issue and a scene change issue of streaming virtual reality data.

11. The method of claim 10 further comprising determining a latency of the communication path between the backend system and each virtual reality device and wherein generating the adaptive guard band further comprises generating the adaptive guard band for the virtual reality device in response to the detected latency of the communication path between the backend system and each virtual reality device.

12. The method of claim 11, wherein generating the adaptive guard band further comprises generating the adaptive guard band for a particular virtual reality device having a narrow width if the detected latency for the communication path between the particular virtual reality device and the backend system is low.

13. The method of claim 11, wherein generating the adaptive guard band further comprises generating the adaptive guard band for a particular virtual reality device having a wide width if the detected latency for the communication path between the particular virtual reality device and the backend system is high.

14. The method of claim 10, wherein generating the adaptive guard band further comprises generating the adaptive guard band for a particular virtual reality device having a wider width in a horizontal direction if the determined motion of the particular virtual reality device is horizontal.

15. The method of claim 10, wherein generating the adaptive guard band further comprises generating the adaptive guard band for a particular virtual reality device having a wider width in a vertical direction if the determined motion of the particular virtual reality device is vertical.

16. The method of claim 10 further comprising determining a motion of each virtual reality device and determining latency of the communication path between the backend system and each virtual reality device and wherein generating the adaptive guard band further comprises generating the adaptive guard band for the virtual reality device in response to the detected motion of the virtual reality device and the detected latency of the communication path between the backend system and each virtual reality device.

17. The method of claim 10 further comprises adjusting a set of encoding parameters when a scene change is detected, encoding the in view portion of the virtual reality data using the adjusted set of if encoding parameters and communicating the encoded in view portion of the virtual reality data to the virtual reality device.

18. The method of claim 17 further comprising determining an inter-view distance for a particular virtual reality device and detecting the scene change when the inter-view distance for the particular virtual reality device exceeds a threshold.

19. A virtual reality system, comprising:

one or more virtual reality devices wherein each virtual reality device is capable of receiving streaming virtual reality data and is capable to displaying the virtual reality data to a user who is wearing the virtual reality device;

a backend system coupled to each virtual reality device over a communication path that streams virtual reality data to each virtual reality device, each virtual reality device receiving a virtual reality data stream that has an in view portion of a frame of virtual reality data and one or more pieces of optimized virtual reality data, the one or more pieces of optimized virtual reality data being an adaptive guard band that surrounds the in view portion of the frame, the adaptive guard band having a width that is not uniform around each side of the in view portion of the frame for reducing an amount of virtual reality data being streamed to each virtual reality device while addressing a head motion artifact latency issue and a scene change issue of streaming virtual reality data;

the backend system receiving data about each virtual reality device, determining a motion of each virtual reality device and generating one or more pieces of optimized virtual reality data for each virtual reality device in response to the data about each virtual reality device and the detected motion of the virtual reality device; and

wherein the backend system generates an adaptive guard band for a particular virtual reality device having a wider width in a diagonal direction if the determined motion of the particular virtual reality device is diagonal.

20. The system of claim 19, wherein the backend system is further configured to determine a latency of the communication path between the backend system and each virtual reality device and wherein the adaptive guard band component is configured to generate the adaptive guard band for each virtual reality device in response to the detected latency of the communication path between the backend system and each virtual reality device.

21. The system of claim 20, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a narrow width if the detected latency for the communication path between the particular virtual reality device and the backend system is low.

22. The system of claim 20, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wide width if the detected latency for the communication path between the particular virtual reality device and the backend system is high.

23. The system of claim 19, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wider width in a horizontal direction if the determined motion of the particular virtual reality device is horizontal.

24. The system of claim 19, wherein the adaptive guard band component is configured to generate an adaptive guard band for a particular virtual reality device having a wider width in a vertical direction if the determined motion of the particular virtual reality device is vertical.

25. The system of claim 19, wherein the backend system is further configured to determine a motion of each virtual reality device and determine a latency of the communication path between the backend system and each virtual reality device and wherein the adaptive guard band component is configured to generate the adaptive guard band for each virtual reality device in response to the detected motion of the virtual reality device and the detected latency of the communication path between the backend system and each virtual reality device.

26. The system of claim 19, wherein the backend system further comprises a scene change component that is configured to adjust a set of encoding parameters when a scene change is detected.

27. The system of claim 26, wherein the scene change component is configured to determine an inter-view distance for a particular virtual reality device and detect the scene change when the inter-view distance for the particular virtual reality device exceeds a threshold.