RENDERING AND ENCODING ADAPTATION TO ADDRESS COMPUTATION AND NETWORK

• US 20130307847A1
• Filed: 12/06/2011
• Published: 11/21/2013
• Est. Priority Date: 12/06/2010
• Status: Abandoned Application

• Current Assignee: Regents of the University of California (University of California)
• Original Assignee: Regents of the University of California (University of California)
• Inventors: Dey, Sujit, Wang, Shaoxuan
• Application Number:
  US13/988,971
• Publication Number:
  US 20130307847A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 2210/08 Bandwidth reduction
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: H04N 19/102 characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/156 Availability of hardware or computational resources, e.g. encoding based on power-saving criteria
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field

Abstract

A method for graphics rendering adaptation by a server that includes a graphics rendering engine that generates graphic video data and provides the video data via a communication resource to a client. The method includes monitoring one or both of communication and computation constraint conditions associated with the graphics rendering engine and the communication resource. At least one rendering parameter used by the graphics rendering engine is set based upon a level of communication constraint or computation constraint. Monitoring and setting are repeated to adapt rendering based upon changes in one or both of communication and computation constraints. In preferred embodiments, encoding adaptation also responds to bit rate constraints and rendering is optimized based upon a given bit rate. Rendering parameters and their effect on communication and computation costs have been determined and optimized. A preferred application is for a gaming processor miming on a cloud based or data center server that services mobile clients over a wireless network for graphics intensive applications, such as massively multi-player online role playing games, or augmented reality.

Claims

1. A method for graphics rendering adaptation by a server that includes a graphics rendering engine that generates graphic video data and provides the video data for encoding and communication via a communication resource to a client, the method comprising:

monitoring one or both of communication and computation constraint conditions associated with the graphics rendering engine and the communication resource;

setting at least one rendering parameter used by the graphics rendering engine based upon a level of communication constraint or computation constraint; and

repeating said monitoring and setting to adapt rendering based upon changes in one or both of communication and computation constraints.

2. The method of claim 1, wherein said monitoring monitors communication delay on the communication resource as a communication constraint.

3. The method of claim 2, wherein said monitoring monitors packet loss on the communication resource as a communication constraint condition.

4. The method of claim 1, wherein said monitoring monitors one or more of server utilization, server cost, cloud cost, and data center cost as a computation constraint.

5. The method of claim 1, further comprising setting encoding bit rate based upon a level of communication constraint.

6. The method of claim 5, wherein said setting of at least one rendering parameter is conducted after said setting encoding bit rate.

7. The method of claim 1, wherein:

said at least one rendering parameter includes a plurality of parameters;

said plurality of parameters include parameters that effect a cost of communication and parameters that effect a cost of computation; and

said setting adjusts a rendering parameter that affects a cost of communication in response to a monitored communication constraint and adjusts a rendering parameter that affects a cost of computation in response to a monitored computation constraint.

8. The method of claim 1, wherein the at least one rendering parameter includes one or more of realistic effect parameters, texture detail parameters, view distance parameters, and rendering frame rate, rendering resolution, and enabling background detail parameters.

9. The method of claim 8, wherein said setting selects among one or more of the realistic effect parameters, texture detail parameters, view distance parameters, and enabling background detail parameters based upon a calculated or determined in advance parameter-user experience impairment effect.

10. The method of claim 9, wherein the realistic effect parameters include one or more of a color depth parameter, multi-sample parameter, texture-filter parameter, and lighting mode parameter.

11. The method of claim 1, wherein said setting of at least one rendering parameter also sets a rendering level that is optimal in view of an encoding bitrate.

12. The method of claim 1, wherein the graphics rendering engine comprises a game engine, the server comprises a cloud server or data center server, and the communication resource comprises a mobile wireless network.

13. The method of claim 1, further comprising

a preliminary procedure to identify or calculate optimal rendering settings for different rendering adaptation levels, where each adaptation level represents a certain communication and computation cost;

and

said setting conducts a run-time level-selection method that automatically adapts the adaptation levels, and thereby the rendering settings, such that the rendering cost will satisfy the communication and computation constraints imposed by fluctuating network bandwidth and server available capacity according to the optimal rendering settings.

14. The method according to claim 13, wherein said setting comprises:

deciding encoding bitrate used to encode the rendered video on the communication resource by, if network round trip delay RDelay or its average is over a predetermined delay or if the network round trip delay RDelay is increasing over a time period, reduce encoding bitrate, otherwise if for a predetermined time, RDelay or its average remains below the predetermined delay and there is no packet loss, then increase the encoding bitrate;

checking and updating the at least one rendering parameter, by selecting and updating a parameter that will affect communication cost, decreasing a level of the parameter that will affect communication cost if the encoding bit rate has decreased, otherwise, if the encoding bit rate has increased and remains increased for a predetermined time, increasing the level of the parameter that will affect communication cost; and

checking and updating the at least one rendering parameter, by selecting and updating a parameter that will affect computation cost depending upon a level of server utilization, decreasing a level of the parameter if server utilization is over a predetermined amount, otherwise if server utilization is below a predetermined amount for a predetermined time, increasing a level of the parameter.

15. A server that adaptively provides graphics to clients over a network, the server comprising:

a video rendering pipeline including a graphic engine and a content engine for rendering video;

an encoding pipeline for encoding video rendered by said video rendering pipeline;

communication monitoring means for monitoring communication network conditions;

computation cost monitoring means for monitoring computation costs; and

software for instructing the video rendering pipeline to adjust rendering parameters in response to communication network conditions monitored by said communication monitoring means and computation cost conditions monitored by said computation cost monitoring means, and for instructing the encoding pipeline to adjust encoding parameters in response to communication network conditions monitored by said communication monitoring means.