System and method for improving the graphics performance of hosted applications

• US 8,851,999 B2
• Filed: 09/14/2011
• Issued: 10/07/2014
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Dietrich, Douglas Sim Jr., Benitez, Nico, Cotter, Timothy
• Primary Examiner(s): D'Agostino, Paul A
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US13/232,985
• Publication Number:
  US 20120115601A1
• Time in Patent Office: 1,119 Days
• Field of Search: 463/40.42, 345/506
• US Class Current: 463/42
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/556 Player lists, e.g. online players, buddy list, black list
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 : Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23406 : involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8458 : involving uncompressed content

Abstract

A system for hosting video games comprising: a game server comprising a central processing unit to process video game program code and a graphics processing unit (GPU) to process graphics commands; one or more back buffers to store video frames in response to the execution of the graphics commands; a front buffer to receive a video frame for rendering on a display and outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a vertical blanking interval (VBI); a frame processing module to increase the number of scan lines associated with the VBI to increase the likelihood that a new frame will be completed in a back buffer and ready for transfer to the front buffer at a time during the VBI.

Claims

1. A server-based system for hosting applications/video games for a plurality of clients comprising:

an application/game server comprising a central processing unit to process application/video game program code and a graphics processing unit (GPU) to process graphics commands and generate a series of video frames for the application/video game;

one or more back buffers to store video frames as the video frames are being created in response to the execution of the graphics commands;

a front buffer to receive a video frame for rendering on a display after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a specified time interval;

a frame processing module to perform real-time profiling of the GPU by performing the operations of;

tracking how long it takes the GPU to execute certain types of operations while in a current mode of operation; and

updating a benchmark data structure with values representing an amount of time needed for each specified type of operation.

2. The system as in claim 1 wherein the certain types of operations comprises rendering of triangles.

3. The system as in claim 1 wherein the certain types of operations comprises rendering of polygons.

4. The system as in claim 1 wherein the frame processing module inserts timestamp fences around work being inserted into a GPU command queue, determines the amount of time consumed performing the work based on differences between the timestamps, and continually updates the values to include an average per-operation time.

5. The system as in claim 1 wherein the current mode of operation comprises depth-only rendering;

lighting;

alpha blending;

or image processing.

6. The system as in claim 1 wherein the frame processing module subsequently queries the benchmark data structure to determine the amount of GPU work remaining in a GPU command queue, and/or the number of busy work operations needed to reach a specified scan out region.

7. The system as in claim 6 wherein the specified scan out region comprises a vertical blanking interval (VBI) or a region outside of a designated region of the front buffer.

8. The system as in claim 6 wherein the frame processing module inserts a number of busy work operations in the GPU command queue needed to reach the specified scan out region.

9. The system as in claim 1 wherein the benchmark data structure is indexed, at least in part, by different values associated with different GPU modes of operation.

10. The system as in claim 9 further comprising:

determining a current mode in which the GPU is operating and using a value associated with the current mode to index the benchmark data structure.

11. The system as in claim 10 wherein the value comprises a hash value.

12. The system as in claim 11 wherein the specified time interval comprises a vertical blanking interval (VBI).

13. The system as in claim 12 wherein the benchmark data structure comprises a database.

14. A method for hosting applications/video games for a plurality of clients comprising:

processing application/video game program code by a central processor of an application/game server and processing graphics commands by a graphics unit (GPU) to generate a series of video frames for the application/video game;

storing video frames in one or more back buffers as the video frames are being created in response to the execution of the graphics commands;

receiving a video frame for rendering on a display in a front buffer after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a specified time interval;

performing real-time profiling of the GPU by performing the operations of;

tracking how long it takes the GPU to execute certain types of operations while in a current mode of operation; and

updating a benchmark data structure with values representing an amount of time needed for each specified type of operation.

15. The method as in claim 14 wherein the certain types of operations comprises rendering of triangles.

16. The method as in claim 14 wherein the certain types of operations comprises rendering of polygons.

17. The method as in claim 14 further comprising:

inserting timestamp fences around work being inserted into a GPU command queue, determining the amount of time consumed performing the work based on differences between the timestamps, and continually updating the values to include an average per-operation time.

18. The method as in claim 14 wherein the current mode of operation comprises depth-only rendering;

lighting;

alpha blending;

or image processing.

19. The method as in claim 14 further comprising:

subsequently querying the benchmark data structure to determine the amount of GPU work remaining in a GPU command queue, and/or the number of busy work operations needed to reach a specified scan out region.

20. The method as in claim 19 wherein the specified scan out region comprises a vertical blanking interval (VBI) or a region outside of a designated center region of the front buffer.

21. The method as in claim 19 further comprising:

inserting a number of busy work operations in the GPU command queue needed to reach the specified scan out region.

22. The method as in claim 14 wherein the benchmark data structure is indexed, at least in part, by different values associated with different GPU modes of operation.

23. The method as in claim 22 further comprising:

determining a current mode in which the GPU is operating and using a value associated with the current mode to index the benchmark data structure.

24. The method as in claim 23 wherein the value comprises a hash value.

25. The method as in claim 24 wherein the specified time interval comprises a vertical blanking interval (VBI).

26. The method as in claim 25 wherein the benchmark data structure comprises a database.