System and method for improving the graphics performance of hosted applications

• US 8,961,316 B2
• Filed: 09/14/2011
• Issued: 02/24/2015
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Dietrich, Douglas Sim Jr., Benitez, Nico, Cotter, Timothy
• Primary Examiner(s): D'Agostino, Paul A
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US13/232,979
• Publication Number:
  US 20120115600A1
• Time in Patent Office: 1,259 Days
• Field of Search: 715/716, 345/213, 345/422, 345/519
• US Class Current: 463/42
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/556 Player lists, e.g. online players, buddy list, black list
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23406 : involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8458 : involving uncompressed content

Abstract

A game server comprising a central processing unit to process video game program code and a graphics processing unit (GPU) to process graphics commands; back buffers to store video frames in response to the execution of the graphics commands; a front buffer to receive a video frame for rendering on a display after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a VBI; a frame processing module to increase the number of scan lines to increase the likelihood that a new frame will be completed in a back buffer and ready for transfer to the front buffer at a time during the VBI.

Claims

1. A server-based system for hosting applications/video games for a plurality of clients comprising:

an application/game server comprising a central processing unit to process application/video game program code and a graphics processing unit (GPU) to process graphics commands and generate a series of video frames for the application/video game;

one or more back buffers to store video frames as the video frames are being created in response to the execution of the graphics commands;

a front buffer to receive a video frame for rendering on a display after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with an interval; and

a frame processing module to increase the number of scan lines associated with the interval relative to the number of scan lines not associated with the interval to increase the likelihood that a new frame will be completed in a back buffer and ready for transfer to the front buffer at a time when the subset of scan lines stored in the front buffer being associated with the interval is processed.

2. The system as in claim 1 wherein increasing the number of scan lines associated with the interval relative to the number of scan lines not associated with the interval comprises configuring the front buffer to have a larger size than the size of the back buffer.

3. The system as in claim 2 wherein the front buffer is configured to be two times the size of the back buffer.

4. The system as in claim 1 wherein in response to detecting a draw operation which will cause the new frame in the back buffer to be transferred to the front buffer, the frame processing module determines an amount of work remaining in a GPU command queue and, if the amount of work remaining in the command queue would cause the newly completed frame from the back buffer to be copied to the front buffer while a current scan line being output from the front buffer is outside of the interval, then, then inserting the draw operation in the command queue.

5. The system as in claim 4 wherein, in response to detecting a draw operation which will cause the new frame in the back buffer to be transferred to the front buffer, the frame processing module determines whether the amount of work remaining in the command queue would cause the newly completed frame from the back buffer to be copied to the front buffer while a current scan line being output from the front buffer is outside of the interval, then the frame processing module submits busy work to the command queue to ensure that the newly completed frame from the back buffer will be copied to the front buffer while a current scan line being output from the front buffer is within the interval.

6. The system as in claim 1 wherein a first thread creates new frames within the one or more back buffers and a second thread determines a most recent frame output from the front buffer and executes a draw operation to cause the new frame in the back buffer to be transferred to the front buffer after the current work remaining in the command queue is completed.

7. The system as in claim 5 wherein the busy work comprises a plurality of operations which have no effect on the visual appearance of the frame in the front buffer.

8. The system as in claim 7 wherein the busy work comprises a plurality of invisible triangles.

9. The system as in claim 7 wherein the busy work comprises a series of vertical quads which miss all of the pixel centers.

10. The system as in claim 7 wherein the busy work comprises a series of long polygons with long shaders that perform trivial math.

11. The systen as in claim 1 wherein the interval comprises a vertical blanking interval (VBI).

13. The method as in claim 11 wherein increasing the number of scan lines associated with the interval relative to the number of scan lines not associated with the interval comprises configuring the front buffer to have a larger size than the size of the back buffer.

12. A method for hosting applications/video games for a plurality of clients comprising:

processing application/video game program code with a central processing unit (CPU) and processing graphics commands and generating a series of video frames for the application/video game with a graphics processing unit (GPU);

storing video frames within one or more back buffer as the video frames are being created in response to the execution of the graphics commands;

receiving a video frame for rendering on a display into a front buffer after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with an interval; and

increasing the number of scan lines associated with the interval relative to the number of scan lines not associated with the interval to increase the likelihood that a new frame will be completed in a back buffer and ready for transfer to the front buffer at a time when the subset of scan lines stored in the front buffer being associated with the VBI is processed.

14. The method as in claim 12 wherein the front buffer is configured to be two times the size of the back buffer.

15. The method as in claim 12 wherein in response to detecting a draw operation which will cause the new frame in the back buffer to be transferred to the front buffer, determining an amount of work remaining in a GPU command queue and, if the amount of work remaining in the command queue would cause the newly completed frame from the back buffer to be copied to the front buffer while a current scan line being output from the front buffer is outside of the VBI, then, then inserting the draw operation in the command queue.

16. The method as in claim 15 wherein, in response to detecting a draw operation which will cause the new frame in the back buffer to be transferred to the front buffer, determining whether the amount of work remaining in the command queue would cause the newly completed frame from the back buffer to be copied to the front buffer while a current scan line being output from the front buffer is outside of the interval, then the frame processing module submits busy work to the command queue to ensure that the newly completed frame from the back buffer will be copied to the front buffer while a current scan line being output from the front buffer is within the interval.

17. The method as in claim 12 wherein a first thread creates new frames within the one or more back buffers and a second thread determines a most recent frame output from the front buffer and executes a draw operation to cause the new frame in the back buffer to be transferred to the front buffer after the current work remaining in the command queue is completed.

18. The method as in claim 16 wherein the busy work comprises a plurality of operations which have no effect on the visual appearance of the frame in the front buffer.

19. The method as in claim 17 wherein the busy work comprises a plurality of invisible triangles.

20. The method as in claim 18 wherein the busy work comprises a series of vertical quads which miss all of the pixel centers.

21. The method as in claim 18 wherein the busy work comprises a series of long polygons with long shaders that perform trivial math.

22. The method as in claim 12 wherein the interval comprises a vertical blanking interval (VBI).