System and method for improving the graphics performance of hosted applications

• US 8,840,477 B2
• Filed: 09/14/2011
• Issued: 09/23/2014
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Dietrich, Douglas Sim Jr., Benitez, Nico, Cotter, Timothy
• Primary Examiner(s): D'Agostino, Paul A
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US13/232,963
• Publication Number:
  US 20120108330A1
• Time in Patent Office: 1,105 Days
• Field of Search: 707/104.1
• US Class Current: 463/42
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/556 Player lists, e.g. online players, buddy list, black list
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 1/00 Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00 architectures of general purpose stored program computers G06F15/76
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 : Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23406 : involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/235 : Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8458 : involving uncompressed content

Abstract

An application/game server comprising a central processing unit to process application/video game program code and a graphics processing unit (GPU) to process graphics commands and generate a series of video frames for the application/video game; one or more back buffers to store video frames as the video frames are being created in response to the execution of the graphics commands; a front buffer-outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a vertical blanking interval (VBI); and a frame processing module to begin copying a newly completed frame from the back buffer to the front buffer before the video data for a prior video frame stored in the front buffer has not been fully scanned out.

Claims

1. A server-based system for hosting applications/video games for a plurality of clients comprising:

an application/game server comprising a central processing unit to process application/video game program code and a graphics processing unit (GPU) to process graphics commands and generate a series of video frames for the application/video game;

one or more back buffers to store video frames as the video frames are being created in response to the execution of the graphics commands;

a front buffer to receive a video frame for rendering on a display after the video frame has been completed in one of the one or more back buffers, the front buffer outputting the video frame for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a vertical blanking interval (VBI); and

a frame processing module to begin copying a newly completed frame from the back buffer to the front buffer before the video data for a prior video frame stored in the front buffer has been fully scanned out.

2. The system as in claim 1 wherein the frame processing module comprises program code for performing the additional operations of:

determining the most recent scan line output from the front buffer;

copying the completed video frame from the back buffer to the front buffer only if the most recent scan line output from the front buffer is outside of a designated center region of the video frame.

3. The system as in claim 1 wherein the frame processing module comprises program code for performing the additional operations of:

executing a video frame mixing operation to combine the newly completed frame with the prior video frame in the front buffer.

4. The system as in claim 3 wherein the mixing operation comprises a cross-fade operation.

5. The system as in claim 3 wherein the mixing operation is a blend operation.

6. The system as in claim 2 wherein, if the current scan line is within the designated center region of the display, the frame processing module waits a period of time until the current scan line has moved outside of the designated center region before copying the newly completed frame from the back buffer to the front buffer.

7. The system as in claim 6 wherein the frame processing module comprises program code for performing the additional operations of:

inserting busy work into a command queue of the GPU based on the period of time to be waited, the amount of busy work approximating the period of time.

8. The system as in claim 7 wherein the busy work comprises a plurality of operations which have no effect on the visual appearance of the frame in the front buffer.

9. The system as in claim 8 wherein the busy work comprises a plurality of invisible triangles.

10. The system as in claim 8 wherein the busy work comprises a series of vertical quads which miss all of the pixel centers.

11. The system as in claim 8 wherein the busy work comprises a series of long polygons with long shaders that perform trivial math.

12. A method for hosting applications/video games for a plurality of clients comprising:

processing graphics commands and generating a series of video frames for the application/video game;

storing video frames in one or more back buffers as the video frames are being created in response to the execution of the graphics commands;

receiving completed video frames in a front buffer for rendering on a display after the video frame has been completed in one of the one or more back buffers, outputting the video frames from the front buffer for display one scan line at a time at a designated scan out frequency, a subset of scan lines stored in the front buffer being associated with a vertical blanking interval (VBI); and

initiating copying a newly completed frame from the back buffer to the front buffer before the video data for a prior video frame stored in the front buffer has been fully scanned out.

13. The method as in claim 12 further comprising:

determining the most recent scan line output from the front buffer; and

copying the completed video frame from the back buffer to the front buffer only if the most recent scan line output from the front buffer is outside of a designated center region of the video frame.

14. The method as in claim 12 further comprising:

executing a video frame mixing operation to combine the newly completed frame with the prior video frame in the front buffer.

15. The method as in claim 14 wherein the mixing operation comprises a cross-fade operation.

16. The method as in claim 14 wherein the mixing operation is a blend operation.

17. The method as in claim 13 wherein, if the current scan line is within the designated center region of the display, waiting a period of time until the current scan line has moved outside of the designated center region before copying the newly completed frame from the back buffer to the front buffer.

18. The method as in claim 17 wherein the frame processing module comprises program code for performing the additional operations of:

inserting busy work into a command queue of the GPU based on the period of time to be waited, the amount of busy work approximating the period of time.

19. The method as in claim 18 wherein the busy work comprises a plurality of operations which have no effect on the visual appearance of the frame in the front buffer.

20. The method as in claim 18 wherein the busy work comprises a plurality of invisible triangles.

21. The method as in claim 18 wherein the busy work comprises a series of vertical quads which miss all of the pixel centers.

22. The method as in claim 18 wherein the busy work comprises a series of long polygons with long shaders that perform trivial math.