System and method for improving the graphics performance of hosted applications

• US 9,682,318 B2
• Filed: 06/27/2016
• Issued: 06/20/2017
• Est. Priority Date: 03/23/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Cotter, Timothy
• Primary Examiner(s): Suhol, Dmitry
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US15/194,514
• Publication Number:
  US 20160303479A1
• Time in Patent Office: 358 Days
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/556 Player lists, e.g. online players, buddy list, black list
• CPC Class Codes: A63F 2300/5593 involving scheduling aspects
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23406 involving management of server-side video buffer
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/235 : Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8458 : involving uncompressed content

Abstract

A method includes the following operations: (1) executing an application that that generates a series of video frames using a graphics processing unit (GPU); (2) determining that a first video frame in the series has been completely written to a back buffer; (3) determining whether a current scan line of a second video frame is outside of a specified region, where the second video frame is stored in a front buffer and is being read out line by line following receipt of a monitored synchronizing signal; (4) presenting the first video frame to the front buffer without frame transitions, if the current scan line is outside of the specified region and is in a vertical blanking interval (VBI); and (5) presenting the first video to the front buffer with frame transitions, if the current scan line is outside of the specified region and is not in the VBI.

Claims

1. A method comprising operations of:

executing an application that includes calls to an application program interface (API) for a graphics engine that generates a series of video frames using a graphics processing unit (GPU);

determining that a first video frame in the series has been completely written to a back buffer;

determining whether a current scan line of a second video frame is outside of a specified region, wherein the second video frame is stored in a front buffer and is being read out line by line following receipt of a monitored synchronizing signal;

inserting busy work for the GPU if the current scan line is inside of the specified region;

presenting the first video frame to the front buffer without frame transitions, if the current scan line is outside of the specified region and is in a vertical blanking interval (VBI); and

presenting the first video frame to the front buffer with frame transitions, if the current scan line is outside of the specified region and is not in the VBI.

2. The method of claim 1, further comprising an operation of transferring the first video frame from a shared buffer to the back buffer.

3. The method of claim 1, wherein the second video frame is being read out line by line to a display.

4. The method of claim 1, wherein the second video frame is being read out line by line to an encoder for compression.

5. The method of claim 4, wherein the encoder generates a compressed video stream that is transmitted over a network to a client device that decompresses and displays the video stream.

6. The method of claim 5, wherein the video stream is low latency.

7. The method of claim 1, wherein each of the operations is performed by one or more servers that receive input from the client device through an inbound routing system which includes a plurality of network switches.

8. The method of claim 1, wherein the busy work includes one or more invisible triangles, each of which takes at least a specified time.

9. The method of claim 1, wherein the busy work includes one or more vertical quads, each of which misses all pixel centers and takes at least a specified time.

10. A method comprising operations of:

executing an application that includes calls to an application program interface (API) for a graphics engine that generates a series of video frames using a graphics processing unit (GPU);

determining that a first video frame in the series has been completely written to a back buffer;

determining whether a current scan line of a second video frame is outside of a specified region, wherein the second video frame is stored in a front buffer and is being read out line by line to a display following receipt of a monitored synchronizing signal;

inserting busy work for the GPU if the current scan line is inside of the specified region;

presenting the first video frame to the front buffer without frame transitions, if the current scan line is outside of the specified region and is in a vertical blanking interval (VBI);

presenting the first video frame to the front buffer with frame transitions, if the current scan line is outside of the specified region and is not in the VBI; and

transferring the first video frame from a shared buffer to the back buffer, wherein each of the operations is performed by one or more servers that receive input from the client device through an inbound routing system which includes a plurality of network switches.

11. The method of claim 10, wherein the frame transitions include a mixing operation.

12. The method of claim 10, wherein the second video frame is being read out line by line to an encoder for compression.

13. The method of claim 12, wherein the encoder generates a compressed video stream that is transmitted over a network to a client device that decompresses and displays the video stream.

14. The method of claim 13, wherein the video stream is low latency.

15. The method of claim 10, the frame transitions include a blending operation.

16. The method of claim 10, wherein the busy work includes one or more invisible triangles, each of which takes at least a specific time.

17. The method of claim 10, wherein the busy work includes one or more vertical quads, each of which misses all pixel centers and takes at least a specified time.

18. A method comprising operations of:

executing an application that includes calls to an application program interface (API) for a graphics engine that generates a series of video frames using a graphics processing unit (GPU);

determining that a first video frame in the series has been completely written to a back buffer;

determining whether a current scan line of a second video frame is outside of a specified region, wherein the second video frame is stored in a front buffer and is being read out line by line to an encoder for compression following receipt of a monitored synchronizing signal;

inserting busy work for the GPU if the current scan line is inside of the specified region;

presenting the first video frame to the front buffer without frame transitions, if the current scan line is outside of the specified region and is in a vertical blanking interval (VBI);

presenting the first video frame to the front buffer with frame transitions, if the current scan line is outside of the specified region and is not in the VBI; and

transferring the first video frame from a shared buffer to the back buffer, wherein each of the operations is performed by one or more servers that receive input from the client device through an inbound routing system which includes a plurality of network switches.

19. The method of claim 18, wherein the encoder generates a compressed video stream that is transmitted over a network to a client device that decompresses and displays the video stream.

20. The method of claim 19, wherein the video stream is low latency.