System and method for improving the graphics performance of hosted applications

• US 10,099,129 B2
• Filed: 09/19/2016
• Issued: 10/16/2018
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Dietrich, Jr., Douglas Sim, Benitez, Nico, Cotter, Timothy
• Primary Examiner(s): Suhol, Dmitry
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US15/269,968
• Publication Number:
  US 20170007923A1
• Time in Patent Office: 757 Days
• Field of Search: None
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: A63F 2300/203 Image generating hardware
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8076 Shooting
• CPC Class Codes: H04H 20/42 Arrangements for resource management
• CPC Class Codes: H04H 60/32 : Arrangements for monitoring conditions of receiving stations, e.g. malfunction or breakdown of receiving stations
• CPC Class Codes: H04L 65/403 : Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 : for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 : intermediate
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/2405 : Monitoring of the internal components or processes of the server, e.g. server load allocation of resources in multiprogramming arrangements G06F9/50; performance measurement of computer activity G06F11/34
• CPC Class Codes: H04N 21/242 : Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection

Abstract

One or more hardware components identify a bottleneck stage within a processor pipeline that processes frames of a video stream. The bottleneck stage has a first clock. An upstream stage receives a feedback signal from the bottleneck stage. The upstream stage has a second clock and the feedback signal includes information as to time required by the bottleneck stage to operate on data and information as to time the data spent queued. The upstream stage adjusts the speed at which the upstream stage operates and queues data to approximate the speed at which the bottleneck stage is operating and queuing data.

Claims

1. A method, comprising operations of:

identifying a bottleneck stage within a processing pipeline that processes frames of a video stream, wherein the bottleneck stage has a first clock;

receiving a feedback signal from the bottleneck stage that is one of a plurality upstream stages of the processing pipeline, wherein one or more of a remaining of the plurality of upstream stages has a second clock and the feedback signal includes information as to time spent by the bottleneck stage operating on data and information as to time the data spent queued; and

adjusting a speed at which one or more of the plurality of upstream stages operates and queues data to approximate a speed at which the bottleneck stage is operating and queuing data, wherein each of the operations is performed by one or more hardware components that include one or more stages within a central processor unit (CPU) and one or more stages within a graphics processing unit (GPU),

wherein a phase-locking component maintains statistical information about the upstream stages including the bottleneck stage, the phase-locking component uses the statistical information to make a prediction that data will be available when the bottleneck stage is ready for the data.

2. The method as in claim 1, wherein the video stream is generated by program code of a video game being played by a user.

3. The method as in claim 2, wherein the video game is executed on a hosting server, wherein the user plays the video game from a client computer, and wherein the pipeline stages are stages within the hosting server.

4. The method of claim 1, wherein the prediction is within a predetermined confidence level.

5. The method of claim 1, wherein the first clock is in a first physical device and wherein the second clock is in a second physical device.

6. The method of claim 5, wherein the first physical device and the second physical device are in different geographic locations.

7. A method, comprising operations of:

identifying a bottleneck stage within a processing pipeline that processes frames of a video stream with a plurality of pipeline stages;

receiving a feedback signal from each of the plurality of pipeline stages including the bottleneck stage and one or more upstream stages, wherein the feedback signal includes information as to time spent by the bottleneck stage operating on data and information as to time the data spent queued; and

adjusting a speed at which the one or more upstream stages are processing frames of the video stream to approximate a speed at which the bottleneck stage is processing the frames of the video stream, wherein the adjusting the speed at which the one or more upstream stages are processing frames of the video stream uses statistical information from each state of the plurality of stages to predict with a predetermined confidence level that data will be ready when the bottleneck stage requires said data to minimize an amount of queuing by the bottleneck stage, wherein the video game is executed on a hosting server, wherein a user plays the video game from a client computer, and wherein the pipeline stages are stages within the hosting server, and the processing pipeline includes one or more stages within a central processor unit (CPU) and one or more stages within a graphics processing unit (GPU).

8. The method as in claim 7, wherein the video stream is generated by program code of a video game being played by a user.

9. The method of claim 7, wherein the statistical information is maintained by a phase-locking component.

10. The method of claim 7, wherein the prediction utilizes relative clocks for the stages of the processing pipeline.

11. The method of claim 10, wherein physical devices of the processing pipeline are in different geographic locations.