System and Method for Compressing Video Based on Latency Measurements and Other Feedback

• US 20160080739A1
• Filed: 11/24/2015
• Published: 03/17/2016
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Furman, Scott, McCool, Robert, Buckley, Ian, Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy
• Granted Patent: US 9,770,657 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/266 : Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A system and method of playing a video game in a streaming mode includes executing a selected video game on an assigned game server, receiving execution control signals corresponding to user inputs to a client device to generate multiple video frames. Encoding the video frames in a shared hardware compression. Some of the encoded video frames depend on a preceding frame. A first and a second encoded video frame are transmitted to the client device. A first feedback signal from a decoder in the client device identifies that the second encoded video frame was not received the decoder. A third encoded video frame dependency is adjusted to produce an adjusted third encoded video frame that depends from the first encoded video frame and the adjusted third encoded video frame is transmitted to the client device.

Claims

1. A method comprising:

generating a user interface in a hosting service control system;

transmitting data for the user interface to a client device, the hosting service control system having communication with a database of user information and enabling communication between the client device and one or more game servers;

receiving a video game selection from the user interface displayed on a display device of the client device;

assigning a first game server to the client device and routing the video game selection to the first game server, the first game server provided with communication with the client device over a network;

receiving a first control signal at the first game server, the first control signal corresponding to a plurality of user inputs to the client device as the user is playing the selected video game;

executing the selected video game on the first game server corresponding to the first control signal received from the client device, the execution of the selected video game is configured to generate a plurality of video frames;

encoding the plurality of video frames in a first portion of a shared hardware compression to produce a corresponding plurality of encoded video frames, wherein some of the plurality of encoded video frames depend on a preceding one of the plurality of encoded video frames;

transmitting a first encoded video frame and a second encoded video frame of the plurality of encoded video frames to the client device, the client device including a decoder for decoding the plurality of encoded video frames;

receiving a first feedback signal from the decoder, the first feedback signal being received by the first game server, the first feedback signal identifying that the first encoded video frame was received by the decoder and that identifying the second encoded video frame was not received the decoder;

adjusting a dependency of a third encoded video frame of the plurality of encoded video frames to produce an adjusted third encoded video frame, wherein the adjusted third encoded video frame is changed to depend from the first encoded video frame; and

transmitting the adjusted third encoded video frame to the client device for decoding by the decoder to produce decoded video frames to be rendered on the display of the client device as the user plays the video game, wherein the first game server is configured to stream additional encoded video frames so that the video game is played in a streaming mode and execution of the selected video game by the first game server that is remote from the client device.

2. The method of claim 1, wherein the client device is disposed in a first location, the hosting service control system is disposed in a second location and the first game server is disposed in a third location and the first game server is one of the one or more game servers.

3. The method of claim 1, wherein at least a portion of the shared hardware compression is disposed in the third location.

4. The method of claim 1, wherein the first game server is configured for playing the selected video game and wherein assigning the first game server to the client device includes the hosting service control system instructing an inbound router portion to transfer subsequent control signals from the client device to the first game server.

5. The method of claim 1, wherein assigning the first game server to the client device further includes the hosting service control system assigning the first portion of the shared hardware compression to the first game server.

6. The method of claim 1, further comprising:

transmitting a fourth encoded video frame of the plurality of encoded video frames to the client device;

receiving a second feedback signal from the decoder, the second feedback signal identifying the fourth encoded video frame was received by the decoder with a latency exceeding a threshold level;

executing the video game to a selected pausing point on the first game server; and

pausing the execution of the game on the first game server at the selected pausing point.

7. The method of claim 6, further comprising:

assigning a second game server to the client device including instructing the inbound router to transfer subsequent control signals from the client device to the second game server and routing the video game selection from the client device to the second game server, the second game server coupled to the client device by the network, the second game server being disposed in a fourth location;

assigning a second portion of the shared hardware compression to the second game server, the second portion of the shared hardware compression being disposed in the fourth location;

transmitting a fifth encoded video frame of the plurality of encoded video frames to the client device; and

receiving a third feedback signal from the decoder, the third feedback signal identifying the fifth encoded video frame was received by the decoder with a latency less than the threshold level.

8. The method of claim 7, wherein pausing the execution of the video game on the first game server at the selected pausing point includes:

capturing a paused game state of the paused execution of the game in the first game server;

transmitting the captured paused game state from the first game server to the hosting service control system;

saving the captured paused game state in the database of user information; and

transmitting the captured paused game state to the second game server.

9. The method of claim 8, wherein assigning the second game server to the client device further includes determining a minimum latency between the client device and the second game server.

10. The method of claim 8, wherein assigning the second game server to the client device further includes evaluating an IP address of the client device.

11. The method of claim 1, wherein assigning the second game server to the client device further includes evaluating a physical location of the client device.

12. The method of claim 1, wherein assigning the first game server to the client device further includes determining a minimum latency between the client device and the first game server.

13. The method of claim 1, wherein assigning the first game server to the client device further includes evaluating an IP address of the client device.

14. The method of claim 1, wherein assigning the first game server to the client device further includes evaluating a physical location of the client device.

15. A method comprising:

receiving a video game selection from a client device, the video game selection received by a hosting service control system having communication with a database of user information and enabling communication between the client device and one or more game servers;

assigning a first game server to the client device and routing the video game selection to the first game server, the first game server provided with communication with the client device over a network;

receiving a first control signal at the first game server, the first control signal corresponding to a plurality of user inputs to the client device as the user is playing the selected video game;

executing the selected video game on the first game server corresponding to the first control signal received from the client device, the execution of the selected video game is configured to generate a plurality of video frames;

encoding the plurality of video frames in a first portion of a shared hardware compression to produce a corresponding plurality of encoded video frames, wherein some of the plurality of encoded video frames depend on a preceding one of the plurality of encoded video frames;

transmitting a first encoded video frame and a second encoded video frame of the plurality of encoded video frames to the client device, the client device including a decoder for decoding the plurality of encoded video frames;

receiving a first feedback signal from the decoder, the first feedback signal being received by the first game server, the first feedback signal identifying that the first encoded video frame was received by the decoder and that identifying the second encoded video frame was not received the decoder;

adjusting a dependency of a third encoded video frame of the plurality of encoded video frames to produce an adjusted third encoded video frame, wherein the adjusted third encoded video frame is changed to depend from the first encoded video frame; and

transmitting the adjusted third encoded video frame to the client device for decoding by the decoder to produce decoded video frames to be rendered on a display of the client device as the user plays the video game, wherein the first game server is configured to stream additional encoded video frames so that the video game is played in a streaming mode and execution of the selected video game by the first game server that is remote from the client device.

16. A system for hosting a streaming on-line video game comprising:

a hosting service control system configured to receive a video game selection from a client device and to enable communication between the client device and one or more game servers;

a database of user information in communication with the hosting service control system;

a first game server in communication with the client device over a network, wherein the hosting service control system assigns the client device to the first game server and routes the video game selection to the first game server, the first game server configured to receive a first control signal, the first control signal corresponding to a plurality of user inputs to the client device as the user is playing the selected video game and to execute the selected video game corresponding to the first control signal, the execution of the selected video game is configured to generate a plurality of video frames; and

a shared hardware compression including a first portion of the shared hardware compression assigned to the first game server for encoding the plurality of video frames and to produce and transmit a corresponding plurality of encoded video frames, wherein some of the plurality of encoded video frames depend on a preceding one of the plurality of encoded video frames, wherein a first encoded video frame and a second encoded video frame of the plurality of encoded video frames are transmitted to the client device, the client device including a decoder corresponding to the shared hardware compression for decoding the plurality of transmitted, encoded video frames and to produce a first feedback signal from the decoder, the first feedback signal being received by the first game server, the first feedback signal identifying that the first encoded video frame was received by the decoder and that identifying the second encoded video frame was not received the decoder;

wherein the shared hardware compression being configured to adjust a dependency of a third encoded video frame of the plurality of encoded video frames to produce an adjusted third encoded video frame, wherein the adjusted third encoded video frame is changed to depend from the first encoded video frame and transmit the adjusted third encoded video frame to the client device for decoding by the decoder to produce decoded video frames to be rendered on a display of the client device as the user plays the video game, wherein the first game server is configured to stream additional encoded video frames so that the video game is played in a streaming mode and execution of the selected video game by the first game server that is remote from the client device.