System and method for selecting a video encoding format based on feedback data

• US 9,781,435 B2
• Filed: 08/16/2016
• Issued: 10/03/2017
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Rob, Buckley, Ian
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US15/238,653
• Publication Number:
  US 20160360208A1
• Time in Patent Office: 413 Days
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/32 using local area network [LAN] connections
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 : Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

Methods and systems for processing instructions by an online video game hosting system are provided. One example method includes receiving a control signal for a video game. The control signal is received by a server of the online video game hosting system, and the control signal is transmitted from a client device to the server over a network. Then, executing the online video game by the server while receiving the control signal. The executing causes generation of a sequence of video frames and encoding the sequence of video frames to generate a first compressed sequence of video frames and generate a second compressed sequence of video frames. Transmitting the first compressed sequence of video frames to the client device over the network for decoding by the client device to produce a first decoded sequence of video frames for displaying on a display of the client device. Receiving feedback information from the client device, the feedback information used to determine whether one or more of the first compressed sequence of video frames has been successfully decoded at the client device and modifying the encoding of the sequence of video frames in response to the feedback information. The second compressed sequence of video frames are directed to a storage associated with the server and wherein the quality of the second compressed sequence of video frames is higher than a quality of the first compressed sequence of video frames.

Claims

1. A computer-implemented method for processing instructions by an online video game hosting system comprising:

receiving a control signal for a video game, the control signal being received by a server of the online video game hosting system, the control signal being transmitted from a client device to the server over a network;

executing the online video game by the server while receiving the control signal, the executing causes generation of a sequence of video frames;

encoding the sequence of video frames to generate a first compressed sequence of video frames and generate a second compressed sequence of video frames;

transmitting the first compressed sequence of video frames to the client device over the network for decoding by the client device to produce a first decoded sequence of video frames for displaying on a display of the client device;

receiving feedback information from the client device, the feedback information used to determine whether one or more of the first compressed sequence of video frames has been successfully decoded at the client device;

modifying the encoding of the sequence of video frames in response to the feedback information, wherein the second compressed sequence of video frames being generated with a quality that is independent of the feedback information received from the client device for the first compressed sequence of video frames; and

directing the second compressed sequence of video frames to a storage associated with the server and wherein the quality of the second compressed sequence of video frames is higher than a quality of the first compressed sequence of video frames.

2. The method as in claim 1 further comprising:

receiving a second control signal by the server, the second control signal being from a second client device;

retrieving the second compressed sequence of video frames from the storage in response to the second control signal; and

transmitting the second compressed sequence of video frames to the second client device for decoding by the second client device to produce a second decoded sequence of video frames for display on a second display of the second client device.

3. The method as in claim 2 wherein transmitting the second compressed sequence of video frames to the second client device includes a time delay from when the first compressed sequence of video frames were transmitted to the client device.

4. The method as in claim 2, wherein the second client device is a spectating client device.

5. The method as in claim 1, wherein the quality of the first compressed sequence of video frames includes a first frame rate and the quality of the second compressed sequence of video frames includes a second frame rate that is greater than the first frame rate.

6. The method as in claim 1, wherein the quality of the first compressed sequence of video frames includes a first frame resolution and the quality of the second compressed sequence of video frames includes a second frame resolution that is greater than the first frame resolution.

7. The method as in claim 1, wherein the quality of the second compressed sequence of video frames includes a second frame resolution that is equal to a frame resolution of the generated sequence of video frames when the second compressed sequence of video frames is decoded by a second client device.

8. The method as in claim 1, wherein the quality of the first compressed sequence of video frames includes a first frame size and the quality of the second compressed sequence of video frames includes a second frame size that is greater than the first frame size.

9. The method as in claim 1, wherein the quality of the first compressed sequence of video frames includes a first scaling factor and the quality of the second compressed sequence of video frames includes a second scaling factor that is less than the first scaling factor.

10. A computer-implemented method for processing instructions by an online video game hosting system comprising:

receiving a first control signal for a video game, the first control signal being received by a server of the online video game hosting system, the first control signal being transmitted from a first client device to the server over a network;

executing the online video game by the server while receiving the control signal, the executing causes generation of a sequence of video frames;

encoding the sequence of video frames to generate a first compressed sequence of video frames and generate a second compressed sequence of video frames, wherein a quality of the second compressed sequence of video frames is higher than a quality of the first compressed sequence of video frames;

transmitting the first compressed sequence of video frames to the first client device over the network for decoding by the first client device to produce a first decoded sequence of video frames for displaying on a first display of the first client device;

directing the second compressed sequence of video frames to a storage associated with the server;

receiving a second control signal by the server, the second control signal being from a second client device;

retrieving the second compressed sequence of video frames from the storage in response to the second control signal; and

transmitting the second compressed sequence of video frames to the second client device for decoding by the second client device to produce a second decoded sequence of video frames for display on a second display of the second client device, wherein the second client device is a spectating client device.

11. The method as in claim 10 wherein transmitting the second compressed sequence of video frames to the second client device includes delaying transmitting the second compressed sequence of video frames to the second client device from a point in time after the first compressed sequence of video frames were transmitted to the first client device.

12. The method as in claim 10 wherein storage associated with the server is a delay buffer that is in communication with the server.

13. The method as in claim 10 further comprising detecting a data rate capacity of the network and wherein the quality of the first compressed sequence of video frames includes a first data rate less than the detected data rate capacity of the network.

14. The method as in claim 13 wherein the quality of the second compressed sequence of video frames includes a second data rate that is independent of the detected data rate capacity of the network.

15. The method as in claim 10, wherein the quality of the first compressed sequence of video frames includes a first frame rate and the quality of the second compressed sequence of video frames includes a second frame rate that is greater than the first frame rate.

16. The method as in claim 10, wherein the quality of the first compressed sequence of video frames includes a first frame resolution and the quality of the second compressed sequence of video frames includes a second frame resolution that is greater than the first frame resolution.

17. The method as in claim 10, wherein the quality of the first compressed sequence of video frames includes a first frame size and the quality of the second compressed sequence of video frames includes a second frame size that is greater than the first frame size.

18. The method as in claim 10, wherein the quality of the first compressed sequence of video frames includes a first scaling factor and the quality of the second compressed sequence of video frames includes a second scaling factor that is less than the first scaling factor.

19. An online video game hosting system comprising:

a hosting service including a server, the server for executing a video game while receiving a control signal for the video game, the control signal being from a first client device to the server over a network, wherein executing the video game includes;

generating a sequence of video frames;

encoding the sequence of video frames by a shared compression resource to generate a first compressed sequence of video frames and generate a second compressed sequence of video frames, wherein a quality of the second compressed sequence of video frames is higher than a quality of the first compressed sequence of video frames;

transmitting the first compressed sequence of video frames to the first client device over the network for decoding by the first client device to produce a first decoded sequence of video frames for displaying on a first display of the first client device;

directing the second compressed sequence of video frames to a delay buffer associated with the server;

receiving a spectating request by the server, the spectating request being from a second client device, wherein the second client device is a spectating client device;

retrieving the second compressed sequence of video frames from the delay buffer in response to the spectating request; and

transmitting the second compressed sequence of video frames to the second client device for decoding by the second client device to produce a second decoded sequence of video frames for display on a second display of the second client device.