System and method for compressing video based on latency measurements and other feedback

• US 9,192,859 B2
• Filed: 08/07/2009
• Issued: 11/24/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US12/538,081
• Publication Number:
  US 20100166065A1
• Time in Patent Office: 2,300 Days
• Field of Search: 348/390.1, 348/387.1, 348/400, 37524001-24026
• US Class Current: 1/1
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/266 : Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A system and method for performing video compression and streaming video from an online hosted video game or application. The online hosted video game or application is streamed to the client with a latency such that the user has the perception that the selected video game or application is responding instantly to the control signals received from the client device. In addition, feedback information received from the client device is used to determine characteristics of a communication channel between a server and the client. The video stream encoding is adjusted based on the detected communication channel characteristics, while maintaining a latency such that the user has the perception that the selected video game or application is responding instantly.

Claims

1. A computer-implemented method for performing video compression and delivery of video from an online hosted video game comprising:

receiving a request by a first server from a client device over a network for playing a low latency video game at a hosting service center comprised of a plurality of hosting servers including the first server;

receiving control signals by the first server, the control signals being received from the client device over the network to a first the first server, the control signals indicating user input to the client device as the user is playing the video game and responsively executing the video game by the first server to generate video output comprising sequences of video images of the video game;

detecting an initial channel characteristic of a first network communication channel between the first server and the client device, the initial channel characteristic including a detected latency and a supported data rate of the first network communication channel;

encoding the video output to generate a first plurality of compressed video frames or portions thereof based on the initial channel characteristic;

transmitting the first plurality of compressed video frames or portions thereof to the client device over the first network communication channel, the compressed video frames or portions thereof configured to be decoded by a decoder on the client device and configured to be rendered on the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output and transmitting the first plurality of compressed video frames to the client device, and decoding the first plurality of compressed video frames on the client device is performed with a latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device;

receiving feedback information from the client device, the feedback information usable to determine a current channel characteristic of the first network communication channel between the first server and the client device;

encoding the video output to generate a second plurality of compressed video frames or portions thereof based on the current channel characteristic, the second plurality of video frames being compressed differently than the first plurality of video frames based on the differences between the initial channel characteristic and the current channel characteristic;

transmitting the second plurality of compressed video frames or portions thereof to the client device over the first network communication channel, the second plurality of compressed video frames or portions thereof being configured to be decoded by the decoder on the client device and configured to be rendered by the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output to generate the second plurality of compressed video frames and transmitting the second plurality of compressed video frames to the client device over the first network communication channel, and decoding the second plurality of compressed video frames on the client device is also performed with the latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device; and

wherein transmitting the first plurality of compressed video frames and the second plurality of compressed video frames includes mitigating a channel bandwidth peak by distributing large compressed video frames over a sequence of successive frame times including dividing each one of the lame compressed video frames into a plurality of tiles including at least one I tile and at least one P tile and all of the remaining tiles of the plurality of tiles being only P tiles.

2. The method as in claim 1 wherein the feedback from the client device indicates whether one or more video frames or portions thereof have not been successfully received and/or decoded, wherein:

(1) if the current channel characteristic indicate a latency above a specified threshold, then encoding a previously presented video frame or portions thereof which is dependent on a previously transmitted video frame or portions thereof;

or

(2) if the current channel characteristic indicate a latency below a specified threshold, then encoding a previously presented video frame to be independent from the previously transmitted video frame or portions thereof.

3. The method as in claim 2 wherein the previously presented frame or portions thereof which is independent from the frame framcs or portions thereof comprise an I-frame.

4. The method as in claim 1 wherein the feedback information comprises an indication that the frame or portions thereof have been successfully received and/or decoded at the client device.

5. The method as in claim 1 wherein the feedback information comprises an indication that the frame or portions thereof have not been successfully received and/or decoded at the client device.

6. The method as in claim 1 wherein the video frames comprise images of a video game being played by a user at the client device.

7. The method as in claim 1 wherein compressing the second plurality of video frames differently comprises employing a higher compression ratio to compress the first plurality of video frames.

22. The method as in claim 1 wherein a forward error correction encoding is used to protect at least one of the first plurality of compressed video frames, the second plurality of compressed video frames, an associated audio stream, or the control signals received by the first server.

23. The method as in claim 22 wherein the forward error correction codes are used to protect at least one of a specified portion of the first plurality of compressed video frames or a specified portion of the second plurality of compressed video frames but not protecting a non-specified portion of the first plurality of compressed video frames and a non-specified portion of the second plurality of compressed video frames.

24. The method as in claim 23 wherein the forward error correction codes are used to protect data causing a visual artifact if lost.

25. The method as in claim 23 further comprising:

adjusting the forward error correction encoding used to protect the at least one of the specified portions of the first plurality of compressed video frames or the specified portions of the second plurality of compressed video frames in response to detecting that one or more video frames or portions thereof have not been successfully received and/or decoded.

26. The method as in claim 1 wherein mitigating the channel bandwidth peak by distributing large compressed video frames over the sequence of successive frame times includes mitigating channel bandwidth peaks by distributing large compressed video frames which would result in bandwidth peaks over a sequence of successive frame times.

27. The method as in claim 1 wherein mitigating the channel bandwidth peak by distributing large compressed video frames over the sequence of successive frame times includes mitigating channel bandwidth peaks by transmitting compressed frames that have more data than can be transmitted at a peak data rate in one frame time over one or more additional frame times.

28. The method as in claim 27 wherein the client device is configured to continue to display the previously decompressed video frame during the one or more additional frame times.

29. The method as in claim 28 wherein the client device is configured to ignore video frames received during the one or more additional frame times.

30. The method as in claim 27 wherein a video frame rate temporarily decreases during the one or more additional frame times.

31. The method as in claim 27 wherein the one or more additional frame time is more than 1 frame time.

32. The method as in claim 1 further comprising dynamically adjusting a determined maximum data rate of at least one of the first plurality of compressed video frames or the plurality of compressed video frames in response to a current detected level of packet loss or latency.

33. The method as in claim 28 wherein the data rate of at least one of the first plurality of compressed video frames or the second plurality of compressed video frames is adjusted by modifying a frame rate.

34. The method as in claim 28 wherein the data rate of at least one of the first plurality of compressed video frames or the second plurality of compressed video frames is adjusted by modifying a compression quality.

35. The method as in claim 28 wherein the data rate at least one of the first plurality of compressed video frames or the second plurality of compressed video frames is adjusted by modifying an image resolution of the corresponding first plurality of compressed video frames or the second plurality of compressed video frames.

36. The method as in claim 35 wherein when the image resolution is modified, the client scales up the image to maintain the same image size on a display screen.

37. The method as in claim 1 wherein a single frame time comprises 1/60 of a second or 1/120 of a second.

8. A system comprising a memory for storing program code and a processor for processing the program code to perform the operations of:

receiving a request by a first server from a client device over a network for playing a low latency video game at a hosting service center comprised of a plurality of hosting servers including the first server;

receiving control signals by the first server, the control signals being received from the client device over the network to the first server, the control signals indicating user input to the client device as the user is playing the video game and responsively executing the video game on by the first server to generate video output comprising sequences of video images of the video game;

detecting an initial channel characteristic of a first network communication channel between the first server and the client device, the initial channel characteristic including a detected latency and a supported data rate of the first network communication channel;

encoding the video output to generate a first plurality of compressed video frames or portions thereof based on the initial channel characteristic;

transmitting the first plurality of compressed video frames or portions thereof to the client device over the first network communication channel, the compressed video frames or portions thereof configured to be decoded by a decoder on the client device and configured to be rendered on the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output and transmitting the first plurality of compressed video frames to the client device, and decoding the first plurality of compressed video frames on the client device is performed with a latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device;

receiving feedback information from the client device, the feedback information usable to determine a current channel characteristic of the first network communication channel between the first server and the client device; and

encoding the video output to generate a second plurality of compressed video frames based on the current channel characteristic, the second plurality of video frames being compressed differently than the first plurality of video frames based on the differences between the initial channel characteristic and the current channel characteristic; and

transmitting the second plurality of compressed video frames or portions thereof to the client device over the first network communication channel, the second plurality of compressed video frames or portions thereof configured to be decoded by the decoder on the client device and configured to be rendered by the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output to generate the second plurality of compressed video frames and transmitting the second plurality of compressed video frames to the client device over the first network communication channel, and decoding the second plurality of compressed video frames on the client device is also performed with the latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device; and

wherein transmitting the first plurality of compressed video frames and the second plurality of compressed video frames includes mitigating a channel bandwidth peak by distributing large compressed video frames over a sequence of successive frame times including dividing each one of the large compressed video frames into a plurality of tiles including at least one I tile and at least one P tile and all of the remaining tiles of the plurality of tiles being only P tiles.

9. The system as in claim 8 wherein the feedback from the client device indicates whether one or more video frames or portions thereof have not been successfully received and/or decoded, wherein:

(1) if the current channel characteristic indicate a latency above a specified threshold, then encoding a previously presented video frame or portions thereof which is dependent on a previously transmitted video frame or portions thereof;

or

(2) if the current channel characteristic indicate a latency below a specified threshold, then encoding a previously presented video frame or portions thereof to be independent from the previously transmitted video frames or portions thereof.

10. The system as in claim 9 wherein the previously presented frame or portions thereof which is independent from the frame or portions thereof comprise an I- frame.

11. The system as in claim 8 wherein the feedback information comprises an indication that the frame or portions thereof have been successfully received and/or decoded at the client device.

12. The system as in claim 8 wherein the feedback information comprises an indication that the frame or portions thereof have not been successfully received and/or decoded at the client device.

13. The system as in claim 8 wherein the video frames comprise images of a video game being played by a user at the client device.

14. The system as in claim 8 wherein compressing the second plurality of video frames differently comprises employing a higher compression ratio to compress the first plurality of video frames.

38. The system as in claim 8 wherein a single frame time comprises 1/60 of a second or 1/120 of a second.

15. A machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:

receiving a request by a first server from a client device over a network for playing a low latency video game at a hosting service center comprised of a plurality of hosting servers including the first server;

receiving control signals by the first server, the control signals being received from the client device over the network to the first server, the control signals indicating user input to the client device as the user is playing the video game and responsively executing the video game by the first server to generate video output comprising sequences of video images of the video game;

detecting an initial channel characteristic of a first network communication channel between the first server and the client device, the initial channel characteristic including a detected latency and a supported data rate of the first network communication channel;

encoding the video output to generate a first plurality of compressed video frames or portions thereof based on the initial channel characteristic;

transmitting the first plurality of compressed video frames or portions thereof to a client device over the first network communication channel, the compressed video frames or portions thereof configured to be decoded by a decoder on the client device and configured to be rendered on the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output and transmitting the first plurality of compressed video frames to the client device, and decoding the first plurality of compressed video frames on the client device is performed with a latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device;

receiving feedback information from the client device, the feedback information usable to determine current a current channel characteristic of the first network communication channel between the first server and the client device; and

encoding the video output to generate a second plurality of compressed video frames or portions thereof based on the current channel characteristic, the second plurality of video frames being compressed differently than the first plurality of video frames based on the differences between the initial channel characteristic and the current channel characteristic; and

transmitting the second plurality of compressed video frames or portions thereof to the client device over the first network communication channel, the second plurality of compressed video frames or portions thereof configured to be decoded by the decoder on the client device and configured to be rendered by the client device as the user plays the video game;

wherein the operations of receiving control signals by the first server, executing the video game, encoding the video output to generate the second plurality of compressed video frames and transmitting the second plurality of compressed video frames to the client device over the first network communication channel, and decoding the second plurality of compressed video frames on the client device is also performed with the latency such that the user has the perception that the selected video game is responding instantly to the control signals received from the client device; and

wherein transmitting the first plurality of compressed video frames and the second plurality of compressed video frames includes mitigating a channel bandwidth peak by distributing large compressed video frames over a sequence of successive frame times including dividing each one of the large compressed video frames into a plurality of tiles including at least one I tile and at least one P tile and all of the remaining tiles of the plurality of tiles being only P tiles.

16. The machine-readable medium as in claim 15 wherein the feedback from the client device indicates whether one or more video frame or portions thereof have not been successfully received and/or decoded, wherein:

(1) if the current channel characteristic indicate a latency above a specified threshold, then encoding a previously presented video frame which is dependent on a previously transmitted video frame;

or

(2) if the current channel characteristic indicate a latency below a specified threshold, then encoding a previously presented video frame to be independent from the previously transmitted video frame.

17. The machine-readable medium as in claim 16 wherein the previously presented frame or portions thereof which is independent from the frame comprise an I- frame.

18. The machine-readable medium as in claim 15 wherein the feedback information comprises an indication that the frame or portions thereof have been successfully received and/or decoded at the client device.

19. The machine-readable medium as in claim 15 wherein the feedback information comprises an indication that the frame or portions thereof have not been successfully received and/or decoded at the client device.

20. The machine-readable medium as in claim 15 wherein the video frames comprise images of a video game being played by a user at the client device.

21. The machine-readable medium as in claim 15 wherein compressing the second plurality of video frames differently comprises employing a higher compression ratio to compress the first plurality of video frames.

39. The machine-readable medium as in claim 15 wherein a single frame time comprises 1/60 of a second or 1/120 of a second.