System and method for compressing video based on detected data rate of a communication channel

• US 8,881,215 B2
• Filed: 01/23/2009
• Issued: 11/04/2014
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., Laan, Roger van der
• Primary Examiner(s): Dubasky, Gigi L
• Application Number:
  US12/359,229
• Publication Number:
  US 20090213927A1
• Time in Patent Office: 2,111 Days
• Field of Search: 725/114, 725/138, 725/144, 463 39- 42, 375/240.04, 37524012-24018
• US Class Current: 725/114
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/21 Server components or server architectures
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: detecting a maximum data rate of a communication channel between a server and a client; dynamically selecting a tile size for encoding a sequence of images based on the detected maximum data rate; logically subdividing each of the sequence of images into a plurality of tiles sized according to the selected tile size, each of the tiles having a defined position within each of the sequence of images; encoding each tile within each image of the sequence of images using a first compression format or a second compression format, wherein a tile at a particular position of a first image in the sequence of images is encoded using the first compression format and the same tile within one or more subsequent images is encoded using the second compression format.

Claims

1. A computer-implemented method comprising:

receiving a request from a client of a user to execute a video game or application;

generating control signals from the client in response to an input device of the client being actuated by the user as the user is playing the video game or using the application;

receiving the control signals and responsively executing the video game or application on a server at a hosting service center, the execution of the video game or application producing a low-latency interactive compressed video stream comprising a plurality of video images;

detecting a maximum data rate of a communication channel between the server and the client;

detecting channel characteristics of the communication channel between the server and the client, the detected channel characteristics including a supported bitrate of the communication channel;

encoding each video image of a sequence of the video images using a first compression format or a second compression format based on the detected maximum data rate, wherein a first video image in the sequence of the video images is encoded using the first compression format and one or more subsequent video images are encoded using the second compression format, wherein encoding of the sequence of the video images is performed using low latency compression to generate the interactive compressed video stream based on the detected channel characteristics;

transmitting compressed video images that have more data than can be transmitted at the supported bitrate of the communication channel in one image time over one or more additional image times, and ignoring one or more subsequent uncompressed video images while still transmitting one of the compressed video images that have more data; and

decoding the compressed video images and rendering the decoded video images on a display of a client;

wherein operations of generating control signals, receiving the control signals at the server and responsively executing the video game or application, encoding the sequence of video images, transmitting the compressed video images, and decoding and rending the decoded video images on the display are performed with a latency such that the user has the perception that the video game or application is responding instantly to the control signals.

2. The method as in claim 1 further comprising:

receiving the control signals transmitted from the client over a first communication channel to the server indicating user input to the client as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising sequences of the video images of the video game or application.

3. The method as in claim 2 further comprising:

compressing the sequence of the video images of the video game or application using a first set of compression characteristics to generate the interactive compressed video stream.

4. The method as in claim 3 further comprising:

compressing the sequence of the video images of the video game or application using a second set of compression characteristics to generate a second compressed video stream for storing on one or more storage servers at the hosting service center, the second compressed video stream usable for playing back prior the video output of the low-latency interactive compressed video stream of the video game or application for the user or one or more other users.

5. The method as in claim 4 wherein the second compressed video stream is compressed using a relatively lower level of compression to produce relatively higher quality video in comparison to the interactive compressed video stream.

6. The method as in claim 1 further comprising:

streaming the low latency video stream from the server to the client over the first communication channel, the low latency video stream being decoded by a decoder on the client and rendered on the client.

7. The method as in claim 1 wherein encoding the sequence of the video images further comprises:

uniquely compressing the video stream based on the detected characteristics of the communication channel or detected processing capabilities of the client.

8. The method as in claim 7 wherein uniquely compressing the video stream based on the detected characteristics of the communication channel between the server and the client comprises uniquely compressing the video stream based on a maximum allowable latency required for the video stream to be received and decoded at the client.

9. The method as in claim 8 wherein uniquely compressing the video stream comprises modifying a resolution of the video stream.

10. The method as in claim 8 further comprising:

detecting that the characteristics of the communication channel between the server and the client has dropped below a specified allowable quality level; and

responsively pausing the execution of the video game or application on the server.

11. The method as in claim 1 wherein detecting characteristics of the communication channel between the server and the client comprises detecting a level of packet loss or latency associated with the communication channel.

12. The method as in claim 11 further comprising responsively adjusting a peak data rate of the interactive compressed video stream based on the detected packet loss or latency.

13. The method as in claim 12 wherein adjusting the peak data rate comprises adjusting a level of compression of the interactive video stream.

14. The method as in claim 12 further comprising:

determining that channel capacity of the communication channel has increased based on continued measurements of the packet loss or latency; and

responsively increasing the peak data rate until the packet loss or latency reaches an unacceptable level.

15. The method as in claim 1 further comprising:

employing forward error correction (FEC) techniques to protect specified portions of the low-latency interactive compressed video stream.

16. The method as in claim 15 further comprising:

determining if a first specified portion of the low-latency interactive compressed video stream would result in a noticeable visual artifact, and, if so, then generating FEC data to protect the first specified portion of the low-latency interactive compressed video stream.

17. The method as in claim 15 further comprising:

generating FEC data for all I-frames output by the server and transmitted to the client.

18. The method as in claim 15 further comprising:

protecting all audio packets generated by the video game or application using the FEC.

19. The method as in claim 18 further comprising:

protecting all control signals generated by the client and sent to the server using the FEC.

20. The computer implemented method as in claim 1 wherein ignoring one or more subsequent uncompressed video images comprises not compressing or transmitting the subsequent uncompressed video images.

21. A non-transitory machine readable storage medium containing instructions that when processed by a computer cause a method to be performed, said method comprising:

generating control signals from the client in response to an input device of the client being actuated by the user as the user is playing the video game or using the application;

receiving the control signals and responsively executing the video game or application on a server at a hosting service center, the execution of the video game or application producing an interactive compressed video stream comprising a plurality of video images;

detecting a maximum data rate of a communication channel between the server and the client;

detecting channel characteristics of the communication channel between the server and the client, the detected channel characteristics including a supported bitrate of the communication channel;

encoding each video image of a sequence of the video images using a first compression format or a second compression format based on the detected maximum data rate, wherein a first video image in the sequence of the video images is encoded using the first compression format and one or more subsequent video images are encoded using the second compression format, wherein encoding of the sequence of video images is performed using low latency compression to generate the interactive compressed video stream based on the detected channel characteristics;

transmitting compressed video images that have more data than can be transmitted at the supported bitrate of the communication channel in one time over one or more additional image times, and ignoring one or more subsequent uncompressed video images while still transmitting one of the compressed video images that have more data; and

decoding the compressed video images and rendering the decoded video images on a display of a client;

wherein the operations of generating control signals, receiving the control signals at the server and responsively executing the video game or application, encoding the sequence of video images, transmitting the compressed video images, and decoding and rending the decoded video images on a display are performed with a latency such that the user has the perception that the video game or application is responding instantly to the control signals.

22. The non-transitory machine readable storage medium as in claim 21 wherein the method further comprises:

receiving the control signals transmitted from the client over a first communication channel to the server indicating user input to the client as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising sequences of the video images of the video game or application.

23. The non-transitory machine readable medium as in claim 21 wherein the method further comprises:

streaming the low latency video stream from the server to the client over the first communication channel, the low latency video stream being decoded by a decoder on the client and rendered on the client.

24. The non-transitory machine readable medium as in claim 21 wherein encoding the sequence of the video images further comprises:

uniquely compressing the video stream based on the detected characteristics of the communication channel or detected processing capabilities of the client.

25. The non-transitory machine readable medium as in claim 24 wherein uniquely compressing the video stream based on the detected characteristics of the communication channel between the server and the client comprises uniquely compressing the video stream based on a maximum allowable latency required for the video stream to be received and decoded at the client.

26. The non-transitory machine readable medium as in claim 25 wherein uniquely compressing the video stream comprises modifying a resolution of the video stream.

27. The non-transitory machine readable medium as in claim 25 wherein the method further comprises:

detecting that the characteristics of the communication channel between the server and the client has dropped below a specified allowable quality level; and

responsively pausing the execution of the video game or application on the server.

28. The non-transitory machine readable medium as in claim 21 wherein detecting characteristics of the communication channel between the server and the client comprises detecting a level of packet loss or latency associated with the communication channel.

29. The non-transitory machine readable medium as in claim 28 further comprising to cause the operation of responsively adjusting a peak data rate of the low-latency interactive compressed video stream based on the detected packet loss or latency.

30. The non-transitory machine readable medium as in claim 21 wherein ignoring one or more subsequent uncompressed video images comprises not compressing or transmitting the subsequent uncompressed video images.

31. A computing system, comprising:

one or more processors;

a non-transitory machine readable storage medium containing instructions that when processed by the one or more processors cause a method to be performed, said method comprising;

generating control signals from the client in response to an input device of the client being actuated by the user as the user is playing the video game or using the application;

receiving the control signals and responsively executing the video game or application on a server at a hosting service center, the execution of the video game or application producing an interactive compressed video stream comprising a plurality of video images;

detecting a maximum data rate of a communication channel between the server and the client;

detecting channel characteristics of the communication channel between the server and the client, the detected channel characteristics including a supported bitrate of the communication channel;

encoding each video image of a sequence of the video images using a first compression format or a second compression format based on the detected maximum data rate, wherein a first video image in the sequence of the video images is encoded using the first compression format and one or more subsequent video images are encoded using the second compression format, wherein encoding of the sequence of video images is performed using low latency compression to generate the interactive compressed video stream based on the detected channel characteristics;

transmitting compressed video images that have more data than can be transmitted at the supported bitrate of the communication channel in one image time over one or more additional image times, and ignoring one or more subsequent uncompressed video images while still transmitting one of the compressed video images that have more data; and

decoding the compressed video images and rendering the decoded video images on a display of a client;

wherein the operations of generating control signals, receiving the control signals at the server and responsively executing the video game or application, encoding the sequence of video images, transmitting the compressed video images, and decoding and rending the decoded video images on a display are performed with a latency such that the user has perception that the video game or application is responding instantly to the control signals.

32. The system as in claim 31 wherein ignoring the one or more subsequent uncompressed video images comprises not compressing or transmitting the subsequent uncompressed video images.