Video compression system and method for compensating for bandwidth limitations of a communication channel

• US 10,130,891 B2
• Filed: 01/23/2009
• Issued: 11/20/2018
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Jebari, Mohammed
• Application Number:
  US12/359,127
• Publication Number:
  US 20090225828A1
• Time in Patent Office: 3,588 Days
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/21 Server components or server architectures
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: detecting a maximum data rate of a communication channel between a server and a client; transmitting a video stream comprising a series of sequential frames from the server to the client; detecting that the maximum data rate will be exceeded if a particular frame of the sequence of frames is transmitted from the server to the client over the communication channel; and in lieu of transmitting the frame which could cause the maximum data rate to be exceeded, causing the client to re-render the previous frame of the sequence of frames, thereby effectively reducing the frame rate of the video stream rendered on the client.

Claims

1. A computer-implemented method for hosting online video games, performing video compression and streaming video over a network comprising:

receiving control signals by a server, the control signals received from a client device over a network to the server, the control signals configured to be in response to a user input from an input device of the client device being actuated by a user as the user is playing a video game;

executing program code of the video game on the server in response to the received control signals;

generating a plurality of video frames resulting from execution of the video game;

encoding by one or more shared compression resources the plurality of video frames using low latency compression to generate a set of low latency compressed video frames;

detecting by the server, a maximum data rate of a communication channel established between the server and the client device, wherein determining the maximum data rate is enabled by sending an increasingly higher data rate test stream to the client device until at least one of a packet loss rate or an increased latency indicates the maximum data rate has been exceeded and then responsively reducing a data rate of the test stream until at least one of the packet loss rate or the latency have reached acceptable values;

updating a compression process state that sets an amount of compression for the communication channel for the detected maximum data rate;

transmitting the set of low latency compressed video frames from the server to the client device using the compression process state;

while transmitting, continuing to monitor when the maximum data rate for the communication channel has decreased based on measurements of at least one of an increase in the packet loss rate or said latency, and responsively reducing a data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency have reached an acceptable level; and

while transmitting, continuing to monitor when the maximum data rate of the communication channel has increased based on continued measurements of at least one of a decrease in packet loss rate or said latency, and responsively increasing the data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency reaches an unacceptable level;

wherein the compression process state is updated to adjust the amount of compression for responsively reducing or increasing the data rate;

wherein determining the maximum data rate includes receiving feedback from the client device for determining said packet loss rate or said latency.

2. The method as in claim 1 wherein the set of low latency compressed video frames are compressed in either a first encoding format or a second encoding format, the first encoding format comprising video frames which are dependent on previous video frames and the second encoding format comprising video frames which are not dependent on previous video frames.

3. The method as in claim 2 wherein the first encoding format is selected to reduce the data rate of the set of low latency compressed video frames to ensure that the maximum data rate of the communication channel will not be exceeded.

4. The method as in claim 1 further comprising:

encoding audio generated by the video game to generate a low latency compressed audio stream.

5. The method as in claim 1 further comprising detecting a change in the maximum data rate of the communication channel and dynamically adjusting a data rate of the set of low latency compressed video frames to accommodate the changed maximum data rate.

6. The method as in claim 5 wherein the data rate is adjusted by modifying a frame rate of the set of low latency compressed video frames.

7. The method as in claim 5 wherein the data rate is adjusted by modifying a quality of the set of low latency compressed video frames.

8. The method as in claim 5 wherein the data rate is adjusted by modifying an image resolution of the set of low latency compressed video frames.

9. The method as in claim 8 wherein when the image resolution is modified, the client device is configured to scale up the image to maintain the same image size on a display screen coupled to the client device.

10. The method as in claim 1 further comprising:

pausing the video game when the communication channel to the client device becomes unresponsive.

11. The method as in claim 1 further comprising generating forward error correction (FEC) data to protect at least a portion of the set of low latency compressed video frames.

12. The method as in claim 11 generating forward error correction (FEC) data to protect the portion of the low latency compressed video frames when a loss of a portion of the set of low latency compressed video frames would result in a noticeable visual artifact.

13. The method as in claim 12 wherein forward error correction (FEC) is only used if a disruption in the set of low latency compressed video frames would be more than a specified amount of time.

14. The method as in claim 11 further comprising:

protecting all audio packets generated by the video game using forward error correction (FEC).

15. The method as in claim 14 further comprising:

protecting the control signals received by the server from the client device using forward error correction (FEC).

16. A computer-implemented method for hosting online video games, performing video compression and streaming video over a network comprising:

receiving a control signal by a server, the control signal received from a client device over a communication channel to the server, the control signal being for executing an online video game by the server, wherein executing the online video game includes generating, encoding and transmitting a set of compressed video frames from the server to the client device;

determining a maximum data rate of the communication channel by sending an increasingly higher data rate data stream of the transmitted set of compressed video frames to the client device until at least one of a packet loss rate or an increase in latency indicates the maximum data rate of the communication channel has been exceeded;

detecting that the maximum data rate of the communication channel has been exceeded based on at least one of a detected increase in the packet loss rate or the increase in latency;

reducing a data rate of the set of compressed video frames to a reduced data rate, wherein the reduced data rate results in at least one of the packet loss rate or the latency to indicate an acceptable level; and

updating a compression process state that sets an amount of compression for the communication channel for the reduced data rate, wherein the compression process state is used to encode the set of compressed video frames;

while transmitting the set of compressed video frames, continuing to monitor when the maximum data rate for the communication channel has decreased based on measurements of at least one of an increase in the packet loss rate or said latency, and responsively reducing a data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency have reached an acceptable level; and

while transmitting the set of compressed video frames, continuing to monitor when the maximum data rate of the communication channel has increased based on continued measurements of at least one of a decrease in packet loss rate or said latency, and responsively increasing the data rate of the set of low latency compressed video frames until at least one of the packet loss rate or said latency reaches an unacceptable level;

wherein the compression process state is updated to adjust the amount of compression for responsively reducing or increasing the data rate;

wherein determining the maximum data rate includes receiving feedback from the client device for determining said packet loss rate or said latency.

17. The method as in claim 16, wherein the compression process state acts to modify a frame rate of the set compressed video frames.

18. The method as in claim 16, wherein the compression process state acts to modify a quality of the set of compressed video frames.

19. The method as in claim 16, wherein the compression process state acts to modify an image resolution of the set of compressed video frames.

20. The method as in claim 19, wherein when the image resolution is modified, the client device is configured to scale up the image to maintain the same image size on a display screen coupled to the client device.