Video compression system and method for reducing the effects of packet loss over a communication channel

• US 10,456,671 B2
• Filed: 08/28/2018
• Issued: 10/29/2019
• Est. Priority Date: 12/05/2007
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Laneau, Ronald
• Application Number:
  US16/115,503
• Publication Number:
  US 20190060747A1
• Time in Patent Office: 427 Days
• Field of Search: None
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2408 Monitoring of the upstream path of the transmission network, e.g. client requests monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8186 specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
• CPC Class Codes: H04N 7/106 for domestic distribution

Abstract

Methods for hosting low-latency streaming interactive audio/video (A/V) include executing one or more video games or applications on a server communicatively coupled to a data network. Packet streams are received from a plurality of users and routed to the one or more video games. The packet streams include user control input that are used to compute A/V data in response. The A/V data are received from the video games or applications. Portions of the A/V data are compressed in parallel using processing units resulting in low-latency streaming compressed A/V data. The low-latency streaming compressed A/V data are routed to each of the users over a corresponding data network communication channel. The executing of video games, receiving of packet streams and A/V data, compressing portions of the A/V data and routing the compressed A/V data are performed with a latency such that at least one user has the perception that the controlled video game is responding instantly.

Claims

1. A computer-implemented method for streaming interactive audio/video (A/V) data comprising:

executing one or more applications on a computer communicatively coupled to a network;

receiving packet data from client devices of a plurality of users and routing the packet data to the corresponding one or more of the applications being executed, the packet data including user control input generated at the client devices coupled to the network, one or more of the applications being operable to compute the A/V data responsive to the user control input;

receiving the A/V data from the one or more applications;

compressing portions of the A/V data in parallel using one or more processing units for generating compressed A/V data;

routing the compressed A/V data to client devices of the plurality of users over corresponding communication channels,

wherein the routing includes,

determining channel bandwidth of a communication channel used to communicate with a client device of a user of the plurality of users, the channel bandwidth determined by monitoring an amount of data loss and/or latency for a test stream transmitted to the client device; and

streaming the compressed A/V data to the client device over the network, wherein a data rate of the streaming of the compressed A/V data to the client device is set based on the channel bandwidth determined for the communication channel used for communicating with the client device.

2. The method of claim 1, wherein determining channel bandwidth includes,

sending the test stream at an increasingly higher data rate of transmission, over the network to the client device using the communication channel;

monitoring feedback from the client device to determine when the data rate of transmission has exceeded the channel bandwidth, wherein the exceeding of the channel bandwidth is determined by an increase in data loss and/or latency at the client device as detected from the feedback; and

responsively adjusting the data rate of transmission of the test stream, so as to maintain the data loss and/or latency at the client device at an acceptable level, the adjusted data rate representing the channel bandwidth of the communication channel.

3. The method of claim 1, wherein the computer is part of a hosting service that includes one or more game servers executing the one or more applications.

4. The method of claim 1, wherein the communication channel includes an inbound routing to receive the packet data from the client devices and an outbound routing to transmit the compressed A/V data to the client devices.

5. The method as in claim 1, further comprising,

monitoring the channel bandwidth of the communication channel during streaming of the compressed A/V data to determine that the channel bandwidth of the communication channel has decreased based on a detected increase in data loss and/or latency;

and

responsively reducing the data rate of transmission for the communication channel until data loss and/or latency at the client device reaches an acceptable level.

6. The method as in claim 1, further comprising,

monitoring the channel bandwidth of the communication channel during streaming of the compressed A/V data to determine that the channel bandwidth of the communication channel has increased based on continued measurements of data loss and/or latency;

and

responsively increasing the data rate of transmission for the communication channel until data loss and/or latency at the client device reaches an unacceptable level.

7. The method of claim 1, wherein the data rate is adjusted in response to a detected maximum data rate of the communication channel.

8. The method of claim 1, wherein the data rate is adjusted by modifying frame rate or by modifying quality of the compression of the compressed A/V data or by modifying a resolution of image of the A/V data,

wherein when the resolution of the image to the client device is modified, the client device is configured to scale up the image to maintain a same image size on a display screen of the client.

9. The method of claim 1, wherein the one or more applications are video games, wherein one or more of the video games are multiplayer video games.

10. A computer-implemented method for streaming interactive audio/video (A/V) data comprising:

executing one or more applications on a computer communicatively coupled to a network;

receiving packet data from client devices of a plurality of users and routing the packet data to the corresponding one or more of the applications being executed, the packet data including user control input generated at the client devices coupled to the network, one or more of the applications being operable to compute the A/V data responsive to the user control input;

receiving the A/V data from the one or more applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in compressed A/V data;

routing the compressed A/V data to client devices of the plurality of users over corresponding communication channels, wherein the routing includes,

dynamically adjusting a data rate of the compressed A/V data streaming through a communication channel to a client device associated with a user of the plurality of users, the adjusting of the data rate is based on channel bandwidth determined for the communication channel and is performed by modifying a resolution of images of the A/V data, wherein when the resolution of the images is modified, the client device associated with the user is configured to scale up the images to maintain a same image size on a display screen of the client device.

11. The method as in claim 10, wherein channel bandwidth is determined by,

sending a test stream at an increasingly higher data rate of transmission, over the network to the client device using the communication channel;

monitoring feedback from the client device to determine when the data rate of transmission has exceeded the channel bandwidth of the communication channel, the exceeding of the channel bandwidth determined by an increase in data loss and/or latency at the client device as determined from the feedback; and

responsively adjusting the data rate of transmission of the test stream, so as to maintain the data loss and/or latency at the client device at an acceptable level, the adjusted data rate representing the channel bandwidth of the communication channel.

12. The method as in claim 11, further comprising:

monitoring the channel bandwidth of the communication channel during streaming of the compressed A/V data to determine that the channel bandwidth of the communication channel has decreased based on a detected increase in data loss and/or latency; and

responsively reducing the data rate for the communication channel until data loss and/or latency have reached an acceptable level.

13. The method as in claim 11, further comprising:

monitoring the channel bandwidth of the communication channel during streaming of the compressed A/V data to determine that the channel bandwidth of the communication channel has increased based on continued measurements of data loss and/or latency; and

responsively increasing the data rate for the communication channel until data loss and/or latency reaches an unacceptable level.

14. The method as in claim 10, wherein routing further includes transmitting the compressed A/V data in a different order than in which they are displayed.

15. A computer-implemented method for streaming interactive audio/video (A/V) data comprising:

executing one or more video games on a computer communicatively coupled to a network;

receiving packet data from client devices of a plurality of users and routing the packet data to the corresponding one or more of the video games being executed, the packet data including user control input generated at the client devices coupled to the network, one or more of the video games being operable to compute the A/V data responsive to the user control input;

receiving the A/V data from the one or more video games;

compressing portions of the A/V data in parallel using one or more processing units of the computer to generate compressed A/V data;

routing the compressed A/V data to client devices of the plurality of users over corresponding communication channels, wherein the routing includes,

dynamically adjusting a data rate of the compressed A/V data transmitted over a communication channel to a client device associated with a user of the plurality of users, the adjusting of the data rate is based on channel bandwidth determined for the communication channel and is performed by modifying a frame rate of the A/V data, in response to detecting maximum data rate representing channel bandwidth of the communication channel.

16. The method of claim 15, wherein detecting maximum data rate includes,

sending a test stream at an increasingly higher data rate of transmission, over the network, to the client device using the communication channel;

monitoring feedback from the client device to determine when the data rate of transmission has exceeded the channel bandwidth of the communication channel, the exceeding of the channel bandwidth determined by an increase in data loss and/or latency at the client device as determined from the feedback; and

responsively adjusting the data rate of transmission of the test stream, so as to maintain the data loss and/or latency at the client device at an acceptable level, the adjusted data rate defining the maximum data rate representing the channel bandwidth of the communication channel.

17. The method of claim 15, further comprising adjusting the data rate by modifying a quality of compression of the compressed A/V data.

18. The method as in claim 15, wherein the one or more video games are multiplayer video games, and wherein when the user of the client device used for playing a particular multiplayer video game drops out, one or more servers pause the particular multiplayer video game being played by the client device.

19. The method as in claim 18, wherein if the user of the client device exits the particular multiplayer video game, other users of the particular multiplayer video game are notified that the user has dropped out and the multiplayer video game is paused for the other users, in response to the user exiting.

20. The method as in claim 15, further comprising streaming the compressed A/V data in a different order than in which they are displayed.