Video compression system and method for reducing the effects of packet loss over a communication channel

• US 10,058,778 B2
• Filed: 09/18/2015
• Issued: 08/28/2018
• Est. Priority Date: 12/05/2007
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Laneau, Ronald
• Application Number:
  US14/858,689
• Publication Number:
  US 20160014437A1
• Time in Patent Office: 1,075 Days
• Field of Search: None
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2408 Monitoring of the upstream path of the transmission network, e.g. client requests monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8186 specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
• CPC Class Codes: H04N 7/106 for domestic distribution

Abstract

Methods for hosting low-latency streaming interactive audio/video (A/V) include executing one or more video games or applications on a server communicatively coupled to a data network. Packet streams are received from a plurality of users and routed to the one or more video games. The packet streams include user control input that are used to compute A/V data in response. The A/V data are received from the video games or applications. Portions of the A/V data are compressed in parallel using processing units resulting in low-latency streaming compressed A/V data. The low-latency streaming compressed A/V data are routed to each of the users over a corresponding data network communication channel. The executing of video games, receiving of packet streams and A/V data, compressing portions of the A/V data and routing the compressed A/V data are performed with a latency such that at least one user has the perception that the controlled video game is responding instantly.

Claims

1. A computer-implemented method for hosting low-latency streaming interactive audio/video (A/V) comprising:

executing one or more video games or applications on a server communicatively coupled to a data network;

receiving packet streams from a plurality of users and routing the packets to the one or more of the video games or applications being executed, the packet streams including user control input, one or more of the video games or applications being operable to compute A/V data responsive to the user control input;

receiving the A/V data from the one or more video games or applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in low-latency streaming compressed A/V data; and

routing the low-latency streaming compressed A/V data to each of the users over a corresponding data network communication channel,

wherein operations of executing the video games or applications, receiving packet streams and A/V data, compressing portions of the A/V data and routing the low latency streaming compressed A/V data are performed with a latency such that at least one user has the perception that a controlled video game or application is responding instantly, wherein routing further includes,

initially determining a peak data rate by sending an increasingly higher data rate test stream over an output routing network until packet loss and/or higher latency indicates channel capacity of the data network communication channel has been exceeded;

determining that the channel capacity has been exceeded based on a detected increase in packet loss and/or latency; and

responsively adjusting the peak data rate until packet loss and/or latency have reached acceptable level.

2. The method as in claim 1 further comprising:

generating forward error correction (FEC) codes to protect the low-latency streaming compressed A/V data and/or the user control input, wherein the FEC codes are used to protect some portions of the low-latency streaming compressed A/V data but not other portions.

3. The method as in claim 2, wherein the FEC codes are used to protect low-latency streaming compressed A/V data that are most likely to cause visual artifacts if lost.

4. The method as in claim 1 further comprising mitigating data network communication channel bandwidth peaks by distributing large compressed video bandwidth peaks of the low-latency streaming compressed A/V data over a sequence of successive video frames.

5. The method as in claim 4, wherein distributing large compressed video bandwidth peaks over a sequence of successive frames further comprises transmitting compressed frames that have more data than can be transmitted at a peak data rate in one frame time over one or more additional frame times.

6. The method as in claim 5, wherein during said additional frame times a client continues to display the previously decompressed frame and wherein during said additional frame times, video frame rate of low-latency streaming compressed A/V data temporarily decreases.

7. The method as in claim 6, further comprising ignoring frames received during said additional frame times.

8. The method as in claim 1, further comprising dynamically adjusting data rate of the low-latency streaming compressed A/V data, wherein the data rate is adjusted in response to a detected maximum data rate of the data network communication channel.

9. The method as in claim 8, wherein the data rate is adjusted by modifying the frame rate or by modifying quality of the compression of the low-latency streaming compressed A/V data or by modifying a resolution of image of the A/V data.

10. The method as in claim 9, wherein when the resolution of the image is modified, a client is configured to scale up the image to maintain a same image size on a display screen of the client.

11. The method as in claim 1, wherein when the data network communication channel to a client completely drops out, one or more servers pause the video game or the application being used by the client.

12. The method as in claim 11, wherein if the client drops out when a user of the client is playing a multiplayer video game, pausing the multiplayer game for other users and notifying the other users that the user has dropped out.

13. The method as in claim 1, further comprising,

determining that the channel capacity has decreased based on a detected increase in packet loss and/or latency;

and

responsively reducing the peak data rate until packet loss and/or latency have reached an acceptable level.

14. The method as in claim 1, further comprising,

determining that the channel capacity may have increased based on continued measurements of packet loss and/or latency;

and

responsively increasing the peak data rate until packet loss and/or latency reaches an unacceptable level.

15. A computer-implemented method for hosting low-latency streaming interactive audio/video (A/V) comprising:

executing one or more video games or applications on a server communicatively coupled to a data network;

receiving packet streams from a plurality of users and routing the packets to the one or more of the video games or applications being executed, the packet streams including user control input, one or more of the video games or applications being operable to compute A/V data responsive to the user control input;

receiving the A/V data from the one or more video games or applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in low-latency streaming compressed A/V data;

routing the low-latency streaming compressed A/V data to each of the users over a corresponding data network communication channel, wherein the operations of executing the video games or applications, receiving packet streams and A/V data, compressing portions of the A/V data and routing the low latency streaming compressed A/V data are performed with a latency such that at least one user has a perception that a corresponding controlled video game or application is responding instantly, wherein the routing further includes,

dynamically adjusting data rate of the low-latency streaming compressed A/V data, the data rate is adjusted by modifying a resolution of images of the A/V data, wherein when the resolution of the images is modified, a client is configured to scale up the images to maintain a same image size on a display screen of the client.

16. The method as in claim 15, further comprising initially determining a peak data rate by sending an increasingly higher data rate test stream over an output routing network until packet loss and/or higher latency indicates channel capacity of the data network communication channel has been exceeded.

17. The method as in claim 16, further comprising:

determining that the channel capacity has decreased based on a detected increase in packet loss and/or latency; and

responsively reducing the peak data rate until packet loss and/or latency have reached an acceptable level.

18. The method as in claim 16, further comprising:

determining that the channel capacity may have increased based on continued measurements of packet loss and/or latency; and

responsively increasing the peak data rate until packet loss and/or latency reaches an unacceptable level.

19. The method as in claim 15, wherein routing further includes transmitting data packets of the low-latency streaming compressed A/V data in a different order than in which they are displayed.

20. A computer-implemented method for hosting low-latency streaming interactive audio/video (A/V) comprising:

executing one or more video games or applications on a server communicatively coupled to a data network;

receiving packet streams from a plurality of users and routing the packets to the one or more of the video games or applications being executed, the packet streams including user control input, one or more of the video games or applications being operable to compute A/V data responsive to the user control input;

receiving the A/V data from the one or more video games or applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in low-latency streaming compressed A/V data;

routing the low-latency streaming compressed A/V data to each of the users over a corresponding data network communication channel, wherein the operations of executing the video games or applications, receiving packet streams and A/V data, compressing portions of the A/V data and routing the low latency streaming compressed A/V data are performed with a latency such that at least one user has a perception that a corresponding controlled video game or application is responding instantly, wherein the routing further includes,

dynamically adjusting data rate of the low-latency streaming compressed A/V data, the data rate is adjusted by modifying a frame rate in response to a detected maximum data rate of the data network communication channel.

21. The method of claim 20, further comprising adjusting the data rate by modifying a quality of the compression of the low-latency streaming compressed A/V data.

22. The method as in claim 20, further comprising adjusting the data rate by modifying a resolution of images of the A/V data.

23. The method as in claim 22, wherein when the resolution of images is modified, a client is configured to scale up the images to maintain a same image size on a display screen of the client.

24. The method as in claim 20, wherein when a communication channel to a client completely drops out, one or more servers pause a multiplayer video game being played by the client.

25. The method as in claim 24, wherein if the client drops out when a user of the client is playing the multiplayer game, other users of the multiplayer video game are notified that the user has dropped out and the multiplayer game is paused for the other users when the user drops out.

26. The method as in claim 20, further comprising transmitting data packets of the low-latency streaming compressed A/V data in a different order than in which they are displayed.