Video compression system and method for reducing the effects of packet loss over a communication channel

• US 8,769,594 B2
• Filed: 01/23/2009
• Issued: 07/01/2014
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): SHANG, ANNAN Q
• Application Number:
  US12/359,140
• Publication Number:
  US 20090225863A1
• Time in Patent Office: 1,985 Days
• Field of Search: 725 87- 98, 725109-116, 370/486, 370/487, 370/535, 370/536, 455/3.06, 455/17, 455/21, 455/23, 455/42, 455/102, 455/452.2, 455/500, 455/509, 463/1, 463/9, 463 30- 33, 463 40- 42, 709217-232
• US Class Current: 725/109
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/2385 Channel allocation H04N21/266 takes precedence; Bandwidth allocation H04N21/24 takes precedence ; allocation of channels according to the instantaneous demands of the users in time-division multiplex systems H04J3/1682; admission control, resource allocation in open networks H04L12/5692; arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896; negotiating bandwidth in wireless networks H04W28/16
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/8186 specially adapted to be executed by a peripheral of the client device, e.g. by a reprogrammable remote control
• CPC Class Codes: H04N 7/106 for domestic distribution

Abstract

A system and method are described below for reducing the effects of packet loss in a video communication system. For example, one embodiment of a computer-implemented method comprises: logically subdividing each of a sequence of images of a video stream into a plurality of tiles, each of the tiles having a defined position within each of the sequence of images; and packing the tiles into a plurality of data packets to maximize the number of tiles which are aligned with boundaries of each of the data packets; and transmitting the data packets containing the tiles over a communication channel from a server to a client.

Claims

1. A computer-implemented method for hosting low-latency streaming interactive audio/video (A/V) comprising:

executing one or more video games or applications on a server communicatively coupled to the Internet;

receiving packet streams from a plurality of users and routing the packets to the one or more of the video games or applications being executed, the packet streams including user control input, one or more of the video games or applications being operable to compute A/V data responsive to the user control input;

receiving the A/V data from the one or more video games or applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in low-latency streaming compressed A/V data; and

routing the low-latency streaming compressed A/V data to each of the users over a corresponding Internet communication channel, wherein the operations of executing the video games or applications, receiving packet streams and A/V data, compressing portions of the A/V data and routing the low latency streaming compressed A/V data are performed with a latency such that at least one user has the perception that the controlled video game or application is responding instantly.

2. The method as in claim 1 wherein the plurality of processing units comprise a plurality of cores of a single processor.

3. The method as in claim 1 wherein the plurality of processing units comprise a plurality of central processing units.

4. The method as in claim 3 wherein FEC codes are used to protect data most likely to cause visual artifacts if lost.

5. The method as in claim 1 further comprising:

generating forward error correction (FEC) data to protect the low-latency streaming compressed A/V data and/or the user control input.

6. The method as in claim 5 wherein FEC codes are used to protect some portions of the low-latency streaming compressed A/V data but not other portions.

7. The method as in claim 1 further comprising mitigating channel bandwidth peaks by distributing large compressed video bandwidth peaks of the low-latency streaming compressed A/V data over a sequence of successive video frames.

8. The method as in claim 7 wherein distributing large compressed video bandwidth peaks over a sequence of successive frames further comprises transmitting compressed frames that have more data than can be transmitted at a peak data rate in one frame time over one or more additional frame times.

9. The method as in claim 8 wherein during said additional frame times a client continues to display the previously decompressed frame.

10. The method as in claim 9 further comprising ignoring frames received during said additional frame times.

11. The method as in claim 7 wherein the video frame rate temporarily decreases during the additional frame times.

12. The method as in claim 7 wherein the additional frame time is 1 frame time.

13. The method as in claim 1 further comprising dynamically adjusting the data rate of the low-latency streaming compressed A/V data.

14. The method as in claim 13 wherein the data rate is adjusted in response to a detected maximum data rate of the channel.

15. The method as in claim 13 wherein the data rate is adjusted by modifying the frame rate.

16. The method as in claim 13 wherein the data rate is adjusted by modifying the quality of the compression of the low-latency streaming compressed A/V data.

17. The method as in claim 13 wherein the data rate is adjusted by modifying a resolution of images of the A/V data.

18. The method as in claim 17 wherein when the image resolution is modified, the client scales up the image to maintain the same image size on a display screen.

19. The method as in claim 1 wherein when a communication channel to a client channel completely drops out, one or more servers pause a game being played by the client.

20. The method as in claim 19 wherein if a client drops out and a user of the client is playing a multiplayer game, the other users are notified that the user has dropped out.

21. The method as in claim 20 wherein the game is paused for the other users when a user drops out.

22. The method as in claim 1 further comprising initially determining a peak data rate by sending an increasingly higher data rate test stream over the output routing network until packet loss and/or higher latency indicates channel capacity has been exceeded.

23. The method as in claim 22 further comprising:

determining that the channel capacity has decreased based on a detected increase in packet loss and/or latency increasing; and

responsively reducing the peak data rate until packet loss and/or latency have reached acceptable values.

24. The method as in claim 22 further comprising:

determining that the channel capacity may have increased based on continued measurements of packet loss and/or latency; and

responsively increasing the peak data rate until packet loss and/or latency reaches an unacceptable level.

25. The method as in claim 1 further comprising transmitting data packets of the low-latency streaming compressed A/V data in a different order than in which they are displayed.

26. The system as in claim 1 further comprising a finite latency of 80 ms or less.

27. The system as in claim 1 further comprising a finite latency of 150 ms or less.

28. A system comprising a memory for storing program code and one or more processors for processing the program code to cause the system to perform the operations of:

executing one or more video games or applications on a server communicatively coupled to the Internet;

receiving packet streams from a plurality of users and routing the packets to the one or more of the video games or applications being executed, the packet streams including user control input, one or more of the video games or applications being operable to compute A/V data responsive to the user control input;

receiving the A/V data from the one or more video games or applications;

compressing portions of the A/V data in parallel using one or more processing units resulting in low-latency streaming compressed A/V data; and

routing the low-latency streaming compressed A/V data to each of the users over a corresponding Internet communication channel, wherein the operations of executing the video games or applications, receiving packet streams and A/V data, compressing portions of the A/V/data and routing the low latency streaming compressed A/V data are performed with a latency such that at least one user has the perception that the controlled video game or application is responding instantly.

29. The system as in claim 28 wherein the plurality of processing units comprise a plurality of cores of a single processor.

30. The system as in claim 28 wherein the plurality of processing units comprise a plurality of central processing units.

31. The system as in claim 28 further comprising:

generating forward error correction (FEC) data to protect the low-latency streaming compressed A/V data and/or the user control input.

32. The system as in claim 31 wherein FEC codes are used to protect some portions of the low-latency streaming compressed A/V data but not other portions.

33. The system as in claim 30 wherein FEC codes are used to protect data most likely to cause visual artifacts if lost.

34. The system as in claim 28 further comprising mitigating channel bandwidth peaks by distributing large compressed video bandwidth peaks of the low-latency streaming compressed A/V data over a sequence of successive video frames.

35. The system as in claim 34 wherein distributing large compressed video bandwidth peaks over a sequence of successive frames further comprises transmitting compressed frames that have more data than can be transmitted at a peak data rate in one frame time over one or more additional frame times.

36. The system as in claim 35 wherein during said additional frame times a client continues to display the previously decompressed frame.

37. The system as in claim 35 wherein the game is paused for the other users when a user drops out.

38. The system as in claim 34 further comprising ignoring frames received during said additional frame times.

39. The system as in claim 34 wherein the video frame rate temporarily decreases during the additional frame times.

40. The system as in claim 34 wherein the additional frame time is 1 frame time.

41. The system as in claim 40 further comprising:

determining that the channel capacity has decreased based on a detected increase in packet loss and/or latency increasing; and

responsively reducing the peak data rate until packet loss and/or latency have reached acceptable values.

42. The system as in claim 40 further comprising:

determining that the channel capacity may have increased based on continued measurements of packet loss and/or latency; and

responsively increasing the peak data rate until packet loss and/or latency reaches an unacceptable level.

43. The system as in claim 28 further comprising dynamically adjusting the data rate of the low-latency streaming compressed A/V data.

44. The system as in claim 28 wherein the data rate is adjusted in response to a detected maximum data rate of the channel.

45. The system as in claim 28 wherein the data rate is adjusted by modifying the frame rate.

46. The system as in claim 28 wherein the data rate is adjusted by modifying the quality of the compression of the low-latency streaming compressed A/V data.

47. The system as in claim 28 wherein the data rate is adjusted by modifying a resolution of images of the A/V data.

48. The system as in claim 47 wherein when the image resolution is modified, the client scales up the image to maintain the same image size on a display screen.

49. The system as in claim 28 wherein when a communication channel to a client channel completely drops out, one or more servers pause a game being played by the client.

50. The system as in claim 49 wherein if a client drops out and a user of the client is playing a multiplayer game, the other users are notified that the user has dropped out.

51. The system as in claim 28 further comprising initially determining a peak data rate by sending an increasingly higher data rate test stream over the output routing network until packet loss and/or higher latency indicates channel capacity has been exceeded.

52. The system as in claim 28 further comprising transmitting data packets of the low-latency streaming compressed A/V data in a different order than in which they are displayed.

53. The system as in claim 28 further comprising a finite latency of 80 ms or less.

54. The system as in claim 28 further comprising a finite latency of 150 ms or less.

55. A system comprising:

a plurality of application servers for executing one or more video games or applications;

an inbound packet routing network receiving packet streams from users over the Internet and routing the packets to the one or more of the video games or applications, the packet streams including user control input, the video games or applications being operable to compute A/V data responsive to the user control input;

one or more video compression units receiving the A/V data from the video games or applications, compressing portions of the A/V data in parallel using a plurality of processing units resulting in low-latency streaming compressed A/V data and routing the low-latency streaming compressed A/V data to each of the users over a corresponding communication channel, wherein each user is provided with a dedicated low-latency streaming compressed A/V data stream representing video images of a video games or application being executed on the application servers on behalf of that user, wherein the operations of executing the video games or applications, computing A/V data responsive to the control signals, compressing portions of the A/V data in parallel, and routing the low-latency streaming compressed A/V data to each of the users are performed with a latency such that at least one user has the perception that the controlled video game or application is responding instantly.

56. The system as in claim 55 wherein the applications comprise video games and wherein the application servers execute the video games in response to the input packet streams received from each of the users.

57. The system as in claim 56 wherein at least one of the video games is a multi-player video game played by a plurality of the users, wherein each of the users is provided with a dedicated low-latency streaming compressed A/V data stream including video images representing that user'"'"'s video game output.

58. The system as in claim 55 further comprising an outbound routing network to route each of the dedicated low-latency streaming compressed A/V data streams to each of the users.

59. The system as in claim 55 further comprising a finite latency of 150 ms or less.

60. The system as in claim 55 further comprising a finite latency of 80 ms or less.