Tile-based system and method for compressing video

• US 8,953,675 B2
• Filed: 01/23/2009
• Issued: 02/10/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Laan, Roger van der, Perlman, Stephen G.
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US12/359,218
• Publication Number:
  US 20090220001A1
• Time in Patent Office: 2,209 Days
• Field of Search: 375/240.01
• US Class Current: 375/240.1
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/21 Server components or server architectures
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: logically subdividing each of a sequence of images into a plurality of tiles, each of the tiles having a defined position within each of the sequence of images, the defined position remaining the same between successive images; encoding one of the tiles at a first defined position in a first image of the sequence of images using a first compression format and encoding the reminder of the tiles in the first image using the second compression format; and encoding one of the tiles at a second defined position in a second image of the sequence of images using the first compression format and encoding the reminder of the tiles in the first image using the second compression format; wherein the second compression format is dependent on previously-encoded tiles encoded according to the first compression format and/or the second compression format.

Claims

1. A computer-implemented method comprising:

receiving at a server a request for playing a low latency video game or executing an application over a network, at least a portion of which includes public network components;

in response to the request, establishing a video game or application session;

generating control signals at a client in response to an input device of the client being actuated by a user as the user is playing the low latency video game or using the application;

transmitting the control signals from the client to the server;

receiving the control signals at the server indicating user input as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising sequences of video images of the video game or application;

detecting channel characteristics of a communication channel between the server and the client over the network;

encoding the video output using low latency compression on the server to generate a low latency sequence of compressed video images, the video output encoded at a bitrate or compression ratio based on the detected channel characteristics, wherein the low latency compression comprises;

subdividing each video image of a sequence of video images of the video output into a first set of one or more image portions and a second set of one or more image portions;

encoding image portions in the first set using a first type of encoding;

encoding image portions in the second set using a second type of encoding;

analyzing the sequences of video images to detect motion and responsively sequencing certain image portions with the first type of encoding more frequently than other image portions based on detected motion within the image portions;

wherein subdividing each video image into image portions and encoding the first set of image portions using the first encoding format and the second set of image portions using the second encoding format produces a more consistent bitrate for the video images and lower latency for the end user than if entire video images were compressed using the first type of encoding and the second type of encoding;

transmitting the low latency sequence of compressed video images from the server to the client over the network, the low latency sequence of compressed video images being decoded by a decoder on the client and rendered on a display of the client;

wherein the operations of generating and transmitting the control signals, receiving the control signals, executing the video game or application, encoding and transmitting the low latency sequence of compressed video images to the client over the network, and decoding and rendering the low latency video stream on a display of the client is performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the control signals.

2. The method as in claim 1 wherein

each of the image portions have a defined position within each of the sequence of images, the defined position remaining the same between successive images;

wherein the second type of encoding is dependent on previously-encoded image portions encoded according to the first type of encoding and/or the second type of encoding.

3. The method as in claim 2 wherein the second type of encoding comprises inter-coding of the image portions.

4. The method as in claim 1 further comprising:

dynamically adjusting a size of the image portions based on detected characteristics of each of the video images.

5. The method as in claim 4 wherein relatively smaller image portions are used to encode areas of the sequence of video images with relatively more scene complexity and/or motion than areas of the sequence of video images with relatively less scene complexity and/or motion.

6. The method as in claim 5 further comprising:

dynamically adjusting a size of the image portions based on detected characteristics of each of the video images.

7. The method as in claim 6 wherein relatively smaller image portions are used to encode areas of the sequence of video images with relatively more scene complexity and/or motion than areas of the sequence of video images with relatively less scene complexity and/or motion.

8. The method as in claim 1 further comprising:

dynamically adjusting the bitrate or compression ratio at which the video output is compressed and streamed to the client based on a detected change in the channel characteristics.

9. The method as in claim 8 wherein the bitrate or compression ratio is adjusted in response to a detected maximum capacity of the communication channel between the server and the client.

10. The method as in claim 9 further comprising:

determining the maximum capacity of the communication channel by sending an increasingly higher bitrate test stream over the Internet to the client until packet loss and/or higher latency indicates the maximum capacity has been exceeded.

11. The method as in claim 10 further comprising:

determining that the maximum channel capacity has decreased based on a detected increase in packet loss and/or latency increasing while the compressed video is being streamed to the client; and

responsively and dynamically reducing the bitrate or increasing the compression ratio until packet loss and/or latency have reached predetermined acceptable values.

12. The method as in claim 10 further comprising:

determining that the maximum channel capacity has increased based on continued measurements of packet loss and/or latency; and

responsively and dynamically increasing the bitrate or decreasing the compression ratio until packet loss and/or latency reaches a predetermined unacceptable level.

13. The method as in claim 8 wherein the bitrate or compression ratio is adjusted by modifying a frame rate of the video output.

14. The method as in claim 8 wherein the bitrate or compression ratio is adjusted by modifying quality of the compressed video.

15. The method as in claim 8 wherein the bitrate or compression ratio is adjusted by modifying image resolution of the compressed video.

16. The method as in claim 15 wherein when the image resolution is modified, the decoder on the client scales up the image to maintain the same image size on a display screen.

17. The method as in claim 1 wherein when a communication channel between the server and the client unexpectedly drops out, the server pauses the video game or application.

18. The method as in claim 17 wherein if the communication channel drops out and a first user of the client is playing a multiplayer game, the other users are notified that the user has dropped out.

19. The method as in claim 18 wherein the video game is paused for the other users when the first user drops out.

20. The method as in claim 1 wherein data packets of the low latency video stream are received by the client in a different order than in which they are displayed.

21. The method as in claim 1 further comprising:

employing forward error correction (FEC) techniques to protect specified portions of the low latency video stream and/or the control signals transmitted from the client.

22. The method as in claim 21 where the FEC techniques are only employed to the extent required based on detected maximum channel capacity of a communication channel between the client and the server.

23. The method as in claim 1 further comprising:

reducing latency by mitigating channel bandwidth peaks of a communication channel between the client and the server.

24. The method as in claim 1 wherein the operations of receiving control signals transmitted from the client, executing the video game or application, compressing the video output, streaming the low latency video stream and decoding the low latency video stream on the client is performed at a latency of less than 80 ms.

25. A computer-implemented system comprising:

a client for receiving input signals from a user indicating that the user wishes to play an online video game or execute an online application;

a server receiving a request from the client over a network, at least a portion of which includes public network components to play the online video game or execute the application;

the server, in response to the request, establishing a video game or application session with the client;

the client generating control signals in response to an input device of the client being actuated by the user as the user is playing the low latency video game or using the application;

the client transmitting the control signals to the server;

the server receiving control signals transmitted from the client or another client indicating user input to the client or other client as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising sequences of video images of the video game or application;

the server detecting channel characteristics of a communication channel between the server and the client over the network and encoding the video output using low latency compression on the server to generate a low latency sequence of compressed video images, the video output encoded at a bitrate or compression ratio based on the detected channel characteristics;

wherein the low latency compression comprises;

lay an online video game or execute an online application;

subdividing each video image of a sequence of video images of the video output into a first set of one or more image portions and a second set of one or more image portions;

encoding image portions in the first set using a first type of encoding;

encoding image portions in the second set using a second type of encoding;

analyzing the sequences of video images to detect motion and responsively sequencing certain image portions with the first type of encoding more frequently than other image portions based on detected motion within the image portions;

wherein subdividing each video image into image portions and encoding the first set of image portions using the first encoding format and the second set of image portions using the second encoding format produces a more consistent bitrate for the video images and lower latency for the end user than if entire video images were compressed using the first type of encoding and the second type of encoding;

the server transmitting the low latency sequence of compressed video images from the server to the client over the network, the low latency sequence of compressed video images being decoded by a decoder on the client and rendered on a display of the client;

wherein the operations of generating and transmitting the control signals, receiving the control signals, executing the video game or application, encoding and transmitting the low latency sequence of compressed video images to the client over the network, and decoding and rendering the low latency video stream on a display of the client is performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the control signals.

26. The system as in claim 25 wherein:

each of the image portions have a defined position within each of the sequence of video images, the defined position remaining the same between successive images, wherein the second type of encoding is dependent on previously-encoded image portions encoded according to the first type of encoding and/or the second type of encoding.

27. The system as in claim 26 wherein the second type of encoding comprises intercoding of image portions.

28. The system as in claim 25 wherein the server performs the additional operations of:

dynamically adjusting the bitrate or compression ratio at which the video output is compressed and streamed to the client based on a detected change in the channel characteristics.

29. The system as in claim 28 wherein the bitrate or compression ratio is adjusted in response to a detected maximum capacity of the communication channel between the server and the client.

30. The system as in claim 28 wherein the bitrate or compression ratio is adjusted by modifying a frame rate of the video output.

31. The system as in claim 28 wherein the bitrate or compression ratio is adjusted by modifying quality of the compressed video.

32. The system as in claim 28 wherein the bitrate or compression ratio is adjusted by modifying image resolution of the compressed video.

33. The system as in claim 32 wherein when the image resolution is modified, the decoder on the client scales up the image to maintain the same image size on a display screen.

34. The system as in claim 28 wherein the server performs the additional operations of:

determining the maximum capacity of the communication channel by sending an increasingly higher bitrate test stream over the Internet to the client until packet loss and/or higher latency indicates the maximum capacity has been exceeded.

35. The system as in claim 34 wherein the server performs the additional operations of:

determining that the maximum channel capacity has decreased based on a detected increase in packet loss and/or latency increasing while the compressed video is being streamed to the client; and

responsively and dynamically reducing the bitrate or increasing the compression ratio until packet loss and/or latency have reached predetermined acceptable values.

36. The system as in claim 34 wherein the server performs the additional operations of:

determining that the maximum channel capacity has increased based on continued measurements of packet loss and/or latency; and

responsively and dynamically increasing the bitrate or decreasing the compression ratio until packet loss and/or latency reaches a predetermined unacceptable level.

37. The system as in claim 25 wherein when a communication channel between the server and the client unexpectedly drops out, the server pauses the video game or application.

38. The system as in claim 37 wherein if the communication channel drops out and a first user of the client is playing a multiplayer game, the other users are notified that the user has dropped out.

39. The system as in claim 38 wherein the video game is paused for the other users when the first user drops out.

40. The system as in claim 25 wherein data packets of the low latency video stream are received by the client in a different order than in which they are displayed.

41. The system as in claim 25 wherein forward error correction (FEC) techniques are employed on the server to protect specified portions of the low latency video stream and/or FEC techniques are performed on the client to protect specified portions of the control signals transmitted from the client.

42. The system as in claim 41 where the FEC techniques are only employed to the extent required based on detected maximum channel capacity of a communication channel between the client and the server.

43. The system as in claim 25 wherein the server performs the additional operations of:

reducing latency by mitigating channel bandwidth peaks of a communication channel between the client and the server.

44. The system as in claim 25 wherein the operations of receiving control signals transmitted from the client, executing the video game or application, compressing the video output, streaming the low latency video stream and decoding the low latency video stream on the client is performed at a latency of less than 80 ms.