System and method for compressing video by allocating bits to image tiles based on detected intraframe motion or scene complexity

• US 9,155,962 B2
• Filed: 01/23/2009
• Issued: 10/13/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: van der Laan, Roger, Perlman, Stephen G.
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/359,227
• Publication Number:
  US 20090213935A1
• Time in Patent Office: 2,454 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 21/21 Server components or server architectures
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A system and method are described below for encoding interactive low-latency video using interframe coding. For example, one embodiment of a computer-implemented method for performing video compression comprises: logically subdividing each of a sequence of images into a plurality of tiles, each of the tiles having a defined position within each of the sequence of images, the defined position remaining the same between successive images; detecting motion or high scene complexity within the sequence of images occurring at each of the positions of each of the tiles; and encoding each tile within each image of the sequence of images using a specified number of bits, the number of bits selected based on the detected amount of motion at the position of each tile across the sequence of images.

Claims

1. A computer-implemented method for hosting online applications or video games, performing video compression, and streaming video comprising:

receiving a request from a user for playing a low latency video game or executing an application;

generating control signals from a client in response to an input device of the client being actuated by the user as the user is playing the low latency video game or using the application;

receiving the control signals, the control signals transmitted over a network, at least a portion of which comprises a public network, as the user is playing the low latency video game or using the application;

routing the control signals to a server on which the requested video game or application is being executed;

responsively executing the low latency video game or application on the server to generate video output comprising a sequence of images of the low latency video game or application;

compressing the sequence of images of the low latency video game or application to generate a compressed video stream, wherein compressing the sequence of images comprises the operations of;

subdividing each of a sequence of images into a plurality of image portions;

detecting motion across the sequence of images occurring at positions of each of the image portions;

allocating the plurality of image portions to a first set of one or more image portions or to a second set of one or more image portions, wherein the frequency with which an image portion is allocated to the first set is based on the detected amount of motion at the position of each image portion across the sequence of images and a number of images that have been compressed since a particular image portion was in the first set of image portions;

encoding image portions in the first set using a first type of encoding;

encoding image portions in the second set using a second type of encoding; and

wherein each image portion within each image of the sequence of images is encoded using a number of bits selected based on the detected amount of motion at the position of each image portion across the sequence of images;

transmitting the compressed video stream to the client over the network,

decompressing the compressed video stream on the client;

rendering the sequence of video images on a display of the client;

wherein the operations of generating the control signals, receiving the control signals transmitted over the network, routing the control signals to a server, executing the low latency video game or application on the server to generate the sequence of images, compressing the sequence of images, transmitting the compressed video stream, decompressing the compressed video stream on the client, and rendering the sequence of video images on the display are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the control signals;

receiving feedback information from the client over the network, the feedback information usable to determine at least one characteristic of a communication channel between the server and the client; and

in response to determining the at least one characteristic of the communication channel, then adjusting the compression of the video images while maintaining the latency such that the user has the perception that the selected video game or application is responding instantly to the control signals, wherein adjusting the compression comprises;

adjusting the frequency with which an image portion is allocated to the first set based on the at least one characteristic of the communication channel; and

adjusting the number of bits selected to encode each image portion based on the at least one characteristic of the communication channel.

2. The method as in claim 1 wherein, for each image being compressed, each image portion of the image is compressed using either intraframe coding or interframe coding.

3. The method as in claim 2 wherein the image portions coded using intraframe coding in a first image in the sequence of images are coded using interframe coding in a second image in the sequence of images and image portions coded using interframe coding in a first image in the sequence of images are coded using intraframe coding in a second image in the sequence of images.

4. The method as in claim 2 wherein the feedback from the client indicates whether one or more images or portions thereof have not been successfully received and/or decoded, wherein:

(1) if the current channel characteristics indicate a latency above a specified threshold, then encoding a new video image or portion thereof which is dependent on previously transmitted video image or portions thereof; and

(2) if the current channel characteristics indicate a latency below a specified threshold, then encoding a new video image or portion thereof to be independent from previously transmitted video image or portions thereof.

5. The method as in claim 1 wherein forward error correction (FEC) encoding is used to protect portions of the compressed video stream, an associated audio stream and/or user control input received from the client.

6. The method as in claim 5 wherein FEC codes are used to protect some specified portions of the compressed video stream but not other portions.

7. The method as in claim 6 wherein FEC codes are used to protect data most likely to cause visual artifacts if lost.

8. The method as in claim 6 further comprising:

adjusting FEC encoding used to protect the portions of the compressed video stream in response to detecting that one or more compressed images or portions thereof have not been successfully received and/or decoded.

9. The method as in claim 1 further comprising mitigating channel bandwidth peaks by distributing large compressed video images which would result in bandwidth peaks over a sequence of successive video image times.

10. The method as in claim 1 further comprising mitigating channel bandwidth peaks by transmitting compressed images that have more data than can be transmitted at a peak data rate in one image time over one or more additional image times.

11. The method as in claim 10 wherein during said additional image times a client continues to display the previously decompressed image.

12. The method as in claim 10 further comprising ignoring images received during said additional image times.

13. The method as in claim 10 wherein the video image rate temporarily decreases during the additional image times.

14. The method as in claim 10 wherein the additional image time is more than 1 image time.

15. The method as in claim 1 further comprising dynamically adjusting a determined maximum data rate of the first and/or second plurality of compressed video images in response to a current detected level of packet loss or latency.

16. The method as in claim 11 wherein the data rate of the stream of first and/or second plurality of compressed video images is adjusted by modifying the image rate.

17. The method as in claim 11 wherein the data rate of the stream of first and/or second plurality of compressed video images is adjusted by modifying the compression quality.

18. The method as in claim 11 wherein the data rate is adjusted by modifying the image resolution of the first and/or second plurality of compressed video images.

19. The method as in claim 18 wherein when the image resolution is modified, the client scales up the image to maintain the same image size on a display screen.

20. The method as in claim 1 wherein the operations of receiving control signals transmitted from the client device over the Internet, executing the video game or application, compressing the sequence of video images, streaming the compressed video stream to the client, and decompressing the compressed video stream are performed with a finite latency of 80 ms or less.

21. The method as in claim 1 wherein the operations of receiving control signals transmitted from the client device over the Internet, executing the video game or application, compressing the sequence of video images, streaming the compressed video stream to the client, and decompressing the compressed video stream are performed with a finite latency of 150 ms or less.