System and method for multi-stream video compression

• US 8,366,552 B2
• Filed: 08/07/2009
• Issued: 02/05/2013
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): DEODHAR, OMKAR A
• Application Number:
  US12/538,041
• Publication Number:
  US 20100167816A1
• Time in Patent Office: 1,278 Days
• Field of Search: 463/31, 463/34, 463/40, 463/42, 463/43
• US Class Current: 463/42
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/43072 : of multiple content streams on the same device
• CPC Class Codes: H04N 21/439 : Processing of audio elementary streams
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method are described for online gaming. For example, a system according to one embodiment comprises: a video game server receiving user inputs related to an online video game and responsively executing program code of the video game to render a sequence of video images; a first stream encoder to compress the sequence of video images and generate a live video stream during a live gaming session with a user of a client device, the first stream encoder receiving channel feedback signals from the client device and responsively adapting compression of the sequence of video images based on the channel feedback signals, the first stream encoder continually transmitting the live video stream to the client device during the live gaming session with the user; a second stream encoder to compress the sequence of video images at a specified video quality and/or compression ratio unrelated to the channel feedback signal during the live gaming session with the user, thereby generating a High Quality (HQ) video stream, the HQ video stream having a relatively higher video quality and/or lower compression ratio than the live video stream; and a storage device for storing the HQ video stream for subsequent playback to the user of the client device and to other users upon request.

Claims

1. A computer-implemented system for online gaming comprising:

a video game or application server receiving user inputs related to an online video game or application from a client device over the Internet and responsively executing program code of the video game or application to render a sequence of video images;

a first stream encoder to compress the sequence of video images and generate a low latency video stream during a live gaming or application session with a user of the client device, the first stream encoder receiving channel feedback signals from the client device over the Internet and responsively adapting compression of the sequence of video images based on the channel feedback signals, the first stream encoder continually transmitting the low latency video stream to the client device over the Internet during the live gaming or application session with the user;

the client device receiving the live video stream over the Internet and rendering the sequence of video images as the user plays the video game or uses the application, wherein the video images are compressed and transmitted over the Internet in response to the user input with a latency such that the user has the perception that the video game or application is responding instantly to the user input;

a second stream encoder to compress the sequence of video images at a specified video quality and/or compression ratio unrelated to the channel feedback signal during the live gaming session with the user, thereby generating a High Quality (HQ) video stream, the HQ video stream having a relatively higher video quality and/or lower compression ratio than the low latency video stream; and

a storage device for storing the HQ video stream for subsequent playback to the user of the client device and to other users upon request.

2. The system as in claim 1 wherein the channel feedback signals indicate whether packets have been successfully received and/or decoded by the client device, wherein the first stream encoder dynamically adapts its compression upon a determination that one or more packets have not been successfully received and/or decoded.

3. The system as in claim 2 wherein dynamically adapting compression comprises generating an I-frame or I-tile.

4. The system as in claim 2 wherein dynamically adapting compression comprises generating a new compressed frame/tile to be dependent on a last frame/tile known to have been successfully received and/or decoded at the client device.

5. The system as in claim 4 wherein the new frame/tile is a P-frame/tile and the last frame/tile known to have been successfully received and/or decoded at the client device is a P-frame/tile.

6. The system as in claim 3 wherein the live stream encoder compresses the sequence of video images using P-frames/tiles until the channel feedback signals indicate that one or more packets have not been successfully received and/or decoded.

7. The system as in claim 1 wherein the first stream encoder adapts compression of the sequence of video images to reduce latency communicated in the channel feedback signals as the compressed sequence of video images are transmitted to the client device.

8. The system as in claim 1 wherein the specified video quality comprises a specified resolution at a specified frame rate.

9. The system as in claim 1 wherein the storage device comprises a delay buffer to temporarily buffer the HQ video stream prior to transmission to one or more users.

10. The system as in claim 1 further comprising:

scaling and recompression logic to scale and recompress the HQ video stream to a new resolution specified by a user.

11. The system as in claim 1 further comprising:

a plurality of additional stream encoders to compress the sequence of video images at a corresponding plurality of specified video quality levels and/or compression ratios unrelated to the channel feedback signal during the live gaming session with the user, thereby generating a plurality of video streams of different video quality levels and/or compression ratios.

12. The system as in claim 11 further comprising:

routing logic for selecting one or more of the HQ stream and the plurality of additional streams; and

scaling and/or recompression logic scaling and/or recompressing the selected HQ stream and/or plurality of additional streams at a specified resolution and/or size, respectively.

13. The system as in claim 1 further comprising:

multicast logic for multicasting the HQ stream to clients of one or more requesting users and/or to one or more other game servers upon request.

14. The system as in claim 1 further comprising:

routing logic to route the HQ streams directly to other game servers and/or clients of requesting users upon request and/or to read the HQ streams from the storage device after a delay and route the delayed HQ streams to other game servers and/or clients of requesting users upon request.

15. The system as in claim 1 further comprising:

mixing logic to mix audio with the HQ video stream; and

routing logic to route the mixed audio and HQ video stream to clients of users upon request.

16. The system as in claim 1 wherein forward error correction (FEC) encoding is used to protect the live video stream, an associated audio stream and/or user control input received from the client.

17. The system as in claim 1 further comprising mitigating channel bandwidth peaks by distributing large compressed video frames of the live video stream which would result in bandwidth peaks over a sequence of successive frame times.

18. The system as in claim 1 further comprising mitigating channel bandwidth peaks by transmitting compressed frames of the live video stream that have more data than can be transmitted at a peak data rate in one frame time over one or more additional frame times.

19. The system as in claim 1 further comprising dynamically adjusting a determined maximum data rate of live video stream in response to a current detected level of packet loss or latency.

20. The system as in claim 1 wherein the video images are compressed and transmitted over the Internet in response to the user input with a finite latency of 80 ms or less.

21. The system as in claim 1 wherein the video images are compressed and transmitted over the Internet in response to the user input with a finite latency of 150 ms or less.

39. The system as in claim 16 wherein FEC codes are used to protect some specified portions of the live video stream but not other portions.

40. The system as in claim 39 wherein FEC codes are used to protect data most likely to cause visual artifacts if lost.

41. The system as in claim 39 further comprising:

adjusting FEC encoding used to protect the portions of the live video stream in response to detecting that one or more video frames or portions of the live video stream have not been successfully received and/or decoded.

42. The system as in claim 18 wherein during said additional frame times a client continues to display the previously decompressed frame.

43. The system as in claim 42 further comprising ignoring frames received during said additional frame times.

44. The system as in claim 18 wherein the video frame rate temporarily decreases during the additional frame times.

45. The system as in claim 18 wherein the additional frame time is more than 1 frame time.

46. The system as in claim 42 wherein the data rate of the live video stream is adjusted by modifying the frame rate.

47. The system as in claim 42 wherein the data rate of the stream of first and/or second plurality of compressed video frames is adjusted by modifying the compression quality.

48. The system as in claim 42 wherein the data rate is adjusted by modifying the image resolution of the first and/or second plurality of compressed video frames.

49. The system as in claim 48 wherein when the image resolution is modified, the client scales up the image to maintain the same image size on a display screen.

22. A computer-implemented method for online gaming comprising:

receiving user inputs related to an online video game or application from a client device over the Internet and responsively executing program code of the video game or application to render a sequence of video images;

compressing the sequence of video images and generate a low latency video stream during a live gaming or application session with a user of the client device;

receiving channel feedback signals from the client device over the Internet and responsively adapting compression of the sequence of video images based on the channel feedback signals,

continually transmitting the low latency video stream to the client device over the Internet during the live gaming or application session with the user;

the client device receiving the live video stream over the Internet and rendering the sequence of video images as the user plays the video game or uses the application, wherein the video images are compressed and transmitted in response to the user input with a latency such that the user has the perception that the video game or application is responding instantly to the user input;

compressing the sequence of video images at a specified video quality and/or compression ratio unrelated to the channel feedback signal during the live gaming session with the user, thereby generating a High Quality (HQ) video stream, the HQ video stream having a relatively higher video quality and/or lower compression ratio than the low latency video stream; and

storing the HQ video stream in a storage device for subsequent playback to the user of the client device and to other users upon request.

23. The method as in claim 22 wherein the channel feedback signals indicate whether packets have been successfully received and/or decoded by the client device, the method further comprising dynamically adapting compression upon a determination that one or more packets have not been successfully received and/or decoded.

24. The method as in claim 23 wherein dynamically adapting compression comprises generating an I-frame or I-tile.

25. The method as in claim 23 wherein dynamically adapting compression comprises generating a new compressed frame/tile to be dependent on a last frame/tile known to have been successfully received and/or decoded at the client device.

26. The method as in claim 25 wherein the new frame/tile is a P-frame/tile and the last frame/tile known to have been successfully received and/or decoded at the client device is a P-frame/tile.

27. The method as in claim 24 wherein compressing the sequence of video images and generate a live video stream further comprises compressing the sequence of video images using P-frames/tiles until the channel feedback signals indicate that one or more packets have not been successfully received and/or decoded.

28. The method as in claim 22 wherein adapting compression further modifying compression of the sequence of video images to reduce latency communicated in the channel feedback signals as the compressed sequence of video images are transmitted to the client device.

29. The method as in claim 22 wherein the specified video quality comprises a specified resolution at a specified frame rate.

30. The method as in claim 22 wherein the storage device comprises a delay buffer to temporarily buffer the HQ video stream prior to transmission to one or more users.

31. The method as in claim 22 further comprising scaling and recompressing the HQ video stream to a new resolution specified by a user.

32. The method as in claim 22 further comprising compressing the sequence of video images at a corresponding plurality of specified video quality levels and/or compression ratios unrelated to the channel feedback signal during the live gaming session with the user, thereby generating a plurality of video streams of different video quality levels and/or compression ratios.

33. The method as in claim 32 further comprising:

selecting one or more of the HQ stream and the plurality of additional streams; and

scaling and/or recompressing the selected HQ stream and/or plurality of additional streams at a specified resolution and/or size, respectively.

34. The method as in claim 22 further comprising:

multicasting the HQ stream to clients of one or more requesting users and/or to one or more game servers upon request.

35. The method as in claim 22 further comprising:

routing the HQ streams directly to other game servers and/or clients of requesting users upon request and/or to read the HQ streams from the storage device after a delay and route the delayed HQ streams to other game servers and/or clients of requesting users upon request.

36. The method as in claim 22 further comprising:

mixing audio with the HQ video stream; and

routing the mixed audio and HQ video stream to clients of users upon request.

37. The method as in claim 22 wherein the video images are compressed and transmitted over the Internet in response to the user input with a finite latency of 80 ms or less.

38. The method as in claim 22 wherein the video images are compressed and transmitted over the Internet in response to the user input with a finite latency of 150 ms or less.