System and method for multi-stream video compression using multiple encoding formats

• US 8,964,830 B2
• Filed: 08/07/2009
• Issued: 02/24/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US12/538,092
• Publication Number:
  US 20100166068A1
• Time in Patent Office: 2,027 Days
• Field of Search: 348/390.1, 348/387.1, 348/400, 37524001-24026, 345/161, 345/156, 345/173, 345/163, 463/42, 463/31, 463/30, 463/43, 463/1
• US Class Current: 375/240.01
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/5593 involving scheduling aspects
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/231 : Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method are described for performing video compression. For example, a method according to one embodiment comprises: encoding a plurality of video frames or portions thereof according to a first encoding format, the first encoding format being optimized for transmission to a client device over a current communication channel; transmitting the plurality of encoded video frames or portions to the client device over the communication channel; concurrently encoding the first plurality of video frames according to a second encoding format, the second encoding format having a relatively higher-quality compressed video and/or a lower compression ratio than the first encoding format; storing the first plurality of video frames encoded in the second encoding format on a storage device; and making the first plurality of video frames encoded in the second encoding format available to the client device and other client devices for playback of the video stream.

Claims

1. A computer-implemented method for implementing online video games or applications over a network including at least one public network component comprising:

generating control signals from a client in response to an input device of the client being actuated by a user as the user is playing a low latency video game or application;

receiving the control signals related to the low latency video game or application transmitted from the client operated by the user over the network which includes at least one public network component;

routing the control signals to a server and responsively executing program code of the low latency video game or application to generate a plurality of video frames resulting from execution of the video game or application;

encoding the plurality of video frames or portions thereof according to a first encoding format to generate a plurality of encoded video frames or portions, the first encoding format being optimized for low latency transmission to a client device over a current network communication channel including the at least one public network component;

transmitting the plurality of encoded video frames or portions to the client device over the network communication channel,

the client device decoding the video frames and rendering the video on a display,

wherein the operations of generating control signals, receiving the control signals, routing the control signals to the server, responsively executing the program code of the low latency video game or application to generate the plurality of video frames, encoding the plurality of video frames according to the first encoding format optimized for low latency transmission to generate the plurality of encoded video frames, transmitting the plurality of encoded video frames, decoding the video frames on the client and rendering the video on the display are performed with a latency such that the user has the perception that the video game or application is responding instantly to the user input;

concurrently encoding the first plurality of video frames according to a second encoding format, the second encoding format having a relatively higher-quality compressed video and/or a lower compression ratio than the first encoding format;

storing the first plurality of video frames encoded in the second encoding format on a storage device;

making the first plurality of video frames encoded in the second encoding format available to the client device and other client devices for playback of the video stream; and

responsively transmitting the first plurality of video frames encoded in the second encoding format to the client and/or other clients upon receiving a request from a user.

2. The method as in claim 1 wherein the first encoding format comprises frames which are dependent on other frames and the second encoding format comprises some frames which are not dependent on other frames.

3. The method as in claim 2 wherein the second encoding format includes I-Frames and wherein the first encoding format does not include I-Frames.

4. The method as in claim 3 wherein the first encoding format comprises only P-Frames.

5. The method as in claim 1 wherein the first plurality of video frames are frames of a video game, generated as a user of the client device plays an online video game and wherein the first encoding format is selected to reduce latency as the frames compressed according to the first encoding format are transmitted to the client device over the current communication channel.

6. The method as in claim 5 wherein the second encoding format is selected to archive the video frames of the online video game, the video frames to be made available for subsequent playback from the archive.

19. The method as in claim 1 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 80 ms or less.

20. The method as in claim 1 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 150 ms or less.

7. A computer system having a memory for storing program code and a processor for processing the program code to perform the operations of:

generating control signals from a client in response to an input device of the client being actuated by a user as the user is playing a low latency video game or application;

receiving the control signals related to the low latency video game or application transmitted from the client operated by the user over the network which includes at least one public network component;

routing the control signals to a server and responsively executing program code of the low latency video game or application to generate a plurality of video frames resulting from execution of the video game or application;

encoding the plurality of video frames or portions thereof according to a first encoding format to generate a plurality of encoded video frames or portions, the first encoding format being optimized for low latency transmission to a client device over a current network communication channel including the at least one public network component;

transmitting the plurality of encoded video frames or portions to the client device over the network communication channel,

the client device decoding the video frames and rendering the video on a display,

wherein the operations of generating control signals, receiving the control signals, routing the control signals to the server, responsively executing the program code of the low latency video game or application to generate the plurality of video frames, encoding the plurality of video frames according to the first encoding format optimized for low latency transmission to generate the plurality of encoded video frames, transmitting the plurality of encoded video frames, decoding the video frames on the client and rendering the video on the display are performed with a latency such that the user has the perception that the video game or application is responding instantly to the user input;

concurrently encoding the first plurality of video frames according to a second encoding format, the second encoding format having a relatively higher-quality compressed video and/or a lower compression ratio than the first encoding format;

storing the first plurality of video frames encoded in the second encoding format on a storage device;

making the first plurality of video frames encoded in the second encoding format available to the client device and other client devices for playback of the video stream; and

responsively transmitting the first plurality of video frames encoded in the second encoding format to the client and/or other clients upon receiving a request from a user.

8. The computer system as in claim 7 wherein the first encoding format comprises frames which are dependent on other frames and the second encoding format comprises some frames which are not dependent on other frames.

9. The computer system as in claim 8 wherein the second encoding format includes I-Frames and wherein the first encoding format does not include I-Frames.

10. The computer system as in claim 9 wherein the first encoding format comprises only P-Frames.

11. The computer system as in claim 7 wherein the first plurality of video frames are frames of a video game, generated as a user of the client device plays an online video game and wherein the first encoding format is selected to reduce latency as the frames compressed according to the first encoding format are transmitted to the client device over the current communication channel.

12. The computer system as in claim 11 wherein the second encoding format is selected to archive the video frames of the online video game, the video frames to be made available for subsequent playback from the archive.

21. The system as in claim 7 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 80 ms or less.

22. The system as in claim 7 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 150 ms or less.

13. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:

generating control signals from a client in response to an input device of the client being actuated by a user as the user is playing a low latency video game or application;

receiving the control signals related to the low latency video game or application transmitted from the client operated by the user over the network which includes at least one public network component;

routing the control signals to a server and responsively executing program code of the low latency video game or application to generate a plurality of video frames resulting from execution of the video game or application;

encoding the plurality of video frames or portions thereof according to a first encoding format to generate a plurality of encoded video frames or portions, the first encoding format being optimized for low latency transmission to a client device over a current network communication channel including the at least one public network component;

transmitting the plurality of encoded video frames or portions to the client device over the network communication channel,

the client device decoding the video frames and rendering the video on a display,

wherein the operations of generating control signals, receiving the control signals, routing the control signals to the server, responsively executing the program code of the low latency video game or application to generate the plurality of video frames, encoding the plurality of video frames according to the first encoding format optimized for low latency transmission to generate the plurality of encoded video frames, transmitting the plurality of encoded video frames, decoding the video frames on the client and rendering the video on the display are performed with a latency such that the user has the perception that the video game or application is responding instantly to the user input;

concurrently encoding the first plurality of video frames according to a second encoding format, the second encoding format having a relatively higher-quality compressed video and/or a lower compression ratio than the first encoding format;

storing the first plurality of video frames encoded in the second encoding format on a storage device;

making the first plurality of video frames encoded in the second encoding format available to the client device and other client devices for playback of the video stream; and

responsively transmitting the first plurality of video frames encoded in the second encoding format to the client and/or other clients upon receiving a request from a user.

14. The machine-readable medium as in claim 13 wherein the first encoding format comprises frames which are dependent on other frames and the second encoding format comprises some frames which are not dependent on other frames.

15. The machine-readable medium as in claim 14 wherein the second encoding format includes I-Frames and wherein the first encoding format does not include I-Frames.

16. The machine-readable medium as in claim 15 wherein the first encoding format comprises only P-Frames.

17. The machine-readable medium as in claim 13 wherein the first plurality of video frames are frames of a video game, generated as a user of the client device plays an online video game and wherein the first encoding format is selected to reduce latency as the frames compressed according to the first encoding format are transmitted to the client device over the current communication channel.

18. The machine-readable medium as in claim 17 wherein the second encoding format is selected to archive the video frames of the online video game, the video frames to be made available for subsequent playback from the archive.

23. The machine-readable medium as in claim 13 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 80 ms or less.

24. The machine-readable medium as in claim 13 wherein the video frames are encoded and transmitted in response to the control signals with a finite latency of 150 ms or less.