System and method for utilizing forward error correction with video compression

• US 9,077,991 B2
• Filed: 08/07/2009
• Issued: 07/07/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/538,062
• Publication Number:
  US 20100166064A1
• Time in Patent Office: 2,160 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/77 involving data related to game devices or game servers, e.g. configuration data, software version or amount of memory
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 19/65 : using error resilience
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/266 : Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method are described for performing video compression. For example, a method according to one embodiment comprises: encoding a plurality of video frames or portions thereof according to a first encoding format; transmitting the plurality of encoded video frames or portions to a client device; receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; in response to detecting that one or more video frames or portions thereof have not been successfully received and/or decoded, then either encoding a new video frame or portion thereof according to a second encoding format or encoding the new video frame to be dependent on the last video frame or portion thereof known to have been successfully received and/or decoded, and: (1) if no forward error correction (FEC) coding is used to transmit the plurality of encoded video frames or portions, then encoding a new video frame or portion thereof using FEC coding; or (2) if a first FEC coding is used to transmit the plurality of encoded video frames or portions, then encoding a new video frame or portion thereof using a second FEC coding, the second FEC coding being a relatively stronger FEC coding than the first FEC coding; and transmitting the new video frame or portion thereof to the client device.

Claims

1. A computer-implemented method for hosting online applications or video games, performing video compression, and transmitting compressed video over a network comprising:

receiving a request from a user for playing a low latency video game or executing an application;

generating control signals from a client in response to an input device of the client being actuated by the user as the user is playing the low latency video game or using the application;

receiving the control signals at a server, the control signals transmitted over a network at least a portion of which comprises a public network, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising a plurality of video frames of the video game or application;

encoding the plurality of video frames using low latency encoding to generate low latency encoded video frames, wherein the low latency encoding comprises;

subdividing each video frame of the plurality into a first set of one or more frame portions and a second set of one or more frame portions;

encoding frame portions in the first set using a first type of encoding;

encoding frame portions in the second set using a second type of encoding;

wherein subdividing each video frame into frame portions and encoding the first set of frame portions using the first encoding format and the second set of frame portions using the second encoding format produces a more consistent bitrate for the video frames and lower latency for the end user than if entire video frames were compressed using the first type of encoding and the second type of encoding;

transmitting the plurality of low latency encoded video frames to a client device over the network;

receiving feedback information from the client device over the network, the feedback information usable to determine whether data contained in the low latency encoded video frames or frame portions has not been successfully received and/or decoded;

in response to detecting that one or more low latency encoded video frames or frame portions thereof have not been successfully received and/or decoded, then adjusting the encoding of the low latency encoded video frames while maintaining a latency such that the user has the perception that the selected video game or application is responding instantly to the signals, wherein adjusting the encoding comprises adjusting the ratio of frame portions compressed using the first type of encoding to the frame portions compressed with the second type of encoding in each frame;

decoding the low latency encoded video frames on the client device and rendering the resulting decoded video frames on a display of the client device;

wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

2. The method as in claim 1 wherein each video frame portion encoded according to the second type of encoding is dependent on a prior video frame portion, respectively, and wherein each video frame portion encoded according to the first encoding format is not dependent on a prior video frame or portion, respectively.

3. The method as in claim 2 wherein the first type of encoding format comprises intra-frame-portion coding and wherein the second encoding format comprises inter-frame-portion coding.

4. The method as in claim 3 wherein the frame portions selected to be included in the first set in a first video frame are selected to be included in the second set in one or more successive video frames following the first video frame.

5. The method as in claim 1 wherein the feedback information comprises an indication that the frames or frame portions have not been successfully received and/or decoded at the client device.

6. The method as in claim 1 wherein the frame portions are selected to be in the first set or the second set based, at least in part, on characteristics of video content contained in the frame portions.

7. The method as in claim 6 wherein the characteristics of the video content includes an amount of change in the video content between successive video frames.

22. The method as in claim 1 wherein adjusting the encoding comprises:

if no forward error correction (FEC) coding is initially used to transmit the plurality of video frames or frame portions, then encoding a new video frame or frame portion thereof using FEC coding;

or

if a first FEC coding is initially used to transmit the plurality of encoded video frames or frame portions, then encoding a new video frame or frame portion thereof using a second FEC coding, the second FEC coding being a relatively stronger FEC coding than the first FEC coding; and

transmitting the new video frame or frame portion thereof to the client device.

23. The method as in claim 1 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 80 ms or less.

24. The method as in claim 1 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 150 ms or less.

31. The method as in claim 7 wherein frame portions having an amount of change in the video content above a specified threshold are included in the first set of frame portions and encoded using the first type of encoding.

32. The method as in claim 31 wherein one or more video frames are subdivided into frame portions having different sizes.

33. The method as in claim 32 wherein the different sizes of the frame portions are selected based on the amount of change detected in video content contained in the frame portions.

34. The method as in claim 33 wherein frame portions having an amount of change in the video content above a specified threshold are given a relatively smaller size than frame portions having an amount of change below the specified threshold.

8. A system for hosting online applications or video games comprising a memory for storing program code and a processor for processing the program code to perform the operations of:

receiving a request from a user for playing a low latency video game or executing an application;

generating control signals from a client in response to an input device of the client being actuated by the user as the user is playing the low latency video game or using the application;

receiving the control signals at a server, the control signals transmitted over a network at least a portion of which comprises a public network, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising a plurality of video frames of the video game or application;

encoding the plurality of video frames using low latency encoding to generate low latency encoded video frames, wherein the low latency encoding comprises;

subdividing each video frame of the plurality into a first set of one or more frame portions and a second set of one or more frame portions;

encoding frame portions in the first set using a first type of encoding;

encoding frame portions in the second set using a second type of encoding;

wherein subdividing each video frame into frame portions and encoding the first set of frame portions using the first encoding format and the second set of frame portions using the second encoding format produces a more consistent bitrate for the video frames and lower latency for the end user than if entire video frames were compressed using the first type of encoding and the second type of encoding;

transmitting the plurality of low latency encoded video frames to a client device over the network;

receiving feedback information from the client device over the network, the feedback information usable to determine whether data contained in the low latency encoded video frames or frame portions has not been successfully received and/or decoded;

in response to detecting that one or more low latency encoded video frames or frame portions thereof have not been successfully received and/or decoded, then adjusting the encoding of the low latency encoded video frames while maintaining a latency such that the user has the perception that the selected video game or application is responding instantly to the signals, wherein adjusting the encoding comprises adjusting the ratio of frame portions compressed using the first type of encoding to the frame portions compressed with the second type of encoding in each frame;

decoding the low latency encoded video frames on the client device and rendering the resulting decoded video frames on a display of the client device;

wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

9. The system as in claim 8 wherein each video frame portion encoded according to the second type of encoding is dependent on a prior video frame portion, respectively, and wherein each video frame portion encoded according to the first encoding format is not dependent on a prior video frame or portion, respectively.

10. The system as in claim 9 wherein the first type of encoding format comprises intra-frame-portion coding and wherein the second encoding format comprises inter-frame-portion coding.

11. The system as in claim 10 wherein the frame portions selected to be included in the first set in a first video frame are selected to be included in the second set in one or more successive video frames following the first video frame.

12. The system as in claim 8 wherein the feedback information comprises an indication that the frames or frame portions have not been successfully received and/or decoded at the client device.

13. The system as in claim 8 wherein the frame portions are selected to be in the first set or the second set based, at least in part, on characteristics of video content contained in the frame portions.

14. The system as in claim 13 wherein the characteristics of the video content includes an amount of change in the video content between successive video frames.

25. The system as in claim 8 wherein adjusting the encoding comprises:

if no forward error correction (FEC) coding is initially used to transmit the plurality of video frames or frame portions, then encoding a new video frame or frame portion thereof using FEC coding;

or

if a first FEC coding is initially used to transmit the plurality of encoded video frames or frame portions, then encoding a new video frame or frame portion thereof using a second FEC coding, the second FEC coding being a relatively stronger FEC coding than the first FEC coding; and

transmitting the new video frame or frame portion thereof to the client device.

26. The system as in claim 8 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 80 ms or less.

27. The system as in claim 8 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 150 ms or less.

35. The system as in claim 14 wherein frame portions having an amount of change in the video content above a specified threshold are included in the first set of frame portions and encoded using the first type of encoding.

36. The system as in claim 35 wherein one or more video frames are subdivided into frame portions having different sizes.

37. The system as in claim 36 wherein the different sizes of the frame portions are selected based on the amount of change detected in video content contained in the frame portions.

38. The system as in claim 37 wherein frame portions having an amount of change in the video content above a specified threshold are given a relatively smaller size than frame portions having an amount of change below the specified threshold.

15. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:

receiving a request from a user for playing a low latency video game or executing an application;

generating control signals from a client in response to an input device of the client being actuated by the user as the user is playing the low latency video game or using the application;

receiving the control signals at a server, the control signals transmitted over a network at least a portion of which comprises a public network, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate video output comprising a plurality of video frames of the video game or application;

encoding the plurality of video frames using low latency encoding to generate low latency encoded video frames, wherein the low latency encoding comprises;

subdividing each video frame of the plurality into a first set of one or more frame portions and a second set of one or more frame portions;

encoding frame portions in the first set using a first type of encoding;

encoding frame portions in the second set using a second type of encoding;

wherein subdividing each video frame into frame portions and encoding the first set of frame portions using the first encoding format and the second set of frame portions using the second encoding format produces a more consistent bitrate for the video frames and lower latency for the end user than if entire video frames were compressed using the first type of encoding and the second type of encoding;

transmitting the plurality of low latency encoded video frames to a client device over the network;

receiving feedback information from the client device over the network, the feedback information usable to determine whether data contained in the low latency encoded video frames or frame portions has not been successfully received and/or decoded;

in response to detecting that one or more low latency encoded video frames or frame portions thereof have not been successfully received and/or decoded, then adjusting the encoding of the low latency encoded video frames while maintaining a latency such that the user has the perception that the selected video game or application is responding instantly to the signals, wherein adjusting the encoding comprises adjusting the ratio of frame portions compressed using the first type of encoding to the frame portions compressed with the second type of encoding in each frame;

decoding the low latency encoded video frames on the client device and rendering the resulting decoded video frames on a display of the client device;

wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

16. The non-transitory machine-readable medium as in claim 15 wherein each video frame portion encoded according to the second type of encoding is dependent on a prior video frame portion, respectively, and wherein each video frame portion encoded according to the first encoding format is not dependent on a prior video frame or portion, respectively.

17. The non-transitory machine-readable medium as in claim 16 wherein the first type of encoding format comprises intra-frame-portion coding and wherein the second encoding format comprises inter-frame-portion coding.

18. The non-transitory machine-readable medium as in claim 17 wherein the frame portions selected to be included in the first set in a first video frame are selected to be included in the second set in one or more successive video frames following the first video frame.

19. The non-transitory machine-readable medium as in claim 15 the feedback information comprises an indication that the frames or frame portions have not been successfully received and/or decoded at the client device.

20. The non-transitory machine-readable medium as in claim 15 wherein the frame portions are selected to be in the first set or the second set based, at least in part, on characteristics of video content contained in the frame portions.

21. The non-transitory machine-readable medium as in claim 20 wherein the characteristics of the video content includes an amount of change in the video content between successive video frames.

28. The non-transitory machine-readable medium as in claim 15 wherein adjusting the encoding comprises:

if no forward error correction (FEC) coding is initially used to transmit the plurality of video frames or frame portions, then encoding a new video frame or frame portion thereof using FEC coding;

or

if a first FEC coding is initially used to transmit the plurality of encoded video frames or frame portions, then encoding a new video frame or frame portion thereof using a second FEC coding, the second FEC coding being a relatively stronger FEC coding than the first FEC coding; and

transmitting the new video frame or frame portion thereof to the client device.

29. The non-transitory machine-readable medium as in claim 15 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 80 ms or less.

30. The non-transitory machine-readable medium as in claim 15 wherein the operations of generating the control signals, receiving the control signals transmitted over the network, executing the video game or application, encoding the plurality of video frames using low latency encoding, transmitting the low latency encoded video frames, decoding the low latency encoded video frames, and rendering the resulting decoded video frames on a display are performed with a finite latency of 150 ms or less.