System and method for video compression using feedback including data related to the successful receipt of video content

• US 8,526,490 B2
• Filed: 08/07/2009
• Issued: 09/03/2013
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US12/538,096
• Publication Number:
  US 20100166066A1
• Time in Patent Office: 1,488 Days
• Field of Search: 375/240, 375/240.01, 375/240.12, 375/240.15, 375/240.25, 375/240.27, 714/774
• US Class Current: 375/240.01
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/266 : Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method for performing video compression are described. For example, a method according to one embodiment comprises: encoding a first plurality of video frames or portions thereof, wherein each encoded video frame or portion thereof is dependent on a previously-encoded video frame or portion thereof, respectively; transmitting the first plurality of encoded video frames or portions to a client device; receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; in response to detecting that a video frame or portion thereof has not been successfully received and/or decoded, encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device; and transmitting the current video frame or portion thereof to the client device.

Claims

1. A computer-implemented method for hosting online applications or video games, performing video compression comprising, and transmitting video comprising:

receiving a request from a user for playing a low latency video game or executing an application which produces a video output;

receiving signals at a server, the signals transmitted over a network which includes public network components, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate the video output comprising a sequence of images of the video game or application;

encoding the video output to generate a sequence of compressed video images, wherein encoding comprises;

encoding a first plurality of video frames or portions thereof, wherein each encoded video frame or portion thereof is dependent on a previously-encoded video frame or portion thereof, respectively;

transmitting the first plurality of encoded video frames or portions to a client device;

receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; and

in response to detecting that a video frame or portion thereof has not been successfully received and/or decoded, encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device;

transmitting the sequence of compressed video images to the client device over the network;

decompressing the sequence of compressed video images on the client device;

wherein the operations of receiving signals, executing the video game or application, encoding the video output, transmitting the sequence of compressed video images, and decompressing the sequence of compressed video images are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

2. The method as in claim 1 wherein the previously-encoded video frame or portion thereof known to have been successfully received and/or decoded comprises the last known video frame or portion thereof known to have been successfully received and/or decoded.

3. The method as in claim 1 wherein the video frames or portions thereof comprise P-frames or P-tiles, respectively.

4. The method as in claim 1 wherein the feedback information comprises an indication that the frames or portions thereof have been successfully received and/or decoded at the client device.

5. The method as in claim 1 wherein the feedback information comprises an indication that the frames or portions thereof have not been successfully received and/or decoded at the client device.

6. The method as in claim 1 wherein encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device further comprises:

retrieving a prior state from a memory, the prior state comprising a state of an encoder after encoding the previously-encoded video frame or portion.

7. The method as in claim 6 further comprising:

decoding the current video frame or portion thereof, wherein decoding comprises retrieving a prior state from a memory, the prior state comprising a state of a decoder after decoding the previously-encoded video frame or portion.

8. The method as in claim 1 further comprising:

decoding the encoded video frames or portion thereof at the client device; and

displaying images associated with each of the video frames or portions thereof on a display on the client device.

9. A system comprising a memory for storing program code and a processor for processing the program code to perform the operations of:

receiving a request from a user for playing a low latency video game or executing an application which produces a video output;

receiving signals at a server, the signals transmitted over a network which includes public network components, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate the video output comprising a sequence of images of the video game or application;

encoding the video output to generate a sequence of compressed video images, wherein encoding comprises;

encoding a first plurality of video frames or portions thereof, wherein each encoded video frame or portion thereof is dependent on a previously-encoded video frame or portion thereof, respectively;

transmitting the first plurality of encoded video frames or portions to a client device;

receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; and

in response to detecting that a video frame or portion thereof has not been successfully received and/or decoded, encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device;

transmitting the sequence of compressed video images to the client device over the network;

decompressing the sequence of compressed video images on the client device;

wherein the operations of receiving signals, executing the video game or application, encoding the video output, transmitting the sequence of compressed video images, and decompressing the sequence of compressed video images are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

10. The system as in claim 9 wherein the previously-encoded video frame or portion thereof known to have been successfully received and/or decoded comprises the last known video frame or portion thereof known to have been successfully received and/or decoded.

11. The system as in claim 9 wherein the video frames or portions thereof comprise P-frames or P-tiles, respectively.

12. The system as in claim 9 wherein the feedback information comprises an indication that the frames or portions thereof have been successfully received and/or decoded at the client device.

13. The system as in claim 9 wherein the feedback information comprises an indication that the frames or portions thereof have not been successfully received and/or decoded at the client device.

14. The system as in claim 9 wherein encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device further comprises:

retrieving a prior state from a memory, the prior state comprising a state of an encoder after encoding the previously-encoded video frame or portion.

15. The system as in claim 14 comprising additional program code to cause the processor to perform the operations of:

decoding the current video frame or portion thereof, wherein decoding comprises retrieving a prior state from a memory, the prior state comprising a state of a decoder after decoding the previously-encoded video frame or portion.

16. The system as in claim 9 comprising additional program code to cause the processor to perform the operations of:

decoding the encoded video frames or portion thereof at the client device; and

displaying images associated with each of the video frames or portions thereof on a display on the client device.

17. A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform the operations of:

receiving a request from a user for playing a low latency video game or executing an application which produces a video output;

receiving signals at a server, the signals transmitted over a network which includes public network components, as the user is playing the video game or using the application and responsively executing the video game or application on the server to generate the video output comprising a sequence of images of the video game or application;

encoding the video output to generate a sequence of compressed video images, wherein encoding comprises;

encoding a first plurality of video frames or portions thereof, wherein each encoded video frame or portion thereof is dependent on a previously-encoded video frame or portion thereof, respectively;

transmitting the first plurality of encoded video frames or portions to a client device;

receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; and

in response to detecting that a video frame or portion thereof has not been successfully received and/or decoded, encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device; and

transmitting the sequence of compressed video images to the client device over the network;

decompressing the sequence of compressed video images on the client device;

wherein the operations of receiving signals, executing the video game or application, encoding the video output, transmitting the sequence of compressed video images, and decompressing the sequence of compressed video images are performed with a latency such that the user has the perception that the selected video game or application is responding instantly to the signals.

18. The non-transitory machine-readable medium as in claim 17 wherein the previously-encoded video frame or portion thereof known to have been successfully received and/or decoded comprises the last known video frame or portion thereof known to have been successfully received and/or decoded.

19. The non-transitory machine-readable medium as in claim 17 wherein the video frames or portions thereof comprise P-frames or P-tiles, respectively.

20. The non-transitory machine-readable medium as in claim 17 wherein the feedback information comprises an indication that the frames or portions thereof have been successfully received and/or decoded at the client device.

21. The non-transitory machine-readable medium as in claim 17 wherein the feedback information comprises an indication that the frames or portions thereof have not been successfully received and/or decoded at the client device.

22. The non-transitory machine-readable medium as in claim 17 wherein encoding a current video frame or portion thereof to be dependent on a previously-encoded video frame or portion thereof known to have been successfully received and/or decoded on the client device further comprises:

retrieving a prior state from a memory, the prior state comprising a state of an encoder after encoding the previously-encoded video frame or portion.

23. The non-transitory machine-readable medium as in claim 22 comprising additional program code to cause the machine to perform the operations of:

decoding the current video frame or portion thereof, wherein decoding comprises retrieving a prior state from a memory, the prior state comprising a state of a decoder after decoding the previously-encoded video frame or portion.

24. The non-transitory machine-readable medium as in claim 17 comprising additional program code to cause the processor to perform the operations of:

decoding the encoded video frames or portion thereof at the client device; and

displaying images associated with each of the video frames or portions thereof on a display on the client device.