System and method for compressing video frames or portions thereof based on feedback information from a client device

• US 9,643,084 B2
• Filed: 04/12/2016
• Issued: 05/09/2017
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US15/097,076
• Publication Number:
  US 20160236083A1
• Time in Patent Office: 392 Days
• Field of Search: 37524007
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/71 using secure communication between game devices and game servers, e.g. by encrypting game data or authenticating players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/234363 : by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/23439 : for generating different versions
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/25833 : involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/266 : Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
• CPC Class Codes: H04N 21/2662 : Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method are described for performing video compression. For example, a method according to one embodiment comprises: encoding a plurality of video frames or portions thereof according to a first encoding format; transmitting the plurality of encoded video frames or portions to a client device; receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has not been successfully received and/or decoded; in response to detecting that a video frame or portion thereof has not been successfully received and/or decoded, encoding a video frame or portion thereof according to a second encoding format; and transmitting the video frames or portions thereof to the client device.

Claims

1. A computer-implemented method for performing video compression comprising:

receiving a control signal by a server, the control signal received from a client device over a network established between the client device and the server, the control signal corresponding to a user input from an input device of the client device being actuated by a user as the user is playing a video game, the network including a communication channel;

executing the video game on the server in response to the received control signal to generate video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a shared video encoding resource to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured for displaying on a display device of the client device;

detecting a change in a communication channel characteristic of the communication channel, wherein the detected change is based on feedback information received by the server, the feedback information being received from the client device; and

dynamically adjusting a compression ratio for the sequence of encoded video frames based on the detected change in the communication channel characteristic.

2. The method of claim 1, wherein, the adjusted compression ratio is based on selection of a first encoding format or a second encoding format.

3. The method of claim 1, wherein the feedback information is further usable to identify a previous video frame successfully received by the client device, wherein a new video frame is encoded according to a second encoding format to be dependent on the previous successfully received video frame.

4. The method of claim 1, wherein the feedback information comprises an indication that the encoded sequence of video frames have been successfully received or decoded at the client device.

5. The method of claim 1, wherein the feedback information comprises an indication that at least one of the encoded sequence of video frames have not been successfully received or decoded at the client device.

6. The method of claim 1, wherein the sequence of encoded video frames comprise images of the video game displayed by the display device of the client device.

7. The method of claim 1, wherein the feedback information includes at least one of an indication that the sequence of encoded video frames have been decoded at the client device or an indication that images associated with each of the sequence of encoded video frames have been displayed by the display device of the client device.

8. The method of claim 1, wherein detecting the change in the communication channel characteristic of the communication channel includes:

detecting the communication channel is inoperative; and

pausing the video game execution in the server.

9. The method of claim 1, wherein the video game is a multiplayer video game and the client device is one of a plurality of client devices and wherein the communication channel is one of a plurality of communication channels established between the server and a respective each one of the plurality of client devices;

wherein detecting the change in the communication channel characteristic of the communication channel includes detecting a first communication channel between the server and a respective first client device of the plurality of client devices is inoperative, wherein the first communication channel is one of the plurality of communication channels; and

notifying at least one of the plurality of client devices that the first client device is not available.

10. The method of claim 9, further comprising pausing the execution of the multiplayer video game in the server for the plurality of client devices, when the first communication channel is inoperative.

11. The method of claim 1, further comprising:

determining that a maximum data rate capacity of the communication channel has decreased based on at least one of a detected increase in packet loss or an increase in latency while the sequence of encoded video frames are being transmitted in a streaming format to the client device; and

responsively and dynamically reducing the data rate of the sequence of encoded video frames until at least one of the packet loss or latency have reached corresponding predetermined acceptable values.

12. A computer-implemented method for performing video compression comprising:

receiving a control signal by a server, the control signal received from a client device over a network established between the client device and the server, the control signal corresponding to a user input from an input device of the client device being actuated by a user as the user is playing a video game, the network including a communication channel;

executing the video game on the server in response to the received control signal to generate video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a shared video encoding resource to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured for displaying on a display device of the client device;

detecting a change in a communication channel characteristic of the communication channel, wherein the detected change is based on feedback information received by the server, the feedback information being received from the client device;

dynamically adjusting an image resolution of the sequence of encoded video frames based on the detected change in the communication channel characteristic; and

transmitting an instruction to the client device to scale a display of each one of the sequence of decoded video frames to maintain an image size corresponding to the display device of the client device.

13. The method of claim 12, wherein, the adjusted image resolution of the sequence of encoded video frames is based on selection of a first encoding format or a second encoding format.

14. The method of claim 12, wherein the feedback information is further usable to identify a previous video frame successfully received by the client device, wherein a new video frame is encoded according to a second encoding format to be dependent on the identified previous video frame that was successfully received by the client device.

15. The method of claim 12, wherein the feedback information comprises an indication that the encoded sequence of video frames have been successfully received or decoded at the client device.

16. The method of claim 12, wherein the feedback information comprises an indication that at least one of the encoded sequence of video frames have not been successfully received or decoded at the client device.

17. The method of claim 12, wherein the sequence of encoded video frames comprise images of the video game displayed by the display device of the client device.

18. The method of claim 12, wherein the feedback information includes at least one of an indication that the sequence of encoded video frames have been decoded at the client device or an indication that images associated with each of the sequence of encoded video frames have been displayed by the display device of the client device.

19. The method of claim 12, wherein detecting the change in the communication channel characteristic of the communication channel includes:

detecting the communication channel is inoperative; and

pausing the video game execution in the server.

20. The method of claim 12, wherein the video game is a multiplayer video game and the client device is one of a plurality of client devices and wherein the communication channel is one of a plurality of communication channels established between the server and a respective each one of the plurality of client devices;

wherein detecting the change in the communication channel characteristic of the communication channel includes detecting a first communication channel between the server and a respective first client device of the plurality of client devices is inoperative, wherein the first communication channel is one of the plurality of communication channels; and

notifying at least one of the plurality of client devices that the first client device is not available.

21. The method of claim 20, further comprising pausing the execution of the multiplayer video game in the server for the plurality of client devices, when the first communication channel is inoperative.

22. The method of claim 12, further comprising:

determining that a maximum data rate capacity of the communication channel has decreased based on an indication of an increased number of the encoded sequence of video frames that have not been successfully received or decoded at the client device while the sequence of encoded video frames are being transmitted in a streaming format to the client device; and

responsively and dynamically reducing the data rate of the sequence of encoded video frames until the number of the encoded sequence of video frames that have not been successfully received or decoded at the client device have reached a corresponding predetermined acceptable value.

23. A computer-implemented method for performing video compression comprising:

receiving a control signal by a server, the control signal received from a client device over a network established between the client device and the server, the control signal corresponding to a user input from an input device of the client device being actuated by a user as the user is playing a video game, the network including a communication channel between the server and the client device;

executing the video game on the server, in response to the received control signal, to generate video output including a sequence of video images of the video game;

encoding each of the sequence of video images in a shared video encoding resource to generate a sequence of encoded video frames;

transmitting the sequence of encoded video frames by the server over the communication channel to the client device for decoding to produce a corresponding sequence of decoded video frames, the sequence of decoded video frames are configured for displaying on a display device of the client device;

receiving feedback information by the server, the feedback information received from the client device; and

determining a maximum data rate capacity of the communication channel by sending an increasingly greater data rate stream of the sequence of encoded video frames to the client device until at least one of a packet loss or an increased latency indicates the maximum data rate capacity of the communication channel has been exceeded.

24. The method of claim 23, further comprising:

determining that the maximum data rate capacity of the communication channel has decreased based on at least one of a detected increase in packet loss or an increase in latency while the sequence of encoded video frames are being streamed to the client device; and

responsively and dynamically reducing the data rate or increasing a compression ratio of the sequence of encoded video frames until at least one of the packet loss or the latency have reached corresponding predetermined acceptable values.

25. The method of claim 24, wherein responsively and dynamically reducing the data rate or increasing the compression ratio of the sequence of encoded video frames includes modifying a frame rate of the sequence of encoded video frames.

26. The method of claim 24, wherein responsively and dynamically reducing the data rate or increasing the compression ratio of the sequence of encoded video frames includes modifying an image resolution of the sequence of encoded video frames.

27. The method of claim 26, wherein modifying the image resolution includes transmitting an instruction to the client device to scale a display of each one of the sequence of decoded video frames to maintain an image size corresponding to the display device of the client device.