System and method for selecting a video encoding format based on feedback data

• US 9,420,283 B2
• Filed: 04/15/2014
• Issued: 08/16/2016
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Rob, Buckley, Ian
• Primary Examiner(s): SENFI, BEHROOZ M
• Application Number:
  US14/253,357
• Publication Number:
  US 20140226713A1
• Time in Patent Office: 854 Days
• Field of Search: 375/240.02, 375/240.01
• US Class Current: 1/1
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/32 using local area network [LAN] connections
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/164 : Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A computer-implemented system and method for performing video compression are described. For example, a method according to one embodiment comprises: encoding a plurality of video frames or portions thereof according to a first encoding format; transmitting the plurality of encoded video frames or portions to a client device; receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames or portions has been successfully received and/or decoded; in response to detecting that one or more video frames or portions thereof have not been successfully received and/or decoded, determining a number of video frames or portions thereof which have not been successfully received and/or decoded and: (1) if the number of video frames or portions thereof which have not been successfully received and/or decoded is above a specified threshold, then encoding a new video frame or portion thereof according to a second encoding format, the second encoding format comprising a format which is not dependent on previously-transmitted video frames or portions thereof; or (2) if the number of video frames or portions thereof which have not been successfully received and/or decoded is below a specified threshold, then encoding a new video frame or portion thereof according to the first encoding format, the new video frame encoded to be dependent on a last known successfully received video frame or portion thereof; and transmitting the new video frame or portion thereof to the client device.

Claims

1. A computer-implemented method for low-latency video game or application hosting and video compression comprising:

receiving control signals in a first hosting service center, the control signals being received from a client device in response to an input device of the client device being actuated by a user as the user is playing the low latency video game or using the application, the control signals being transmitted from the client device to the first hosting service center over one or more networks;

executing a video game or application on one or more servers at the first hosting service center in response to the control signals received from the client device as the user plays the video game or uses the application;

generating video of the executed video game or application by the one or more servers, the video comprising a plurality of video frames;

encoding the plurality of video frames of the video stream to generate a low-latency encoded video stream;

transmitting the low-latency encoded video stream to the client device over the one or more networks for decoding and displaying the decoded video stream on the client device;

wherein the operations of receiving the control signals, executing the video game or application, generating video, encoding the plurality of video frames, and transmitting the low-latency encoded video stream over the one or more networks to the client device, and decoding and displaying the video on the client device are performed with a latency such that the user of the client device has the perception that the video game or application is responding instantly;

receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames has been successfully received and decoded at the client device;

modifying the encoding of the plurality of video frames in response to detecting that one or more video frames have not been successfully received and decoded at the client device;

wherein encoding the plurality of video frames of the video stream to generate the low-latency encoded video stream further includes encoding the plurality of video frames of the video stream to generate the low-latency encoded video stream and generate a second encoded video stream having a quality independent of the feedback information received from the client device and directing the second encoded video stream to a buffer within the first hosting service center and wherein the second encoded video stream has a first scaling factor wherein the first scaling factor is a full resolution of the video stream.

2. The method as in claim 1 further comprising:

determining latency associated with communicating with the client device; and

(1) if the latency is above a specified threshold, then encoding a new video frame according to a second encoding format, the second encoding format comprising a format which is not dependent on previously-transmitted video frames;

or

(2) if the latency is below a specified threshold, then encoding a new video frame according to the first encoding format, the new video frame encoded to be dependent on a last known successfully received video frame.

3. The method as in claim 2 wherein the frames comprise P-frames or P-tiles, respectively, when encoded according to the first encoding format and wherein the frames encoded comprise I-frames or I-tiles, respectively, when encoded according to the second encoding format.

4. The method as in claim 1 wherein modifying the encoding of the plurality of video frames comprises:

(1) if the number of video frames which have not been successfully received and decoded at the client device is above a specified threshold, then encoding a new video frame according to a second encoding format, the second encoding format comprising a format which is not dependent on previously-transmitted video frames;

or

(2) if the number of video frames which have not been successfully received and decoded at the client device is below a specified threshold, then encoding a new video frame according to the first encoding format, the new video frame encoded to be dependent on a last known successfully received video frame; and

transmitting the new video frame to the client device.

5. The method as in claim 1 wherein the feedback information comprises an indication that the frames have not been successfully received and decoded at the client device.

6. The method as in claim 1 wherein the video frames comprise images of a video game being played at the client device.

7. The method as in claim 1 further comprising:

decoding the encoded video frames at the client device; and

displaying images associated with each of the video frames on a display device on the client device.

8. The method as in claim 1, wherein the first encoded video stream has a second scaling factor wherein the second scaling factor is less than a full resolution of the video stream.

9. The method as in claim 8, further comprising determining a window size in the client device wherein the second scaling factor corresponds to the determined window size in the client device.

10. The method as in claim 9, wherein further comprising generating a third encoded video stream having a quality independent of the feedback information received from the client device and directing the third encoded video stream to a buffer within the first hosting service center, the third encoded video stream having a third scaling factor, wherein the third scaling factor is less than the first scaling factor.

11. The method as in claim 10, further comprising:

wherein the received control signals include a first spectator request from a first spectator client device;

determining a window size in the first spectator request;

selecting one of the second encoded video stream or third encoded video stream having a scaling factor corresponding to the determined window size in the first spectator request; and

transmitting the selected encoded video stream to the first spectator client device.

12. The method as in claim 10, wherein the third scaling factor is one half resolution of the video stream.

13. The method as in claim 10, wherein the third scaling factor is one quarter resolution of the video stream.

14. The method as in claim 1, wherein the received control signal includes a rewind control signal;

and

selecting a video frame from the second encoded video stream corresponding to the received rewind signal by the first hosting service center; and

transmitting the second encoded video stream beginning with the selected video frame from the first hosting service center to the client device.

15. The method as in claim 14, wherein the received rewind control signal is received from a first spectator client device.

16. The method as in claim 14, wherein the second encoded video stream includes a plurality of I-frames.

17. The method of claim 16, wherein selecting the video frame from the second encoded video stream corresponding to the received rewind signal by the first hosting service center includes:

selecting a nearest succeeding I-frame corresponding to the received rewind signal; and

decoding backward in time to the video frame from the second encoded video stream corresponding to the received rewind signal.

18. The method as in claim 8, wherein the second encoded video stream includes a plurality of I-frames having a periodicity of not more than one I-frame in every twelve video frames.

19. The method as in claim 8, wherein the second encoded video stream includes a plurality of I-frames having a consistent periodicity.

20. The method as in claim 19, wherein the server includes a speed multiplier device accessible by the user through the client device, wherein the speed multiplier device allows the user to select a fast forward or a rewind of the second encoded video stream and wherein the speed multiplier device uses an exact multiple of the I-frames periodicity.

21. The method of claim 20, wherein the received control signals include at least one of a fast-forward control signal or a rewind control signal and wherein the received control signal selects an I-frame in the second encoded video stream and transmits the second encoded video stream to the client device and initiates the decoding of the second encoded video stream at the selected I-frame in the second encoded video stream in the client device.

22. The method of claim 21, wherein the received control signals include at least one of a fast-forward control signal or a rewind control signal and wherein the received control signal is received from a spectator client device and wherein the received control signal selects an I-frame in the second encoded video stream and transmits the second encoded video stream to the spectator client device and initiates the decoding of the second encoded video stream at the selected I-frame in the second encoded video stream in the spectator client device.

23. A computer-implemented method for low-latency video game or application hosting and video compression comprising:

receiving control signals in at least one server, the control signals being received from a client device in response to a user input to the client device as the user is playing the low latency video game or using the application, the control signals being transmitted from the client device to the at least one server over one or more networks;

executing a video game or application on the at least one server in response to the control signals received from the client device;

generating video of the executed video game or application by the at least one server, the video comprising a plurality of video frames;

encoding the plurality of video frames of the video stream to generate a first low-latency encoded video stream and a second encoded video stream, the second compressed video stream having a quality independent of the a data rate capacity of the network between the client device and the at least one server;

transmitting the first low-latency encoded video stream to the client device over the one or more networks for decoding and displaying the decoded video stream on the client device;

receiving feedback information from the client device, the feedback information usable to determine whether data contained in the video frames in the first low-latency encoded video stream has been successfully received and decoded at the client device; and

modifying the encoding of the plurality of video frames in the first low-latency encoded video stream in response to detecting that one or more video frames have not been successfully received and decoded at the client device; and

directing the second encoded video stream to a buffer within the server.

24. The method as in claim 23, wherein the second encoded video stream includes a plurality of I-frames having a consistent periodicity.

25. The method as in claim 24, wherein the at least one server includes a speed multiplier device accessible by the user through the client device, wherein the speed multiplier device allows the user to select a fast forward or a rewind of the second encoded video stream and wherein the speed multiplier device uses an exact multiple of the I-frames periodicity.

26. The method of claim 25, wherein the received control signals include at least one of a fast-forward control signal or a rewind control signal and wherein the received control signal selects an I-frame in the second encoded video stream and transmits the second encoded video stream to the client device and initiates the decoding of the second encoded video stream at the selected I-frame in the second encoded video stream in the client device.

27. The method of claim 25, wherein the received control signals include at least one of a fast-forward control signal or a rewind control signal and wherein the received control signal is received from a spectator client device and wherein the received control signal selects an I-frame in the second encoded video stream and transmits the second encoded video stream to the spectator client device and initiates the decoding of the second encoded video stream at the selected I-frame in the second encoded video stream in the spectator client device.