System and method for remote-hosted video effects

• US 10,375,426 B2
• Filed: 06/28/2013
• Issued: 08/06/2019
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., Toy, Michael, Cotter, Timothy S., Poichet, Jerome, Olbrich, Paul Andrew
• Primary Examiner(s): Goodarzi, Nasser M
• Assistant Examiner(s): Ocak, Adil
• Application Number:
  US13/931,387
• Publication Number:
  US 20130298178A1
• Time in Patent Office: 2,230 Days
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/00 Network arrangements, protocols or services for supporting real-time applications in data packet communication real-time or near real-time messaging, e.g. instant messaging [IM] H04L51/04; selective video distribution H04N21/00
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/764 at the destination reformatting of additional data in video clients H04N21/4355
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 : Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/166 : concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2143 : located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/23 : Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4122 : additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/422 : Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/44209 : Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4728 : for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6373 : for rate control , e.g. request to the server to modify its transmission rate flow control in packet networks H04L47/10
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04N 7/24 : Systems for the transmission of television signals using pulse code modulation H04N21/00 takes precedence

Abstract

A method for remote-hosted video effects includes rendering an application frame buffer in a hosting service that is streaming interactive video. Only a portion of the application frame buffer is subsequently displayed on a local device.

Claims

1. A method for delivering video over a packetized network comprising:

subdividing, by a hosting service, video frames of a video stream into a plurality of tiles and encoding the plurality of tiles as I-tiles and P-tiles;

transmitting, by the hosting service, compressed video comprising an initial set of encoded I-tiles followed by a sequence of encoded P-tiles over a communication channel to a client device configured with a decoder for decompressing the encoded tiles, wherein each P-tile is encoded to be dependent on a previous P-tile when there is no indication of unsuccessfully received tiles, said transmitting includes sending a first tile before sending a second tile within the sequence of encoded P-tiles;

receiving, by the hosting service from the client device, an acknowledgement message that the second tile was successfully received by the client device but not receiving an acknowledgement message related to the first tile; and

responsive to receiving the acknowledgement message that the second tile was successfully received but not the acknowledgement message related to the first tile, transmitting a new set of encoded I-tiles to the client device, wherein the client device is configured with a memory device for storing a past state of the decoder and wherein the past state includes at least a state associated with a P-tile which was the last P-tile that was successfully received by the decoder.

2. The method of claim 1, wherein a single I-tile of the new set of encoded I-tiles is transmitted at a different position within a set of successive encoded frames.

3. The method of claim 1, wherein the new set of encoded I-tiles are individually spread across multiple encoded frames.

4. The method of claim 1, further comprising generating a new sequence of encoded P-tiles for streaming to the client device.

5. The method of claim 2, wherein each of said successive encoded frame includes four tiles.

6. A method for delivering video over a packetized network comprising:

subdividing, by a hosting service, video frames of a video stream into a plurality of tiles and encoding the plurality of tiles as I-tiles and P-tiles;

transmitting, by the hosting service, compressed video comprising a sequence of encoded P-tiles over a communication channel to a client device configured with a decoder for decompressing the encoded tiles, wherein each P-tile is encoded to be dependent on a previous P-tile when there is no indication of unsuccessfully received tiles, said transmitting includes sending a first tile before sending a second tile within the sequence of encoded P-tiles;

receiving, by the hosting service from the client device, an acknowledgement message that the second tile was successfully received by the client device but not receiving an acknowledgement message associated with the first tile, wherein the decoder client device is configured with a memory device for storing a past state of the decoder and wherein the past state includes at least a state associated with a P-tile which was the last P-tile that was successfully received by the decoder; and

responsive to receiving the acknowledgement message that the second tile was successfully received but not receiving the acknowledgement message associated with the first tile, encoding, by the hosting service, a next P-tile dependent upon the last P-tile that was successfully received by the decoder.

7. The method of claim 6, further comprising storing in a memory device of the hosting service a past state of an encoder utilized to generate the encoded P-tiles.

8. The method of claim 7, wherein the past state of the decoder includes combined data from prior P-tiles.

9. The method of claim 7 wherein the past state of the encoder comprises combined data from prior encoded P-tiles.

10. The method of claim 7, further comprising retrieving the past state of the encoder following receipt of the acknowledgement message that the second tile was successfully received but not the acknowledgement message associated with the first tile.

11. The method of claim 10, wherein the next P tile is generated from the past state of the encoder which includes a state associated with the last P-tile that was successfully received by the decoder.