System and method for protecting certain types of multimedia data transmitted over a communication channel

• US 9,084,936 B2
• Filed: 01/23/2009
• Issued: 07/21/2015
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/359,150
• Publication Number:
  US 20090196516A1
• Time in Patent Office: 2,370 Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 2009/245 visual
• CPC Class Codes: A63F 2009/247 audible, e.g. using a loudspeaker
• CPC Class Codes: A63F 2009/2488 Remotely playable
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 : between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 : Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2143 : located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/262 : Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/4307 : Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 21/43074 : of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/439 : Processing of audio elementary streams
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 28/16 : Central resource management; Negotiation of resources or communication parameters, e.g. negotiating bandwidth or QoS [Quality of Service]
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

A system and method are described for protecting certain types of multimedia data transmitted over a communication channel. For example, one embodiment of a computer-implemented method comprises: logically subdividing each of a sequence of images into a plurality of tiles, each of the tiles having a defined position within each of the sequence of images, the defined position remaining the same between successive images; encoding one or more of the tiles in each image of the sequence of images using a first compression format and encoding the reminder of the tiles in each image using the second compression format, the second compression format dependent on tiles previously-encoded by the first and/or the second compression formats; generating a forward error correction (FEC) code for tiles encoded using the first compression format; transmitting the FEC code with each of the tiles encoded using the first compression format to a client.

Claims

1. A computer-implemented method for hosting online applications or video games, performing video compression comprising, and transmitting video comprising:

receiving a request from a first user for playing a low latency video game or executing an application which produces a video output;

generating first control signals on a first client device in response to user input from an input device of the first client device being actuated by the first user as the first user is playing the video game or using the application;

capturing video and audio of the first user as the first user is playing the video game or using the application;

generating second control signals from a second client in response to an input device of the second client being actuated by a second user as the second user is playing the low latency video game or using the application;

transmitting the first control signals over a first communication channel of the first client which includes at least one public network component and transmitting the video and audio of the first user from the first client to the one or more servers;

transmitting the second control signals over a second communication channel of the second client which includes at least one public network component;

routing the first control signals and second control signals to one or more servers, as the first and second users are playing the video game or using the application;

responsively executing the video game or application on the one or more servers based on both the first control signals and the second control signals to generate video output comprising at least one sequence of video images of the video game or application;

determining characteristics of the first client and/or the first communication channel;

determining characteristics of the second client and/or the second communication channel;

compressing the at least one sequence of video images of the video game or application in accordance with the determined characteristics of the first client and/or the first communication channel using low latency compression to generate a first sequence of compressed video images having a first set of characteristics;

compressing the sequence of video images of the video game or application in accordance with the determined characteristics of the second client and/or the second communication channel using low latency compression to generate a second sequence of compressed video images having a second set of characteristics;

incorporating the video and/or audio of the first user into the second sequence of video images compressed in accordance with the determined characteristics of the second client and/or the second communication channel to update the second sequence of compressed video images;

wherein compressing the at least one sequence of video images to generate the first and the second sequences of video images further comprises;

logically subdividing each of a sequence of images into a plurality of image portions;

encoding one or more of the image portions in each image of the sequence of images using a first compression format and encoding the remainder of the image portions in each image using a second compression format, the second compression format dependent on image portions previously-encoded using the first and/or the second compression formats;

generating a forward error correction (FEC) code for image portions encoded using at least the first compression format;

transmitting the first sequence of compressed video images and the FEC code with each of the image portions encoded using the first compression format to the first client,

transmitting the updated second sequence of compressed video images including the video and audio of the first user and the FEC codes with each of the image portions encoded using the first compression format to the second client,

wherein compressed video images that have more data than can be transmitted at a peak data rate in one image time are transmitted over one or more additional image times;

decompressing the first sequence of compressed video images on the first client and decompressing the updated second sequence of compressed video images including the video and audio of the first user on the second client and using at least some of the FEC codes to correct errors in at least some of the image portions;

rendering the video images contained in the first sequence of compressed video images on a display of the first client and rendering the video images contained in the updated second sequence of compressed video images including the video and audio of the first user on a display of the second client;

wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a latency such that both the first and second users have the perception that the selected video game or application is responding instantly to the signals.

2. The method as in claim 1 further comprising:

selecting a set of image portions compressed using the second compression format which are relatively more likely to have an impact on image rendering if lost than the remainder of the image portions compressed using the second compression format; and

generating a forward error correction (FEC) code for image portions within the selected set; and

transmitting the FEC code with each of the image portions within the second set to the first and second clients.

3. The method as in claim 1 further comprising:

generating FEC codes for audio content associated with the sequence of video images; and

transmitting the FEC codes and the audio content to the first and second clients.

4. The method as in claim 1 wherein the signals received by the one or more servers include control codes, the method further comprising:

receiving the control codes and FEC codes associated with the control codes, the control codes defining an operation to be performed related to the sequence of images;

decoding the FEC codes to determine whether an error exists in the control codes; and

if an error exists, attempting to fix the error using the FEC codes.

5. The method as in claim 1 wherein the first compression format comprises intraframe coding.

6. The method as in claim 5 wherein the intraframe coding comprises a Moving Pictures Experts Group (MPEG) I-frame.

7. The method as in claim 5 wherein the second compression format comprises interframe coding.

8. The method as in claim 7 wherein the interframe coding comprises an MPEG P-frame.

9. The method as in claim 1 further comprising:

receiving feedback information from the first client over the network, the feedback information usable to determine at least one characteristic of the first communication channel; and

receiving feedback information from the second client over the network, the feedback information usable to determine at least one characteristic of the second communication channel.

10. The method as in claim 9 wherein the feedback information is usable to determine whether data contained in the first and second sequences of compressed video images has not been successfully received and/or decoded.

11. The method as in claim 9 further comprising:

in response to determining the at least one characteristic of the first or second communication channels, then adjusting the compression of the video images.

12. The method as in claim 11 wherein adjusting the compression of the at least one sequence of video images includes modifying a frame rate of the sequence of video images.

13. The method as in claim 11 wherein adjusting the compression of the at least one sequence of video images includes modifying a resolution of the sequence of video images.

14. The method as in claim 1 further comprising:

detecting a level of packet loss or latency of compressed video images transmitted to the client; and

dynamically adjusting a data rate of the first and second sequences of compressed video images transmitted to the first and second clients, respectively, in response to the detected level of packet loss or latency.

15. The method as in claim 14 wherein the data rate of the first and second sequences of compressed video images transmitted to the first and second clients, respectively, is adjusted by modifying an image rate.

16. The method as in claim 14 wherein the data rate of the first and second sequences of compressed video images transmitted to the first and second clients, respectively, is adjusted by modifying compression quality.

17. The method as in claim 14 wherein the data rate of the first and second sequences of compressed video images transmitted to the first and second clients, respectively, is adjusted by modifying the image resolution of the compressed video images.

18. The method as in claim 17 wherein when the image resolution is modified, the first and/or second clients scale up the image to maintain the same image size on a display screen.

19. The method as in claim 1 wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a finite latency of 80 ms or less.

20. The method as in claim 1 wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a finite latency of 150 ms or less.

26. The method as in claim 1 wherein the one or more servers generate a first video output comprising a first sequence of video images of the video game or application for the first user and generate a second video output comprising a second sequence of video images for the second user,

wherein the first sequence of video images are compressed in accordance with the determined characteristics of the first client device and/or the first communication channel to generate the first compressed sequence of video images having the first set of characteristics, and

wherein the second sequence of video images are compressed in accordance with the determined characteristics of the second client device and/or the second communication channel to generate the second compressed sequence of video images having the second set of characteristics.

27. The method as in claim 26 wherein the characteristics of the first client device or the second client device includes a specified resolution of the display of the first client device or the second client device, respectively.

28. The method as in claim 26 wherein the determined characteristics of the first communication channel or the second communication channel include a current supported bitrate from the one or more servers to the first client or the second client, respectively.

29. The method as in claim 26 wherein a first server responsively executes the video game or application to generate the first video output comprising the first sequence of video images of the video game or application for the first user and a second server responsively executes the video game or application to generate the second video output comprising the second sequence of video images for the second user, the first server and the second server being communicatively interconnected over a local communication channel.

30. The method as in claim 29 further comprising:

a third server communicatively coupled to the first server and the second server and providing central control to the first server and the second server as the first server and second server execute the video game or application.

31. The method as in claim 1 further comprising:

generating third control signals from a third client in response to an input device of the third client being actuated by a third user, the third user spectating but not controlling the video game or application;

compressing the sequence of images of the video game or application to generate a third sequence of compressed video images;

transmitting the third sequence of compressed video images to the third client;

decompressing the third sequence of compressed video images on the third client; and

rendering the third sequence of compressed video images on a display of the third client.

21. A computer-implemented system comprising:

a first client device generating first control signals in response to user input from an input device of the first client device being actuated by a first user as the first user is playing a video game or using an application;

a first client device capturing video and audio of the first user as the first user is playing the video game or using the application;

a second client device generating second control signals in response to user input from an input device of the second client device being actuated by a second user as the second user is playing a video game or using an application;

a first network interface on the first client device transmitting the first control signals to one or more video game or application servers over a network which includes at least one public network component and transmitting the video and audio of the first user from the first client to the one or more servers;

a second network interface on the second client device transmitting the second control signals to one or more video game or servers over a network which includes at least one public network component;

a network component to route the first and second control signals to the one or more video game or application servers executing the video game or application;

the one or more video game or application servers receiving the first and second control signals related to the video game or application over the network which includes the at least one public network component, and responsively executing program code of the video game or application based on both the first and second control signals to generate video output comprising at least one sequence of video images of the video game or application;

the one or more servers determining characteristics of the first client and/or a first communication channel to the first client and determining characteristics of the second client and/or a second communication channel to the second client;

a low latency encoder to compress the at least one sequence of video images in accordance with the determined characteristics of the first client and/or first communication channel and generate a first sequence of compressed video images having a first set of characteristics,

the latency encoder to compress the at least one sequence of video images in accordance with the determined characteristics of the second client and/or second communication channel and generate a second sequence of compressed video images having a second set of characteristics,

incorporating the video and/or audio of the first user into the second sequence of video images compressed in accordance with the determined characteristics of the second client and/or the second communication channel to update the second sequence of compressed video images,

wherein compressing the at least one sequence of video images to generate the first and the second sequences of video images further comprises;

logically subdividing each of a sequence of images into a plurality of image portions;

encoding one or more of the image portions in each image of the sequence of images using a first compression format and encoding the remainder of the image portions in each image using a second compression format, the second compression format dependent on image portions previously-encoded using the first and/or the second compression formats;

FEC encoding logic to generate a forward error correction (FEC) code for image portions encoded using at least the first compression format;

wherein the first and updated second sequences of compressed video images including the video and audio of the first user and the FEC codes are transmitted with each of the image portions encoded using the first compression format to the first and second client devices, respectively,

wherein compressed video images that have more data than can be transmitted at a peak data rate in one image time are transmitted over one or more additional image times;

wherein the first client device decompresses the first sequence of compressed video images and the second client device decompresses the updated second sequence of compressed video images including the video and audio of the first user , the first and second clients using at least some of the FEC codes to correct errors in at least some of the image portions;

wherein the video images in the first and updated second sequences of video images including the video and audio of the first user are rendered on displays of the first and second client devices, respectively;

wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a latency such that both the first and second users have the perception that the selected video game or application is responding instantly to the signals.

22. The system as in claim 21 wherein the first compression format comprises intraframe coding.

23. The system as in claim 21 wherein the second compression format comprises interframe coding.

24. The system as in claim 21 wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a finite latency of 80 ms or less.

25. The system as in claim 21 wherein the operations of generating first and second control signals, capturing video and audio of the first user, transmitting the first and second control signals over the network, transmitting the video and audio of the first user, routing the first and second control signals transmitted over the network to one or more servers, executing the video game or application, compressing the video output using low latency compression to generate the first and second sequences of compressed video images, incorporating the video and/or audio of the first user into the second sequence of video images, generating the FEC codes, transmitting the first and second sequences of compressed video images and FEC codes to the first and second clients, respectively, decompressing the first and second sequences of compressed video images on the first and second clients, respectively, and using the FEC codes to correct errors in at least some of the tiles, and rendering the video images are performed with a finite latency of 150 ms or less.