System and method for compressing streaming interactive video

• US 8,893,207 B2
• Filed: 12/05/2007
• Issued: 11/18/2014
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: OL2 Incorporated (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): VU, NGOC K
• Application Number:
  US11/999,461
• Publication Number:
  US 20090119736A1
• Time in Patent Office: 2,540 Days
• Field of Search: 375/240, 375/240.01, 725/91, 725/114, 725/116, 725/133, 725/141, 725/153
• US Class Current: 725/116
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 47/25 with rate being modified by the source upon detecting a change of network conditions
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/169 characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 19/85 : using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/226 : Characteristics of the server or Internal components of the server
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 28/06 : Optimizing the usage of the radio link, e.g. header compression, information sizing , discarding information system modifying transmission characteristic according to link quality by modifying frame length H04L1/0007; dynamic adaptation of the packet size for flow control or congestion control H04L47/365
• CPC Class Codes: H04W 4/00 : Services specially adapted for wireless communication networks; Facilities therefor

Abstract

A server center for hosting low-latency streaming interactive audio/video (NV) includes a plurality of servers that run one or more applications an inbound routing network that receives packet streams from users via a first network interface and routes the packets to one or more of the servers. The packet streams include user control input. One or more of the servers are operable to compute A/V data responsive to the user control input. A compression unit is coupled to receive the A/V data from the one or more of the servers and output compressed A/V data therefrom. An output routing network that routes the compressed A/V data to each of the users over a corresponding communication channel via a second interface, the compression unit is operable to modify a compression rate responsive to current characteristics of the corresponding communication channel for each user so as to optimize performance of the one or more applications.

Claims

1. A server center for hosting low-latency streaming interactive video, comprising:

a plurality of servers configured to run one or ore twitch video games or applications;

an inbound routing network that receives packet streams from client devices via a first network interface and routes the packet streams to one or more of the servers, the packet streams including user control input to at least one of the one or more twitch video games or applications, one or more of the servers being operable to compute video data responsive to the user control input, the client devices being located a remote distance beyond a premises where any one of the plurality of servers is located;

a compression unit coupled to receive the video data from the one or more of the servers and output compressed streaming interactive video therefrom; and

an output routing network operable to route the compressed streaming interactive video to each of the client devices over a corresponding communication channel via a second network interface coupled to the Internet, the client devices not running the one or more twitch video games or applications locally;

wherein the compressed streaming interactive video is compressed with a worst-case, round-trip latency of 90 milliseconds, from the user control input to display of a response to the user control input on a client device of a user, over a transmission distance of up to 1500 miles.

2. The server center of claim 1 wherein at least one of the client devices lacks a processing capability needed to execute the twitch video game locally.

3. The server center of claim 2 wherein at least one of the client devices comprises a set-top box operable to connect to the Internet.

4. The server center of claim 2 wherein at least one of the client devices comprises a mobile device operable to wirelessly connect to the Internet.

5. The server center of claim 1 wherein one or more of the servers includes a graphics processing unit.

6. The server center of claim 1 wherein plurality of servers are distributed geographically.

7. The server center of claim 1 wherein one or more of the plurality of servers are configured with identical performance characteristics.

8. The server center of claim 1 wherein one or more of he plurality of servers are configured with different performance characteristics.

9. The server center of claim 2 wherein the processing capability is sufficient to forward keyboard/controller input to the Internet and decode/decompress the compressed streaming interactive video data received.

10. The server center of claim 1 further comprising mass storage shared by the plurality of servers, the mass storage including a RAID or a SAN.

11. The server center of claim 10 wherein the mass storage stores the one or more twitch video games or applications.

12. The server center of claim 1 wherein output routing network is further operable to route uncompressed audio associated with the one or more twitch video games or applications to each of the client devices over the Internet.

13. The server center of claim 1 wherein the corresponding communications channel comprises a routing path over the Internet.

14. The server center of claim 1 wherein the servers are cooperatively arranged in a parallel processing configuration.

15. The server center of claim 1 wherein the compression unit is further operable to modify a compression rate responsive to current characteristics of the corresponding communication channel for each user so as to optimize performance.

16. An apparatus comprising:

a server unit that runs a twitch video game or real-time application to produce uncompressed streaming interactive video; and

a compression unit coupled to receive the uncompressed streaming interactive video from the server unit and output compressed streaming interactive video therefrom, the server unit and the compression unit being located at a premises, the compressed streaming interactive video being delivered over a packetized network from the premises to client devices associated with users, each client device being located geographically remote to the premises, each client device not running the twitch video game or real-time application locally;

wherein the compressed streaming interactive video is compressed with a worst-case, round-trip latency of 90 milliseconds, from the user control input to display of a response to the user control input on a client device of a user, over a transmission distance of up to 1500 miles.

17. The apparatus of claim 16 wherein the twitch video game or real-time applications includes a television or movie application.

18. The apparatus of claim 16 wherein the server unit is configured to receive video control commands from at least one of the client devices, the video control commands including at least one of pause, fast forward, rewind commands.

19. The apparatus of claim 16 wherein the server unit is configured to receive live broadcasts through a network connection, a satellite connection, a media player, or a cable television connection.

20. The apparatus of claim 16 further comprising a storage unit to store recorded media content.

21. The apparatus of claim 20 wherein the recorded media content includes movies and/or programs.

22. The apparatus of claim 20 wherein the recorded media content includes personal data of the users.

23. The apparatus of claim 22 wherein the personal data includes photographs.

24. The apparatus of claim 20 wherein the recorded media content includes video games.

25. The apparatus of claim 16 further comprising an interface providing a connection to the packetized network.

26. The apparatus of claim 25 wherein the packetized network is the Internet, the compressed streaming interactive video being delivered from the premises to the client devices via the Internet.

27. An apparatus comprising:

a server unit that runs a twitch video game to produce uncompressed video data therefrom;

a compression unit coupled to receive the video data from the server unit and output compressed streaming interactive video therefrom, the compressed streaming interactive video being compressed with a worst-case, round-trip latency of 90 milliseconds, from user control input to display of a response to the user control input on a client device of a user, over a transmission distance of up to 1500 miles; and

wherein the server unit and the compression unit are located at a premises, the compressed streaming interactive video being delivered over a packetized network from the premises to client devices associated with users, each client device being located geographically remote to the premises, each client device not running the twitch video game locally.

28. The apparatus of claim 27 wherein the server unit is configured to receive video control commands from at least one of the client devices, the video control commands including at least one of pause, fast forward, rewind commands.

29. The apparatus of claim 27 wherein the server unit is configured to receive live broadcasts through a network connection, a satellite connection, a media player, or a cable television connection.

30. The apparatus of claim 27 further comprising a storage unit to store recorded media content.

31. The apparatus of claim 30 wherein the recorded media content includes movies and/or programs.

32. The apparatus of claim 30 wherein the recorded media content includes personal data of the users.

33. The apparatus of claim 32 wherein the personal data includes photographs.

34. The apparatus of claim 27 further comprising a transceiver operable to deliver the compressed streaming interactive video to the client devices over the packetized network.

35. The apparatus of claim 27 wherein the packetized network comprises the Internet.