Hosting and broadcasting virtual events using streaming interactive video

• US 10,150,030 B2
• Filed: 08/10/2015
• Issued: 12/11/2018
• Est. Priority Date: 12/05/2007
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Salce, Jason P
• Application Number:
  US14/822,833
• Publication Number:
  US 20160030841A1
• Time in Patent Office: 1,219 Days
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 2009/245 visual
• CPC Class Codes: A63F 2009/247 audible, e.g. using a loudspeaker
• CPC Class Codes: A63F 2009/2488 Remotely playable
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 : between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 : Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/2143 : located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/262 : Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/4307 : Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
• CPC Class Codes: H04N 21/43074 : of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/439 : Processing of audio elementary streams
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04W 28/16 : Central resource management; Negotiation of resources or communication parameters, e.g. negotiating bandwidth or QoS [Quality of Service]
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

Methods for hosting and operating a twitch service and systems are provided. One method includes running a twitch video game or real-time application at a hosting service center. The twitch video game or real-time application is configured to be played or used by a user on a client device remote to the hosting service center via compressed streaming interactive video. The hosting service center processes input received from the user, wherein the compressed streaming interactive video is compressed with a worst-case, round-trip latency from a user control input to display of a response to the user control input on the client device of the user. The method further includes broadcasting the twitch action video game or real-time application in one or more streams of real-time compressed digital video from the hosting service center to one or more viewing client devices over the Internet. The method also includes selectively overlaying, by the hosting service center, feedback data (e.g., text, comments, audio, etc.) from one or more of the viewing client devices onto the one or more streams of real-time compressed digital video.

Claims

1. A method comprising:

running one or more twitch video games or real-time applications on servers of a hosting service center, the one or more twitch video games or real-time applications being played or used by one or more users on one or more client devices remote to the hosting service center via compressed streaming interactive video, the compressed streaming interactive video being compressed to achieve a worst-case round-trip latency from a user control input to display of a response to the user control input on the client device of the user, over a transmission distance between the one or more client devices and a server of the hosting service center;

broadcasting the one or more twitch video games or real-time applications as one or more streams of real-time compressed digital video from the hosting service center to one or more viewing client devices over the Internet, and wherein the one or more viewing client devices receives one or more of the streams of real-time compressed digital video; and

overlaying with the streams of real-time compressed digital video, audio from one or more of the viewing client devices, such that the audio is included in the broadcasting of the one or more twitch video games or real-time applications, and wherein the compression is adjusted to reduce a data rate to one or more of the client devices to maintain said streams of real-time compressed digital video within said worst-case round trip latency.

2. The method of claim 1 further comprising selectively delaying the one or more streams of real-time compressed digital video before broadcasting to the one or more viewing client devices.

3. The method of claim 1 wherein the worst case latency is achieved in part by a frame computation process of the one or more of the servers responsive to the user control input, and a video compression process which produces the compressed streaming interactive video stream.

4. The method of claim 1 wherein one of the one or more twitch video games or real-time applications comprises a video game tournament.

5. The method of claim 1 wherein the one or more streams of real-time compressed digital video comprises one or more IP multicasts within the hosting service center.

6. The method of claim 1 further comprising sending, by the hosting service center, the one or more streams of real-time compressed digital video to a Content Delivery Network (CDN) for distribution to the viewing client devices over the Internet.

7. The method of claim 1, further comprising,

overlaying, by the hosting service center, text comments received from one or more of the viewing client devices onto the one or more streams of real-time compressed digital video.

8. A method comprising:

running a twitch video game or real-time application at a hosting service center, the twitch video game or real-time application being played or used by a user on a client device remote to the hosting service center via compressed streaming interactive video, the hosting service center processing input received from the user wherein the compressed streaming interactive video is compressed with a worst-case round-trip latency from a user control input to display of a response to the user control input on the client device of the user; and

broadcasting the twitch action video game or real-time application in one or more streams of real-time compressed digital video from the hosting service center to one or more viewing client devices over the Internet;

overlaying, by the hosting service center, feedback data from one or more of the viewing client devices onto the one or more streams of real-time compressed digital video, said feedback data includes audio and text data received from users of one or more of said viewing client devices; and

adjusting said compression to maintain said worst-case round trip latency of the one or more streams of real-time compressed digital video.

9. The method of claim 8 further comprising selectively delaying the one or more streams of real-time compressed digital video broadcast to the one or more viewing client devices.

10. The method of claim 8 wherein uncompressed video generated by the one or more servers is utilized by the hosting service center in one or more forms in addition to the compressed streaming interactive video.

11. The method of claim 10 wherein the one of the one or more forms comprises a compressed digital video stream generated without client feedback.

12. The method of claim 8 wherein the one or more streams of real-time compressed digital video adapts dynamically to characteristics of an Internet channel between the hosting service center and the one or more viewing client devices.

13. The method of claim 8 wherein the one or more streams of real-time compressed digital video comprises one or more IP multicasts within the hosting service center.

14. The method of claim 8 further comprising sending, by the hosting service center, the one or more streams of real-time compressed digital video to a Content Delivery Network (CDN) for distribution to the one or more viewing client devices.

15. The method of claim 8, wherein the feedback data includes one or more of text data, audio data, video data, comments data, or two or more thereof.

16. The method of claim 8, wherein the hosting service center comprises one or more servers that run the twitch video game or real-time application, wherein the user interacts with the twitch video game or real-time application via the compressed streaming interactive video encoded from uncompressed video generated by the one or more servers.