Streaming interactive video integrated with recorded video segments

• US 9,573,059 B2
• Filed: 01/21/2015
• Issued: 02/21/2017
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., Laan, Roger van der
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): HALL, SHAUNA-KAY N
• Application Number:
  US14/601,494
• Publication Number:
  US 20150141145A1
• Time in Patent Office: 762 Days
• Field of Search: 463/31
• US Class Current: 1/1
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/115 : Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 : Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/154 : Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/166 : concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/587 : involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/21 : Server components or server architectures
• CPC Class Codes: H04N 21/2143 : located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/21805 : enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23 : Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/234363 : by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/234381 : by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/2368 : Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/2387 : Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/2743 : Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/2747 : Remote storage of video programs received via the downstream path, e.g. from the server
• CPC Class Codes: H04N 21/4341 : Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/47217 : for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04N 7/17318 : Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 7/17336 : Handling of requests in head-ends
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]
• CPC Class Codes: H04W 88/14 : Backbone network devices

Abstract

A method comprising executing a program on a server of a server center that generates and streams a user page in the form of streaming interactive video on a display of a user computer connected to the server via the Internet, the user page including a plurality of video windows, one or more of the video windows including streaming interactive video as well as simultaneously recorded video segments which are replayable by the user.

Claims

1. A non-transitory computer-readable storage medium encoded with a computer program, which, when executed, is operable to:

run a twitch video game or real-time application on one or more servers of a hosting service center, thereby generating uncompressed streaming interactive video;

compress the uncompressed streaming interactive video by a compression unit of the hosting service center, thereby producing compressed streaming interactive video, the twitch video game or real-time application being played or used by one or more users remote to the hosting service center via the compressed streaming interactive video, the compressed streaming interactive video being compressed with a worst-case, round-trip latency of 90 ms, from user input received at a client device until a response appears on a display associated with the client device, over a distance of up to 1500 miles from the hosting service center;

record, by the hosting service center, a copy of the compressed video output generated from the running of the twitch video game or real-time application, the recording occurring as the twitch video game or real-time application is being played or used, the copy being recorded in a buffer of the hosting service center; and

run a user interface application at the hosting service center that allows at least one of the one or more users to selectively replay any portion of the recorded copy of the compressed video output.

2. The non-transitory computer-readable storage medium of claim 1 wherein the user interface application further allows each of the one or more users to replay any portion of the recorded copy of the compressed video output with Digital Video Recorder (DVR) control functionality.

3. The non-transitory computer-readable storage medium of claim 2 wherein the DVR functionality includes pause, rewind, and fast forward functions.

4. The non-transitory computer-readable storage medium of claim 1 wherein the computer program, when executed, is further operable to allow, via the user interface application, each of the one or more users to control visual presentation aspects of the twitch video game or real-time application while the twitch video game or real-time application is being replayed, the visual presentation aspects including a camera view.

5. The non-transitory computer-readable storage medium of claim 1 wherein the computer program, when executed, is further operable to generate, by the hosting service center, different video for transmission to the user responsive to a change in the camera view.

6. The non-transitory computer-readable storage medium of claim 1 wherein the computer program, when executed, is further operable to allow, via the user interface application, each of the one or more users to post a specified video segment of the recorded copy of the compressed video output on a user page available for viewing by other users.

7. The non-transitory computer-readable storage medium of claim 1 wherein the computer program, when executed, is further operable to allow, via the user interface application, a user viewing the specified video segment to rate the specified video segment.

8. The non-transitory computer-readable storage medium of claim 7 wherein the computer program, when executed, is further operable to overdub audio onto the specified video segment.

9. The non-transitory computer-readable storage medium of claim 1 wherein replay of the recorded copy of the compressed video output loops continuously.

10. The non-transitory computer-readable storage medium of claim 1 wherein the compressed streaming interactive video is in a different format than the recorded copy of the compressed video output.

11. The non-transitory computer-readable storage medium of claim 10 wherein the different format comprises a different compression algorithm.

12. The non-transitory computer-readable storage medium of claim 10 wherein the different format comprises a different screen resolution.

13. The non-transitory computer-readable storage medium of claim 10 wherein the different format comprises a different frame rate.

14. The non-transitory computer-readable storage medium of claim 1 wherein the compressed streaming interactive video comprises a bidirectional stream and the recorded copy of the compressed video output comprises a unidirectional stream.

15. A non-transitory computer-readable storage medium encoded with a computer program, which, when executed, is operable to:

run a twitch video game or real-time application on one or more servers of a hosting service center, the twitch video game or real-time application being played by a user remote to the hosting service center via compressed streaming interactive video transmitted over the Internet, the compressed streaming interactive video being compressed with a worst-case, round-trip latency of 90 ms, from user input received at a client device until a response appears on a display associated with the client device, over a distance of up to 1500 miles from the hosting service center, the running of the twitch video game or real-time application at the hosting service center producing multiple compressed video streams, at least one of the multiple compressed video streams comprising a bidirectional stream, and at least another of the multiple compressed video streams comprising a unidirectional stream; and

broadcast the unidirectional stream over the Internet to multiple destinations, the multiple destinations including at least one spectator of the twitch video game or real-time application.

16. The non-transitory computer-readable storage medium of claim 15 wherein the bidirectional stream is adaptive.

17. The non-transitory computer-readable storage medium of claim 15 wherein the unidirectional stream is non-adaptive.

18. The non-transitory computer-readable storage medium of claim 15 wherein the at least one of the multiple video streams recorded is the unidirectional stream.

19. The non-transitory computer-readable storage medium of claim 15 wherein the at least one of the multiple destinations is a delay buffer of the hosting service center.

20. A computer-implemented method comprising:

running a twitch video game or real-time application on one or more servers of a hosting service center, thereby generating uncompressed streaming interactive video;

compressing the uncompressed streaming interactive video, thereby producing compressed streaming interactive video;

transmitting the compressed streaming interactive video over a packetized network to one or more client devices associated with one or more users remote to the hosting service center, wherein the compressed streaming interactive video is compressed with a worst-case, round-trip latency of 90 ms, from user input received at a client device until a response appears on a display associated with the client device, over a distance of up to 1500 miles from the hosting service center;

recording, by the hosting service center, a copy of the compressed video output generated from the running of the twitch video game or real-time application, the recording occurring as the twitch video game or real-time application is being played or used, the copy being recorded in a buffer of the hosting service center; and

running a user interface application at the hosting service center that allows at least one of the one or more users to selectively replay any portion of the recorded copy of the compressed video output.

21. The computer-implemented method of claim 20 wherein the user interface application further allows each of the one or more users to replay any portion of the recorded copy of the compressed video output with Digital Video Recorder (DVR) control functionality.

22. The computer-implemented method of claim 21 wherein the DVR functionality includes pause, rewind, and fast forward functions.

23. The computer-implemented method of claim 20 wherein the user interface application further allows each of the one or more users to control visual presentation aspects of the twitch video game or real-time application while the twitch video game or real-time application is being replayed, the visual presentation aspects including a camera view.

24. The computer-implemented method of claim 23 further comprising generating, by the hosting service center, different video for transmission to the user responsive to a change in the camera view.

25. The computer-implemented method of claim 20 wherein the user interface application further allows each of the one or more users to post a specified video segment of the recorded copy of the compressed video output on a user page available for viewing by other users.

26. The computer-implemented method of claim 25 wherein the user interface application further allows a user viewing the specified video segment to rate the specified video segment.