System for combining recorded application state with application streaming interactive video output

• US 10,286,315 B2
• Filed: 04/29/2014
• Issued: 05/14/2019
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger
• Primary Examiner(s): Pendleton, Brian T
• Assistant Examiner(s): Saint Cyr, Jean D
• Application Number:
  US14/264,492
• Publication Number:
  US 20140237512A1
• Time in Patent Office: 1,841 Days
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/497 Partially or entirely replaying previous game actions
• CPC Class Codes: A63F 13/525 Changing parameters of virtual cameras
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/2143 located in a single building, e.g. hotel, hospital or museum arrangements specially adapted for plural spots in a confined site in broadcast systems H04H20/63; adaptations for transmission by electric cable for domestic distribution in television systems H04N7/106
• CPC Class Codes: H04N 21/21815 comprising local storage units
• CPC Class Codes: H04N 21/231 Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/4147 PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/47202 for requesting content on demand, e.g. video on demand
• CPC Class Codes: H04N 21/47217 : for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/64322 : IP
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/8173 : End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 7/106 : for domestic distribution
• CPC Class Codes: H04N 9/79 : Processing of colour television signals in connection with recording
• CPC Class Codes: H04N 9/87 : Regeneration of colour television signals H04N9/80 takes precedence

Abstract

An apparatus for recording application state information over time while simultaneous recording application streaming interactive video output. A playback means in which the recorded application video may be played back or the application may be re-executed with the application state information.

Claims

1. A computer-implemented method comprising:

running a real-time application on one or more servers of a hosting service center, wherein the real-time application is a game and wherein the real-time application is being played by a first user of a first client device remote to the hosting service center via compressed streaming interactive video transmitted over the Internet;

continually storing, in a delay buffer of the one or more servers, application state information of the real-time application along with the compressed streaming interactive video as the real-time application is being played, the application state information as stored in the delay buffer is usable to generate additional views of the real-time application for replay; and

running a user interface (UI) application on another server of the hosting service center, the UI application allowing a second user of a second client device remote to the hosting service center to,

rewind based on the application state information;

replay a segment of the played real-time application based on the application state information stored in the delay buffer; and

control a camera view for the replay of the segment of the played real-time application, the control of the camera view causing dynamic generating of video frames by processing the application state information as stored in the delay buffer for one or more additional fly-through perspectives for the segment of the played real-time application;

wherein the one or more additional fly-through perspectives for the segment that is replayed to the second user of the second client device provided by the dynamic generating of video frames are different from a view of the real-time application displayed to the first user on the first client device.

2. The computer-implemented method of claim 1, wherein the application state information supports a plurality of replays with changes in camera views from different fly-through perspectives.

3. The computer-implemented method of claim 1, wherein the control of the camera view is accomplished through Digital Video Recorder (DVR) control functionality.

4. The computer-implemented method of claim 3 wherein the DVR control functionality includes pause, rewind, and fast forward functions.

5. The computer-implemented method of claim 4 wherein the DVR control functionality includes 3D DVR.

6. The computer-implemented method of claim 1 wherein the UI application allows the second user to choose a random selection of games to view at once.

7. The computer-implemented method of claim 1 wherein different video is generated when the application state information is being played and the camera view changes.

8. The computer-implemented method of claim 1 further comprising allowing the first user to specify an application state segment to record and retain.

9. The computer-implemented method of claim 8 further comprising making the application state segment available on a user page.

10. The computer-implemented method of claim 1 wherein the application state information includes video output by the real-time application streamed to the users of the real-time application.

11. The computer-implemented method of claim 1 wherein the application state information includes different video output by the real-time application than that streamed to the users of the real-time application.

12. The computer-implemented method of claim 2 wherein the reproduced video is uncompressed.

13. The computer-implemented method of claim 2 wherein the reproduced video is compressed.

14. One or more computer-readable media that are non-transitory and that store a program, wherein the program, when executed, instructs a processor to perform the following operations:

run a real-time application on one or more servers of a hosting service center, wherein the real-time application is a game and wherein the real-time application is being played by a first user of a first client device remote to the hosting service center via compressed streaming interactive video transmitted over the Internet and a second user of a second client device;

continually storing, in a delay buffer of the one or more servers, application state information of the real-time application along with the compressed streaming interactive video as the real-time application is being played, the application state information as stored in the delay buffer is usable to generate additional views of the real-time application for replay; and

running a user interface (UI) application on another server of the hosting service center, the UI application allowing a second user of a second client device remote to the hosting service center to,

rewind based on the application state information;

replay a segment of the played real-time application based on the application state information stored in the delay buffer; and

control a camera view for the replay of the segment of the played real-time application, the control of the camera view causing dynamic generating of video frames by processing the application state information as stored in the delay buffer for one or more additional fly-through perspectives for the segment of the played real-time application;

wherein the one or more additional fly-through perspectives for the segment that is replayed to the second user of the second client device provided by the dynamic generating of video frames are different from a view of the real-time application displayed to the first user on the first client device.

15. The computer-readable media of claim 14 further comprising using the application state information to reproduce video of the real-time application.

16. The computer-readable media of claim 15 wherein the reproduced video is compressed.

17. The computer-readable media of claim 14 wherein the UI application allows the second user to choose a random selection of games to view at once.

18. The computer-readable media of claim 14 wherein different video is generated when the application state information is being played and the camera view changes.

19. A computer-implemented method comprising:

running a real-time application on one or more servers of a hosting service center, wherein the real-time application is a game and wherein the real-time application is being played by a first user of a first client device and a second user of a second client device, wherein the first and second client devices are remote to the hosting service center and wherein the real-time application is being played via compressed streaming interactive video transmitted over the Internet;

continually storing, in a delay buffer of the one or more servers, application state information of the real-time application along with the compressed streaming interactive video as the real-time application is being played by the first user, the application state information is usable to generate additional views of the real-time application for replay; and

running a user interface (UI) application on another server of the hosting service center, the UI application allowing the second user of the second client device remote to the hosting service center to,

rewind based on the application state information;

replay a segment of the played real-time application based on the application state information stored in the delay buffer; and

control a camera view for the replay of the segment of the played real-time application, the control of the camera view causing dynamic generating of video frames by processing the application state information as stored in the delay buffer for one or more additional fly-through perspectives for the segment of the played real-time application;

wherein the one or more additional fly-through perspectives for the segment that is replayed to the second user of the second client device provided by the dynamic generating of video frames are different from a view of the real-time application displayed to the first user on the first client device or the second user on the second client device.

20. The method of claim 19, wherein the one or more additional fly-through perspectives for the segment includes video of the first user as the first user plays the real-time application.