System and method for streaming game video

• US 9,669,307 B2
• Filed: 04/12/2016
• Issued: 06/06/2017
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Perlman, Stephen G., van der Laan, Roger, Cotter, Timothy, Furman, Scott, McCool, Robert, Buckley, Ian
• Primary Examiner(s): Suhol, Dmitry
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US15/097,207
• Publication Number:
  US 20160220901A1
• Time in Patent Office: 420 Days
• Field of Search: None
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/33 using wide area network [WAN] connections
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 13/493 Resuming a game, e.g. after pausing, malfunction or power failure
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 2300/402 Communication between platforms, i.e. physical link to protocol
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: A63F 2300/554 by saving game or status data
• CPC Class Codes: A63F 2300/556 Player lists, e.g. online players, buddy list, black list
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/611 : for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 : adapting media to network capabilities
• CPC Class Codes: H04L 65/765 : intermediate
• CPC Class Codes: H04L 67/131 : Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/107 : between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 : Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/146 : Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/169 : characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
• CPC Class Codes: H04N 19/172 : the region being a picture, frame or field
• CPC Class Codes: H04N 19/188 : the unit being a video data packet, e.g. a network abstraction layer [NAL] unit
• CPC Class Codes: H04N 19/436 : using parallelised computational arrangements
• CPC Class Codes: H04N 19/61 : in combination with predictive coding
• CPC Class Codes: H04N 21/233 : Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/2343 : involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2381 : Adapting the multiplex stream to a specific network, e.g. an Internet Protocol [IP] network transmission of MPEG streams over ATM H04L12/5601
• CPC Class Codes: H04N 21/4781 : Games
• CPC Class Codes: H04N 21/6125 : involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/6405 : Multicasting data broadcast and multicast in packet switching networks H04L12/18
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server
• CPC Class Codes: H04N 21/6587 : Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/106 : for domestic distribution

Abstract

A system and method are described for managing the state of an online video game. A method includes initiating a new online video game in response to user input from a client device, the online video game being in a first state on a first server when initiated; executing the online video game on the server, thereby causing the online video game to enter a second state; pausing or terminating the online video game; determining differences between the first state and the second state and generating difference data containing the differences; transmitting the difference data over a network to a second server; and recreating the second state from the difference data and the first state in response to user input indicating that the user wishes to resume the online video game and in response to the second server being selected as the server on which to execute the video game.

Claims

1. A server computing device, comprising:

a processor configured to execute a game application in accordance with signals received from a client computing device to generate a sequence of video images corresponding to graphical output of the executed game application, the client computing device remotely located from the server computing device;

a network connection configured to connect the server computing device to a network over which data is transmittable to the client computing device;

a data storage device configured to store data associated with graphical output of the executed game application; and

an encoder configured to compress the sequence of video images corresponding to graphical output of the executed game application into at least two different video streams of different formats including a live video stream and a high quality video stream, the live video stream having a format based on characteristics of the network over which data is transmittable to the client computing device and based on video processing and display capabilities of the client computing device, the high quality video stream being of higher quality than the live video stream, the encoder configured to provide the live video stream to the network connection for transmission over the network to the client computing device, the encoder also configured to simultaneously provide the high quality video stream to the data storage device for storage therein in support of video playback functions as requested by the client computing device.

2. The server computing device as recited in claim 1, wherein the format of the high quality video stream includes insertion of an I-Frame at a constant periodicity, and wherein an available rate of rewinding of the graphical output of the executed game application from a specified time is a multiple of the constant periodicity when the available rate of rewinding exceeds the constant periodicity, and wherein an available rate of a fast-forwarding of the graphical output of the executed game application from a specified time is a multiple of the constant periodicity when the available rate of fast-forwarding exceeds the constant periodicity.

3. The server computing device as recited in claim 1, wherein the format of the live video stream includes P-Frames.

4. The server computing device as recited in claim 3, wherein the format of the high quality video stream includes a periodic insertion of an I-Frame.

5. The server computing device as recited in claim 4, wherein at least one I-Frame is inserted within every 12 successive frames in the high quality video stream.

6. The server computing device as recited in claim 4, wherein the format of the high quality video stream includes P-Frames positioned between the periodic insertion of the I-Frame.

7. The server computing device as recited in claim 3, wherein the format of the high quality video stream includes only I-Frames.

8. The server computing device as recited in claim 1, further comprising:

stream replay logic configured to direct transmission of the high quality video stream from the data storage device to the network connection for transmission over the network to the client computing device in accordance with a video playback request received from the client computing device.

9. The server computing device as recited in claim 8, the video playback request specifying playback of the graphical output of the executed game application from a specified time, or rewinding of the graphical output of the executed game application from a specified time, or fast-forwarding of the graphical output of the executed game application from a specified time, or pausing of the graphical output of the executed game application at a specified time.

10. The server computing device as recited in claim 9, wherein the format of the high quality video stream includes insertion of an I-Frame at a constant periodicity, and wherein a rate of rewinding of the graphical output of the executed game application from the specified time is a multiple of the constant periodicity when the rate of rewinding exceeds the constant periodicity, and wherein a rate of fast-forwarding of the graphical output of the executed game application from the specified time is a multiple of the constant periodicity when the rate of fast-forwarding exceeds the constant periodicity.

11. The server computing device as recited in claim 8, wherein the stream replay logic is configured to decode a portion of the high quality video stream present within the data storage device corresponding to the video playback request and then re-encode the decoded portion of the high quality video stream into a format suitable for transmission over the network to the client computing device.

12. The server computing device as recited in claim 8, wherein the stream replay logic is configured to directly provide a portion of the high quality video stream present within the data storage device corresponding to the video playback request to the network connection for transmission over the network to the client computing device.

13. A method for operating a server computing device, comprising:

executing a game application on a processor in accordance with signals received from a client computing device to generate a sequence of video images corresponding to graphical output of the executed game application, the client computing device remotely located from the server computing device;

operating an encoder configured to compress the sequence of video images corresponding to graphical output of the executed game application into at least two different video streams of different formats including a live video stream and a high quality video stream, the live video stream having a format based on characteristics of a network over which data is transmittable to the client computing device and based on video processing and display capabilities of the client computing device, the high quality video stream being of higher quality than the live video stream;

operating the encoder to provide the live video stream to a network connection for transmission over the network to the client computing device; and

operating the encoder to simultaneously provide the high quality video stream to a data storage device for storage therein in support of video playback functions as requested by the client computing device.

14. The method as recited in claim 13, wherein the format of the high quality video stream includes insertion of an I-Frame at a constant periodicity, and wherein an available rate of rewinding of the graphical output of the executed game application from a specified time is a multiple of the constant periodicity when the available rate of rewinding exceeds the constant periodicity, and wherein an available rate of fast-forwarding of the graphical output of the executed game application from a specified time is a multiple of the constant periodicity when the available rate of fast-forwarding exceeds the constant periodicity.

15. The method as recited in claim 13, wherein the format of the live video stream includes P-Frames.

16. The method as recited in claim 15, wherein the format of the high quality video stream includes a periodic insertion of an I-Frame.

17. The method as recited in claim 16, wherein at least one I-Frame is inserted within every 12 successive frames in the high quality video stream.

18. The method as recited in claim 16, wherein the format of the high quality video stream includes P-Frames positioned between the periodic insertion of the I-Frame.

19. The method as recited in claim 15, wherein the format of the high quality video stream includes only I-Frames.

20. The method as recited in claim 13, further comprising:

directing transmission of the high quality video stream from the data storage device to the network connection for transmission over the network to the client computing device in accordance with a video playback request received from the client computing device.

21. The method as recited in claim 20, wherein the video playback request specifies playback of the graphical output of the executed game application from a specified time, or rewinding of the graphical output of the executed game application from a specified time, or fast-forwarding of the graphical output of the executed game application from a specified time, or pausing of the graphical output of the executed game application at a specified time.

22. The method as recited in claim 21, wherein the format of the high quality video stream includes insertion of an I-Frame at a constant periodicity, and wherein a rate of rewinding of the graphical output of the executed game application from the specified time is a multiple of the constant periodicity when the rate of rewinding exceeds the constant periodicity, and wherein a rate of fast-forwarding of the graphical output of the executed game application from the specified time is a multiple of the constant periodicity when the rate of fast-forwarding exceeds the constant periodicity.

23. The method as recited in claim 20, wherein directing transmission of the high quality video stream from the data storage device to the network connection includes decoding a portion of the high quality video stream present within the data storage device corresponding to the video playback request and then re-encoding the decoded portion of the high quality video stream into a format suitable for transmission over the network to the client computing device.

24. The method as recited in claim 20, wherein directing transmission of the high quality video stream from the data storage device to the network connection includes directly providing a portion of the high quality video stream present within the data storage device corresponding to the video playback request to the network connection for transmission over the network to the client computing device.