System and method for combining multiple game or application views into a single media stream

• US 9,700,789 B2
• Filed: 03/12/2013
• Issued: 07/11/2017
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Cotter, Timothy
• Primary Examiner(s): SKAARUP, JASON M
• Application Number:
  US13/797,271
• Publication Number:
  US 20140274368A1
• Time in Patent Office: 1,582 Days
• Field of Search: 463 31
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/795 for finding other players; for building a team; for providing a buddy list
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams

Abstract

A system and method are described for combining multiple views of a video game or application into a single video stream (or set of video streams). For example, a system according to one embodiment comprises: a plurality of application/video game instances executed for a particular multi-player video game or application on a hosting service, the application/video game instances being executed in response to control signals received from a plurality of clients; compositing logic executed at the hosting service to combine video streams generated by each of the application/video game instances into a single video stream for a first one of the clients; and compression logic to compress the single video stream with low latency such that the user of the first one of the clients has the perception that the video game or application is being executed locally.

Claims

1. A system comprising:

a plurality of application/video game instances executed for a particular multi-player video game or application on a hosting service that includes one or more servers, the application/video game instances being executed in response to control signals received from a plurality of clients;

compositing logic executed by one or more of the servers of the hosting service to combine video streams generated by each of the application/video game instances into a single video stream for a first one of the clients; and

compression logic of a hardware compressor of the hosting service operating in communication with the one or more servers to compress the single video stream with low latency such that the user of the first one of the clients has a perception that the video game or application is being executed locally, the hardware compressor is configured to dynamically adjust a resolution of the single video stream sent to the first one of the clients based on bandwidth feedback received from the first one of the clients, the bandwidth feedback includes a current state of a communication channel between the first one of the clients and at least one of the servers of the hosting service, wherein the resolution being dynamically adjusted is based on the current state of the communication channel, so as to maintain the perception that the video game or application is being executed locally;

wherein the compositing logic generates a separate video stream for each client, each video stream having a primary view generated from the perspective of a user of the client and a plurality of secondary views generated from the perspectives of the other users participating in the multi-player video game or application, wherein the hardware compressor is configured to individually compress said separate video stream to each one of the clients and dynamically adjust the resolution of each one of the clients independently of one another so that each one of the clients maintains a substantial perception that the video game or application is being executed locally for the respective client.

2. The system as in claim 1 wherein at least some of the clients receive video streams having a single view and other of the clients receive view streams having multiple-views generated by the compositing logic.

3. The system as in claim 1 wherein the views combined by the compositing logic are consistent for the view streams generated for each of the clients.

4. The system as in claim 1 wherein each of the client devices have different view processing capabilities.

5. The system as in claim 4 wherein at least some of the client devices are touch-screen tablet devices.

6. The system as in claim 4 wherein at least some of the client devices are smartphone devices.

7. A method comprising:

executing a plurality of application/video game instances for a particular multi-player video game or application on a hosting service having one or more servers, the application/video game instances being executed in response to control signals received from a plurality of clients;

combining video streams generated by each of the application/video game instances into a single video stream for a first one of the clients, the combining being processed by one or more of the servers;

compressing the single video stream with low latency such that the user of the first one of the clients has a perception that the video game or application is being executed locally, wherein the compressing is executed by a hardware compressor of the hosting service, the compressing is configured to dynamically adjust a resolution of the single video stream sent to the first one of the clients based on bandwidth feedback received from the first one of the clients so as to maintain the perception that the video game or application is being executed locally, the bandwidth feedback includes a current state of a communication channel between the first one of the clients and at least one of the servers of the hosting service; and

generating a separate video stream for each client, each video stream having a primary view generated from the perspective of a user of the client and a plurality of secondary views generated from the perspectives of the other users participating in the multi-player video game or application, wherein the hardware compressor is configured to individually compress said separate video stream to each one of the clients and dynamically adjust the resolution of each one of the clients independently of one another so that each one of the clients maintains a substantial perception that the video game or application is being executed locally for the respective client.

8. The method as in claim 7 wherein at least some of the clients receive video streams having a single view and other of the clients receive view streams having multiple-views generated by the combining.

9. The method as in claim 7 wherein the views combined are consistent for the view streams generated for each of the clients.

10. The method as in claim 7 wherein each of the client devices have different view processing capabilities.

11. The method as in claim 10 wherein at least some of the client devices are touch-screen tablet devices.

12. The method as in claim 10 wherein at least some of the client devices are smartphone devices.

13. A system for processing online video gaming operations, the online gaming operations enable serving video games to one or more clients, comprising:

a plurality of video game instances executed for a particular multi-player video game on a hosting service of the online video gaming operations, the hosting service has one or more servers located in one or more data centers, the video game instances being executed in response to control signals received from respective ones of a plurality of clients;

compositing logic executed by one or more of the servers of the hosting service to combine video streams generated by each of the video game instances into one or more single video streams, each single video stream generated for each one of the plurality of clients; and

a hardware compressor operating in communication with the one or more servers to compress the single video stream with low latency such that each of the users of the clients has a perception that the video game is being executed locally, the hardware compressor is configured to dynamically adjust a resolution of each of the single video streams sent to the clients based on bandwidth feedback received from each of the clients so as to maintain the perception that the video game is being executed locally;

wherein each of the single video streams directed to each client has a primary view generated from a perspective of a user of the client receiving the single video stream and a plurality of secondary views generated from a perspective of other users participating in the multi-player video game;

wherein the hardware compressor is configured to individually compress each of the single video streams sent to each of the clients and dynamically adjust a resolution of the single video streams independently of one another so that each one of the clients maintains a substantial perception that the video game is being executed locally for the respective client, and wherein the bandwidth feedback is associated with a current state of a communication channel between respective ones of the clients and at least one of the servers of the hosting service, wherein some of the servers are located one or more of the data centers.

14. The system as in claim 13, wherein each of the clients have different view processing capabilities, wherein the view processing capabilities include processing power, decoders, resolution processing, display configurations, or two or more thereof.

15. The system as in claim 13, wherein said clients are defined by one or more of a touch-screen tablet device, a smartphone device, a personal computer, a game console, a television, a portable device, or a computing device having access to a network.

16. The system of claim 13, wherein the hardware compressor uses an H.264 encoder.