Command sentinel

• US 9,349,201 B1
• Filed: 01/13/2012
• Issued: 05/24/2016
• Est. Priority Date: 08/03/2006
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Lau, Brian K. F., Yong, Kelvin Michael, Rickeby, Claes Christian, Breed, Ryan Hamilton, Galano, Eleazar T. III, English, Austin, Gault, Andrew Buchanan, Pereira, Rui Filipe Andrade, Perry, David
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): Akhavannik, Mohammad H
• Application Number:
  US13/350,216
• Time in Patent Office: 1,593 Days
• Field of Search: 345/501
• US Class Current: 1/1
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/409 Data transfer via television network
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video

Abstract

A video server is configured to provide streaming video to players of computer games over a computing network. The video server can provided video of different games to different players simultaneously. This is accomplished by rendering several video streams in parallel using a single GPU (Graphics Processing Unit). The output of the GPU is provided to graphics processing pipelines that are each associated with a specific client/player and are dynamically allocated as needed. A client qualifier may be used to assure that only clients capable of presenting the streaming video to a player at a minimum level of quality receive the video stream. Video frames provided by the video server optionally include overlays added to the output of the GPU. These overlays can include voice data received from another game player. These overlays may be used to prevent presentation of non-allowed input controls.

Claims

1. A video server system comprising:

an application engine configured to execute a first computer application;

a graphics processing unit configured to generate a graphic output in response to graphics data received from the first computer application;

sentinel logic configured to monitor the graphic output and determine that a blocked input control is included within the graphic output, the sentinel logic configured to send a blocked command to disable the blocked input control, the sentinel logic provides instruction to adjust the graphic output so as to modify the blocked input control, wherein the sentinel logic is configured to determine that the blocked input control is included within the graphic output by identifying a bit pattern that is predefined for the blocked input control;

an overlay logic that receives instruction from the sentinel logic to adjust the graphic output, the overlay logic is configured to process the modifying by placing an overlay on a portion of the graphic output that includes the blocked input control, the overlay configured to change an appearance of the blocked input control, wherein the sentinel logic is further configured to examine the overlay to ensure absence of any blocked input control in the overlay prior to the overlay being placed on the portion of the graphic output that includes the blocked input control;

an encoding pipeline configured to generate an encoded video stream using the graphic output;

an output logic configured to communicate the encoded video stream to a remote client via a communication network, the encoded video stream including the graphic output that includes the modified blocked input control; and

a microprocessor configured to execute the sentinel logic at the video server system.

2. The system of claim 1, wherein the first computer application is a video game.

3. The system of claim 1, wherein the graphic data includes a three dimensional game environment and the graphic processing unit is configured to render the graphic output based on the game environment and a point of view.

4. The system of claim 1, wherein the sentinel logic is configured to search for the bit pattern by examining a plurality of bit patterns within the graphic output, and to search for different members of the plurality of bit patterns depending on a state of the first computer application.

5. The system of claim 1, wherein the sentinel logic is configured to monitor only a fraction of graphic output frames generated by the graphics processing unit in response to the first computer application.

6. The system of claim 1, wherein the sentinel logic is configured to notify the overlay logic when a blocked input control is determined to be present in the graphic output.

7. The system of claim 1, wherein the sentinel logic is configured to determine that a blocked input control is included within the graphic output by monitoring memory within the graphics processing unit or by monitoring memory shared by more than one encoding pipeline.

8. The system of claim 1, further comprising state logic configured to change operation of the sentinel logic responsive to a state of the first computer application, the change including selecting which input controls are blocked input controls.

9. The system of claim 1, further comprising discovery logic configured to identify previously unidentified blocked input controls within the graphic output.

10. The system of claim 9, wherein the discovery logic is configured to identify the previously unidentified blocked input controls by monitoring actions of the first computer application.

11. The video server system of claim 1, wherein the sentinel logic is configured to send the blocked command to the first computer application so that subsequent graphics data received from the first computer application does not include the blocked input control.

12. A video server system comprising:

an application engine configured to execute a first computer application;

a graphics processing unit configured to generate a graphic output in response to graphics data received from the first computer application;

sentinel logic configured to monitor the graphic output and determine that a blocked input control is included within the graphic output, the sentinel logic configured to send a blocked command to disable the blocked input control, the sentinel logic provides instruction to adjust the graphic output so as to modify the blocked input control, wherein the sentinel logic is configured to determine that the blocked input control is included within the graphic output by identifying a bit pattern that is predefined for the blocked input control;

an overlay logic that receives instruction from the sentinel logic to adjust the graphic output, the overlay logic is configured to process the modifying by placing an overlay on a portion of the graphic output that includes the blocked input control, the overlay configured to change an appearance of the blocked input control, wherein the sentinel logic is further configured to examine the overlay to ensure absence of any blocked input control in the overlay prior to the overlay being placed on the portion of the graphic output that includes the blocked input control;

an encoding pipeline configured to generate an encoded video stream using the adjusted graphic output;

an output logic configured to communicate the encoded video stream to a remote client via a communication network, the encoded video stream including the graphic output that includes the modified blocked input control;

client control logic configured to open a browser window or browser tab on the remote client in response to receiving a blocked command from the remote client; and

a microprocessor configured to execute the sentinel logic.

13. The system of claim 12, further comprising command processing logic configured to communicate the blocked command received from the sentinel logic to the first computer application.

14. The system of claim 12, wherein the browser window or browser tab is configured for local execution of the first computer application on the client.

15. The system of claim 12, wherein the client control logic is further configured to load a website into the browser window or browser tab.

16. The system of claim 12, wherein the client control logic is further configured to log into a social networking website.