Voice overlay

• US 8,888,592 B1
• Filed: 06/29/2010
• Issued: 11/18/2014
• Est. Priority Date: 06/01/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Pereira, Rui Filipe Andrade, Gault, Andrew Buchanan, Perry, David
• Primary Examiner(s): Rowland, Steve
• Application Number:
  US12/826,489
• Time in Patent Office: 1,603 Days
• Field of Search: 463/35, 463/42
• US Class Current: 463/35
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/12 involving interaction between a plurality of game devices, e.g. transmisison or distribution systems protocols for networked virtual reality, networked games H04L29/06034
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/409 Data transfer via television network
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/552 for downloading to client devices, e.g. using OS version, hardware or software profile of the client device
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105

Abstract

A video server is configured to provide streaming video to players of computer games over a computing network. The video server can provided video of different games to different players simultaneously. This is accomplished by rendering several video streams in parallel using a single GPU (Graphics Processing Unit). The output of the GPU is provided to graphics processing pipelines that are each associated with a specific client/player and are dynamically allocated as needed. A client qualifier may be used to assure that only clients capable of presenting the streaming video to a player at a minimum level of quality receive the video stream. Video frames provided by the video server optionally include overlays added to the output of the GPU. These overlays can include voice data received from another game player.

Claims

1. A system comprising:

first game logic configured to generate first audio data for a game state of a video game;

an audio input configured to receive second audio data from a first client of the video game;

a first encoding pipeline dynamically provisioned based on determined audio capabilities of a second client, the first encoding pipeline including audio overlay logic and an audio encoder, the audio encoder provisioned based on presence of a corresponding audio decoder at the second client, the audio overlay logic defined to scale the second audio data as received from the first client, the audio overlay logic further defined to place the second audio data in a scaled form as an audio overlay on the first audio data as generated by the first game logic, the audio encoder defined to encode the first audio data with the second audio data in the scaled form overlaid thereon into a first audio packet, the first audio packet formatted for decoding by the corresponding audio decoder at the second client;

an output configured to provide the first audio packet to the second client over a communication network; and

an electronic processor configured to execute the first game logic or the first encoding pipeline.

2. The system of claim 1, wherein the first game logic is further configured to provide a game environment based on game rules of the video game.

3. The system of claim 1, wherein the first game logic is further configured to receive a game state of the video game from a geographically remote game server.

4. The system of claim 1, wherein the audio input is configured to receive the second audio data from more than one client.

5. The system of claim 1, wherein the audio input is configured to receive the second audio data as a second audio packet, and wherein the audio overlay logic is defined to decode the second audio data prior to scaling the second audio data.

6. The system of claim 1, wherein the first client and the audio input are communicatively coupled by a computing network.

7. The system of claim 1, wherein the second audio data includes voice data.

8. The system of claim 1, wherein the first audio packet includes both voice data received from a plurality of clients and audio generated by the first game logic.

21. The system of claim 1, further comprising:

a second encoding pipeline configured to generate game video based on events within the video game, wherein the output is further configured to provide the game video to the second client.

22. The system of claim 21, wherein the first encoding pipeline, the second encoding pipeline and the game logic are all disposed within a same computing device.

23. The system of claim 21, wherein the events within the video game are subject to rules of the first game logic.

9. A method of generating game audio, the method comprising:

operating a computer system to execute game logic to generate game audio for a game state of a video game;

receiving audio data from a first player of the video game over a computing network;

provisioning a dynamic encoding pipeline for a second player of the video game, the dynamic encoding pipeline provisioned based on audio capabilities of a computing system of the second player of the video game, the dynamic encoding pipeline including audio overlay logic and an audio encoder, the audio encoder provisioned based on presence of a corresponding audio decoder at the computing system of the second player of the video game, the audio overlay logic defined to scale the audio data received from the first player of the video game;

executing the overlay logic to scale the audio data received from the first player of the video game;

executing the overlay logic to overlay a scaled version of the audio data received from the first player on the game audio generated by the game logic;

executing the audio encoder to encode the overlaid game audio and scaled audio data to generate an audio stream; and

delivering the audio stream to the second player of the video game over the computing network.

10. The method of claim 9, wherein the video game is a configured to be played by multiple geographically distributed players connected via the computing network.

11. The method of claim 9, wherein the video game audio includes sounds associated with actions occurring at the game state of the video game.

12. The method of claim 9, wherein the audio data includes the voice of the first player.

13. The method of claim 9, wherein the audio data includes music provided by the first player.

14. The method of claim 9, wherein the first player and the second player are in different geographic locations.

15. The method of claim 9, further comprising:

providing a video stream, resulting from game rules of the video game, to the second player of the video game.

16. The method of claim 9, further comprising:

receiving the game state of the video game from a remote game server.

17. The method of claim 9, further comprising:

determining that the audio data from the first player of the video game should be overlaid on the game audio based on a social relationship between the first player and the second player.

18. A non-transitory computer readable medium having stored thereupon:

computing instructions for generating game audio for a game state of a video game;

computing instructions for receiving audio data from at least a first player of the video game over a computing network;

computing instructions for provisioning a dynamic encoding pipeline for a second player of the video game, the dynamic encoding pipeline provisioned based on audio capabilities of a computing system of the second player of the video game, the dynamic encoding pipeline including audio overlay logic and an audio encoder, the audio encoder provisioned based on presence of a corresponding audio decoder at the computing system of the second player of the video game, the audio overlay logic defined to scale the audio data received from the first player of the video game;

computing instructions for executing the overlay logic to scale the audio data received from the first player of the video game;

computing instructions for overlaying a scaled version of the audio data received from the first player on the game audio;

computing instructions for encoding the overlaid game audio and scaled audio data to generate an audio stream; and

computing instructions for delivering the audio stream to the second player of the video game over the computing network.

19. The non-transitory computer readable medium of claim 18, wherein the computing instructions for receiving audio data from at least the first player are configured to receive the audio data from more than one player of the game in parallel.

20. The non-transitory computer readable medium of claim 18, wherein the audio stream includes a data packet comprising both the scaled audio data and the game audio.