System and method for managing audio and video channels for video game players and spectators

• US 9,339,728 B2
• Filed: 06/13/2012
• Issued: 05/17/2016
• Est. Priority Date: 12/10/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Sullivan, David R., McCool, Robert, Perlman, Stephen G.
• Primary Examiner(s): D'Augustino, Paul A
• Assistant Examiner(s): DOSHI, ANKIT B
• Application Number:
  US13/495,904
• Publication Number:
  US 20130123019A1
• Time in Patent Office: 1,434 Days
• Field of Search: 463/42
• US Class Current: 1/1
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/35 Details of game servers
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/358 Adapting the game course according to the network or server load, e.g. for reducing latency due to different connection speeds between clients
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/407 Data transfer via internet
• CPC Class Codes: A63F 2300/513 server hierarchy, e.g. local, regional, national or dedicated for different tasks, e.g. authenticating, billing
• CPC Class Codes: A63F 2300/534 for network load management, e.g. bandwidth optimization, latency reduction
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: A63F 2300/572 Communication between players during game play of non game information, e.g. e-mail, chat, file transfer, streaming of audio and streaming of video
• CPC Class Codes: A63F 2300/577 for watching a game played by other players
• CPC Class Codes: A63F 2300/6072 of an input signal, e.g. pitch and rhythm extraction, voice recognition
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

A computer-implemented system and method are described for managing audio chat for an online video game or application. For example, a system according to one embodiment comprises: an online video game or application execution engine to execute an online video game or application in response to input from one or more users of the video game or application and to responsively generate audio and video of the video game or application; and a chat subsystem to establish audio chat sessions with the one or more users and one or more spectators to the video game or application, the chat subsystem establishing a plurality of audio chat channels including a spectator channel over which the spectators participate in audio chat and a user channel over which the users participate in audio chat.

Claims

1. A system for managing audio chat sessions for online video games or applications comprising:

an online video game or application execution engine of a hosting service to execute an online video game or application in response to input from one or more users playing the video game or application and to responsively generate audio and video of the video game or application, the hosting service accessed over a network, wherein the input from one or more users is received from remote devices of the one or more users over the network and the generated audio and video of the video game or application is transmitted to the remote devices of the one or more users and to remote devices of one or more spectators to the video game or application; and

a chat subsystem of the hosting service to establish audio chat sessions with the one or more users playing the video game and the one or more spectators to the video game or application, the chat subsystem establishing a plurality of audio chat channels including a spectator channel over which the spectators participate in audio chat and a user channel over which the users participate in audio chat, wherein the spectator channel and the user channel are managed by the chat subsystem so as to enable audio chat sessions using the spectator channel and the user channel to be conducted concurrently, the chat subsystem configured to assign a priority level to each of the spectator channel and the user channel, the priority level used to set a priority of processing of the audio packets for the respective channel.

2. The system as in claim 1 wherein one or more spectators and users of the video game or application may join the spectator channel to participate in audio chat over the spectator channel and wherein users may join the user channel to participate in audio chat with other users over the user channel.

3. The system as in claim 1 wherein a first user channel is established for a first team of users playing the video game or application and a second user channel is established for a second team of users playing the video game.

4. The system as in claim 1 wherein the chat audio subsystem further establishes a private chat channel between a user and one or more spectators, the private chat channel established in response to control signals received from the user and/or the spectators permitted to join the private channel.

5. The system as in claim 1 wherein the chat subsystem further comprises a plurality of audio chat server modules supporting active audio chat sessions with each of the users and/or spectators, and wherein each audio chat channel comprises a plurality of nodes, each of the nodes associated with one of the audio chat server modules.

6. The system as in claim 5 wherein a server module may be associated with multiple nodes in multiple different audio chat channels, and wherein if a server module is associated with multiple nodes, the user or spectator associated with that server module may participate in audio chat for each of the audio chat channels having nodes with which the server module is associated.

7. The system as in claim 5 wherein a server module associated with one of the users is associated with a first node within the user chat channel and a second node within the spectator chat channel.

8. The system as in claim 1 wherein the chat subsystem includes a selective muting function for each user and/or spectator, the muting function allowing the user and/or spectator to mute other selected users and/or spectators associated with the user channel and/or the spectator channel, wherein in response to the user and/or spectator muting one of the other users and/or spectators, the chat subsystem will not include audio packets from those other users and/or spectators in the chat audio stream sent to the user and/or spectator.

9. The system as in claim 1 wherein the chat subsystem comprises program code for performing the operations of:

determining if audio contained in an audio packet received from a particular user and/or spectator is above a specified energy threshold;

if the audio contained in the audio packet is above the specified energy threshold, then mixing the audio with audio from other users and/or spectators to create a mixed audio stream; and

transmitting the mixed audio stream to the users and/or spectators.

10. The system as in claim 9 wherein if the audio contained in the audio packet is not above the specified energy threshold, then dropping the audio packet.

11. The system as in claim 9 wherein the chat subsystem comprises program code for performing the additional operation of:

mixing the audio from users and/or spectators above the specified energy threshold with audio from the online video game or application.

12. The system as in claim 11 further comprising:

an audio/video compression module for compressing the video received from the online video game or application and the audio received from the video game or application mixed with audio from the users and/or spectators.

13. The system as in claim 12 wherein the audio/video compression module compresses separate independent audio/video streams for each user and/or spectator using different compression.

14. The system as in claim 13 wherein the different compression is selected based on the current network connection with the client devices used by each of the users and/or spectators.

15. The system as in claim 13 wherein the different compression is selected based on the processing capabilities of client devices used by each of the users and/or spectators.

16. The system as in claim 1 wherein the chat subsystem performs the additional operations of establishing video chat sessions with the one or more users and/or one or more spectators to the video game or application, the chat subsystem allowing authorized users and/or spectators to view video of other users and/or spectators as a video game or application is played or used.

17. The system as in claim 16 wherein the chat subsystem performs the additional operations of establishing a plurality of video chat channels including a spectator video chat channel over which the spectators participate in video chat and a user video chat channel over which the users participate in video chat.

18. The system as in claim 16 wherein video of each spectator and/or user participating in video chat session is displayed in a video chat window within the graphical user interface of one or more other spectators and/or users participating in the video chat session.

19. The system as in claim 17 wherein the video chat window is overlaid over a portion of the video of the video game or application being played or used.

20. The system as in claim 1 wherein the chat subsystem comprises one or more filters to alter the voices of selected users and/or spectators.

21. The system as in claim 20 wherein at least one of the filters is configured to alter the perceived gender of the user and/or spectator.

22. The system as in claim 20 wherein at least one of the filters is configured to alter the voice of the user and/or spectator to sound like a character in the video game or application.

23. The system as in claim 20 wherein at least one of the filters is configured disguise the voice of the user and/or spectator to hide the identity of the user and/or spectator.

24. The system as in claim 20 wherein users on the same team hear the actual voices of other team users while opponents or spectators hear the altered voices of the team users.

25. The system as in claim 1 wherein the chat subsystem performs the additional operation of establishing a coach chat channel over which one or more video game or application coaches and one or more users participate in audio and/or video chat as the online video game or application is played.

26. The system as in claim 1 wherein the latency between a time at which a user generates a control signal to control execution of the video game or application via a user input device connected to a client and a time at which the video displayed at the client is altered in response to the control signal is such that the user perceives the video game or application to be responding instantly to control via the user input device.

27. The system as in claim 1 wherein the chat subsystem comprises a low-latency audio codec for coding the audio chat streams.

28. The system as in claim 27 wherein the low-latency audio codec comprises a Constrained Energy Lapped Transform (CELT) codec.

29. The system as in claim 27 wherein the low-latency audio codec comprises an error-tolerant codec.

30. The system as in claim 27 wherein a first audio codec and/or error concealment and/or error correction technique is used for the voice chat stream transmitted from each user/spectator to the chat subsystem and after the voice chat streams are processed, mixed and/or transformed, a second audio codec and/or error concealment and/or error correction technique is used for the voice chat stream for delivery to each user/spectator.

31. The system as in claim 1 wherein the chat subsystem comprises translation logic for translating voices in a first language into a second language, wherein the voice in the second language is transmitted to one or more specified users and/or spectators.

32. The system as in claim 31 wherein the chat subsystem includes text translation logic for generating text representing words spoken by the users and/or spectators in the first language and/or the second language, the text being transmitted to one or more specified users and/or spectators.

33. The system as in claim 1 wherein the chat subsystem includes text translation logic for generating text representing words spoken by the users and/or spectators, the text being transmitted to one or more specified users and/or spectators.

34. A method for managing audio chat sessions for online video games or applications comprising:

executing an online video game or application on a server of a hosting service in response to input from one or more users of the video game or application and responsively generating audio and video of the video game or application, wherein the input from one or more users is received by the hosting service from remote devices of the one or more users over a network and the generated audio and video of the video game or application is transmitted to the remote devices of the one or more users and to remote devices of one or more spectators to the video game or application;

establishing audio chat sessions with the one or more users and the one or more spectators to the video game or application, wherein establishing comprises setting up a plurality of audio chat channels including a spectator channel over which the spectators participate in audio chat and a user channel over which the users participate in audio chat, the spectator channel and the user channel used to conduct audio chat sessions concurrently, each of the plurality of audio chat channels accorded a priority level, the priority level used to set priority of processing of the audio packets received through respective chat channels.

35. The method as in claim 34 wherein one or more spectators and users of the video game or application may join the spectator channel to participate in audio chat over the spectator channel and wherein users may join the user channel to participate in audio chat with other users over the user channel.

36. The method as in claim 34 wherein a first user channel is established for a first team of users playing the video game or using the application and a second user channel is established for a second team of users playing the video game or using the application.

37. The method as in claim 34 further comprising establishing a private chat channel between a user and one or more spectators, the private chat channel established in response to control signals received from the user and/or the spectators permitted to join the private channel.

38. The method as in claim 34 further comprising providing a plurality of audio chat server modules supporting active audio chat sessions with each of the users and/or spectators, and wherein each audio chat channel comprises a plurality of nodes, each of the nodes associated with one of the audio chat server modules.

39. The method as in claim 38 wherein a server module may be associated with multiple nodes in multiple different audio chat channels, and wherein if a server module is associated with multiple nodes, the user or spectator associated with that server module may participate in audio chat for each of the audio chat channels having nodes with which the server module is associated.

40. The method as in claim 38 wherein a server module associated with one of the users is associated with a first node within the user chat channel and a second node within the spectator chat channel.

41. The method as in claim 34 further comprising providing a selective muting function for each user and/or spectator, the muting function allowing the user and/or spectator to mute other selected users and/or spectators associated with the user channel and/or the spectator channel, wherein in response to the user and/or spectator muting one of the other users and/or spectators, audio packets from those other users and/or spectators will not be included in the chat audio stream sent to the user and/or spectator.

42. The method as in claim 34 further comprising:

determining if audio contained in an audio packet received from a particular user and/or spectator is above a specified energy threshold;

if the audio contained in the audio packet is above the specified energy threshold, then mixing the audio with audio from other users and/or spectators to create a mixed audio stream; and

transmitting the mixed audio stream to the users and/or spectators.

43. The method as in claim 42 wherein if the audio contained in the audio packet is not above the specified energy threshold, then dropping the audio packet.

44. The method as in claim 42 further comprising:

mixing the audio from users and/or spectators above the specified energy threshold with audio from the online video game or application.

45. The method as in claim 44 further comprising compressing the video received from the online video game and the audio received from the video game or application mixed with audio from the users and/or spectators.

46. The method as in claim 45 wherein separate independent audio/video streams are compressed for each user and/or spectator using different compression.

47. The method as in claim 46 wherein the different compression is selected based on the current network connection with the client devices used by each of the users and/or spectators.

48. The method as in claim 46 wherein the different compression is selected based on the processing capabilities of client devices used by each of the users and/or spectators.

49. The method as in claim 34 wherein the chat subsystem performs the additional operations of establishing video chat sessions with the one or more users and/or one or more spectators to the video game or application, the chat subsystem allowing authorized users and/or spectators to view video of other users and/or spectators as a video game or application is played or used.

50. The method as in claim 49 wherein the chat subsystem performs the additional operations of establishing a plurality of video chat channels including a spectator video chat channel over which the spectators participate in video chat and a user video chat channel over which the users participate in video chat.

51. The method as in claim 49 wherein video of each spectator and/or user participating in video chat session is displayed in a video chat window within the graphical user interface of one or more other spectators and/or users participating in the video chat session.

52. The method as in claim 50 wherein the video chat window is overlaid over a portion of the video of the video game or application being played.

53. The method as in claim 34 wherein the chat subsystem comprises one or more filters to alter the voices of selected users and/or spectators.

54. The method as in claim 53 wherein at least one of the filters is configured to alter the perceived gender of the user and/or spectator.

55. The method as in claim 53 wherein at least one of the filters is configured to alter the voice of the user and/or spectator to sound like a character in the video game or application.

56. The method as in claim 53 wherein at least one of the filters is configured disguise the voice of the user and/or spectator to hide the identity of the user and/or spectator.

57. The method as in claim 53 wherein users on the same team hear the actual voices of other team users while opponents or spectators hear the altered voices of the team users.

58. The method as in claim 34 wherein the chat subsystem performs the additional operation of establishing a coach chat channel over which one or more video game or application coaches and one or more users participate in audio and/or video chat as the online video game or application is played.

59. The method as in claim 34 wherein the latency between a time at which a user generates a control signal to control execution of the video game or application via a user input device connected to a client and a time at which the video displayed at the client is altered in response to the control signal is such that the user perceives the video game or application to be responding instantly to control via the user input device.

60. The method as in claim 34 wherein the chat subsystem comprises a low-latency audio codec for coding the audio chat streams.

61. The method as in claim 60 wherein the low-latency audio codec comprises a Constrained Energy Lapped Transform (CELT) codec.

62. The method as in claim 60 wherein the low-latency audio codec comprises an error-tolerant codec.

63. The method as in claim 60 wherein a first audio codec and/or error concealment and/or error correction technique is used for the voice chat stream transmitted from each user/spectator to the chat subsystem and after the voice chat streams are processed, mixed and/or transformed, a second audio codec and/or error concealment and/or error correction technique is used for the voice chat stream for delivery to each user/spectator.

64. The method as in claim 34 wherein the chat subsystem comprises translation logic for translating voices in a first language into a second language, wherein the voice in the second language is transmitted to one or more specified users and/or spectators.

65. The method as in claim 64 wherein the chat subsystem includes text translation logic for generating text representing words spoken by the users and/or spectators in the first language and/or the second language, the text being transmitted to one or more specified users and/or spectators.

66. The method as in claim 34 wherein the chat subsystem includes text translation logic for generating text representing words spoken by the users and/or spectators, the text being transmitted to one or more specified users and/or spectators.