Spectator audio analysis in online gaming environments

• US 10,300,394 B1
• Filed: 06/05/2015
• Issued: 05/28/2019
• Est. Priority Date: 06/05/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Evans, Ethan Zane, George, Michael Martin, Frazzini, Michael Anthony, Hirst, Alistair Robert, Van Gorder, Matthew James
• Primary Examiner(s): Yen, Jason T
• Application Number:
  US14/732,582
• Time in Patent Office: 1,453 Days
• Field of Search: None
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

An audio analysis system that may analyze participant audio input in a game spectating environment. The audio analysis system may receive audio input for participants (broadcasters and/or spectators) in a game spectating system. The audio analysis system may analyze the audio input to generate audio analysis information for broadcasts, and provide the information to the game spectating system. The audio analysis information may be applied in the game spectating system in various ways. For example, the information may be used to determine active or popular broadcasts, broadcasters, games, and/or recordings of broadcasts, and the active or popular content may be indicated to spectators via a game spectating user interface. The audio analysis system may provide an application programming interface (API) to receive the audio input and to provide audio analysis information.

Claims

1. A system, comprising:

one or more computing devices configured to implement a game spectating system configured to;

broadcast video received from a broadcaster device to a plurality of spectator devices, wherein the video includes game play of a respective broadcaster participating as a player in a game session of an online game;

receive spectator audio input for a plurality of spectators associated with the plurality of spectator devices;

analyze the spectator audio input to map the spectator audio input to events of the game session based at least in part on audio patterns for the spectators to generate audio analysis information, wherein the audio analysis information indicates a state of the plurality of spectators for the events; and

provide one or more indications of the state of the plurality of spectators to the plurality of spectator devices to affect the broadcasted video.

2. The system as recited in claim 1, wherein the one or more indications of the state of the plurality of spectators include one or both of visual indications or audio indications.

3. The system as recited in claim 1, wherein the game spectating system is configured to:

iteratively perform said receive, said analyze, and said provide during the broadcast;

detect a change in state of the plurality of spectators while iteratively performing said receive and said analyze; and

change at least one of the one or more indications of the state of the plurality of spectators in response to said detecting.

4. The system as recited in claim 1, wherein the game spectating system is configured to perform said broadcast, said receive, said analyze, and said provide for a plurality of broadcast channels, each broadcast channel corresponding to one of a plurality of broadcasters.

5. The system as recited in claim 1, wherein, to analyze the spectator audio input to generate audio analysis information, the game spectating system is configured to:

apply signal analysis to audio input signals corresponding to the plurality of spectators to determine audio signal metrics;

map the audio signal metrics to the audio patterns for the spectators to determine spectator audio analysis data;

normalize and aggregate the spectator audio analysis data; and

analyze the aggregated spectator audio analysis data to determine the audio analysis information.

6. The system as recited in claim 1, wherein the audio analysis information indicates an emotional state of the plurality of spectators, and wherein the one or more indications include a visual indication that modifies a virtual representation of the plurality of spectators on the plurality of spectator devices according to the emotional state of the plurality of spectators.

7. The system as recited in claim 1, wherein the audio analysis information indicates an emotional state of the plurality of spectators, and wherein the one or more indications include an audio indication that audibly indicates the emotional state of the plurality of spectators.

8. The system as recited in claim 1, wherein the game session involves a plurality of game characters acting within a game universe as controlled by a plurality of players via respective player devices, and wherein the game spectating system is further configured to provide one or both of a visual indication or an audio indication of the state of the plurality of spectators to the player devices.

9. The system as recited in claim 1, wherein the game spectating system is configured to present a game spectating user interface (UI) on the respective spectator devices, wherein the UI includes a plurality of UI elements corresponding to a plurality of broadcast channels, each broadcast channel corresponding to one of a plurality of broadcasters, and wherein the game spectating system is configured to modify display of at least one of the UI elements according to the audio analysis information to highlight one or more of the broadcast channels.

10. A method, comprising:

iteratively performing, by an audio analysis module implemented on one or more computing devices;

receiving spectator audio input for a plurality of spectators watching a broadcast from a game spectating system on respective client devices, wherein the broadcast shows game play of a respective broadcaster participating as a player in a game session of an online game;

analyzing the spectator audio input based at least in part on mapping the spectator audio input to events of the game session based at least in part on audio patterns for the spectators to generate audio analysis information; and

providing the audio analysis information to the game spectating system, wherein the audio analysis information affects the broadcast.

11. The method as recited in claim 10, wherein the audio analysis information indicates states of the plurality of spectators, the method further comprising providing, by the game spectating system, one or more indications of the states of the plurality of spectators to at least one client device, wherein the one or more indications of the states of the plurality of spectators include one or both of visual indications or audio indications.

12. The method as recited in claim 10, wherein the audio analysis information indicates one or more of the plurality of spectators as spectators of interest according to their respective audio input, the method further comprising providing, by the game spectating system, output indicating the spectators of interest to at least one client device, wherein the output includes one or both of video of the one or more spectators of interest or the audio input from the one or more spectators of interest.

13. The method as recited in claim 10, wherein the audio analysis information affects one or both of execution of the game session within the online game system or presentation of the broadcast of the game session on the client devices.

14. The method as recited in claim 10, wherein the game spectating system concurrently streams a plurality of broadcasts on a plurality of broadcast channels, each broadcast channel corresponding to one of a plurality of broadcasters.

15. The method as recited in claim 14, wherein the game spectating system presents a game spectating user interface (UI) on the respective client devices, wherein the UI includes a plurality of UI elements corresponding to the plurality of broadcast channels, and wherein the method further comprises modifying display of at least one of the UI elements according to the audio analysis information.

16. The method as recited in claim 10, wherein analyzing the spectator audio input to generate audio analysis information comprises:

mapping audio input signals corresponding to the plurality of spectators to the audio patterns for the spectators to determine audio analysis data for the spectators; and

determining the audio analysis information from the determined audio analysis data.

17. The method as recited in claim 10, wherein analyzing the spectator audio input to generate audio analysis information comprises:

normalizing and aggregating a plurality of audio input signals corresponding to two or more of the plurality of spectators to generate an aggregate audio signal; and

applying signal analysis to the aggregate audio signal to determine the audio analysis information.

18. The method as recited in claim 10, wherein analyzing the spectator audio input to generate audio analysis information comprises:

applying signal analysis to audio input signals corresponding to the plurality of spectator to determine audio signal metrics for the spectator, wherein the audio signal metrics include vocal metrics and speech metrics;

applying an emotion recognition technique to the vocal metrics to determine emotion content from the audio input signals;

applying a speech recognition technique to the speech metrics to determine speech content from the audio input signals; and

comparing the determined emotion content and speech content to known audio content patterns to determine the audio analysis information.

19. The method as recited in claim 10, further comprising:

analyzing broadcaster audio input to generate broadcaster audio analysis information; and

providing the broadcaster audio analysis information to the game spectating system, wherein the broadcaster audio analysis information affects the broadcast.

20. The method as recited in claim 10, wherein the audio analysis information indicates an emotional state of the plurality of spectators watching the broadcast, the method further comprising providing a visual indication of the emotional state to the client devices for display.

21. The method as recited in claim 10, wherein the audio analysis information indicates an emotional state of the plurality of spectators watching the broadcast, wherein the game session involves a plurality of players participating in the game session on respective player devices, and wherein the method further comprises providing a visual indication of the emotional state of the plurality of spectators to the player devices for display.

22. The method as recited in claim 10, further comprising determining at least one event in the broadcast according to additional spectator audio input comprising additional audio patterns matching the audio patterns used to generate the audio analysis information, the method further comprising providing a visual indication of the at least one event to the client devices for display.

23. The method as recited in claim 10, further comprising determining at least one highlight segment from the broadcast according to the audio analysis information, the method further comprising providing the at least one highlight segment from the game session to a highlight user interface on at least some of the client devices for viewing.

24. The method as recited in claim 10, further comprising tagging a recording of the broadcast with metadata indicating at least one event in the broadcast as indicated by the audio analysis information.

25. The method as recited in claim 10, further comprising providing the audio analysis information to an online game system executing the game session, wherein the audio analysis information affects one or both of execution of the game session within the online game system or visual presentation of the game session on client devices.

26. The method as recited in claim 10, wherein the audio analysis module is a component of the game spectating system.

27. The method as recited in claim 10, wherein the audio analysis module is implemented as a service on a network, and wherein the audio analysis module provides an application programming interface (API) to receive audio input from one or more audio analysis clients including the game spectating system and to provide audio analysis information to the one or more audio analysis clients including the game spectating system.

28. The method as recited in claim 10, further comprising:

receiving commentator audio input from a commentator providing commentary for the game session being broadcast;

analyzing the commentator audio input to generate commentator audio analysis information; and

providing the commentator audio analysis information to the game spectating system, wherein the commentator audio analysis information affects the broadcast.

29. The method as recited in claim 10, further comprising:

segmenting the received spectator audio input according to affiliations of the plurality of spectators to generate two or more spectator audio input groups;

wherein said analyzing the spectator audio input to generate audio analysis information comprises analyzing the two or more spectator audio input groups to generate separate audio analysis information for each spectator audio input group.

30. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement an audio analysis system configured to:

receive spectator audio input for a plurality of spectators watching two or more broadcasts from a game spectating system on respective client devices, wherein each broadcast shows game play of a respective broadcaster participating as a player in a game session of an online game;

analyze the spectator audio input to map the spectator audio input to events of the game session based at least in part on audio patterns for the spectators to generate audio analysis information, wherein the audio analysis information indicates states of the plurality of spectators; and

provide the audio analysis information to the game spectating system, wherein the audio analysis information affects one or both of execution of the game session within the online game system or presentation of the broadcast of the game session on the client devices.

31. The non-transitory computer-accessible storage medium as recited in claim 30, wherein the audio analysis information indicates an emotional state of the spectators watching the broadcasts, and wherein a visual indication of the emotional state of the spectators is provided to at least one broadcaster device for display and to at least one client device for display.