Determining highlights in a game spectating system

• US 10,363,488 B1
• Filed: 06/29/2015
• Issued: 07/30/2019
• Est. Priority Date: 06/29/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Willette, Michael Anthony, Gilmore, Patrick, Evans, Ethan Zane, George, Michael Martin, Frazzini, Michael Anthony
• Primary Examiner(s): Torimiro, Adetokunbo O
• Application Number:
  US14/754,575
• Time in Patent Office: 1,492 Days
• Field of Search: 463 1, 463 20, 463 22, 463 25, 463 31, 463 39, 463 40, 463 42
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/63 by the player, e.g. authoring using a level editor
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/5533 using program state or machine event data, e.g. server keeps track of the state of multiple players on in a multiple player game
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4756 for rating content, e.g. scoring a recommended movie
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Methods and apparatus for determining highlights from broadcasts in spectating environments. A highlight service obtains highlight data for broadcasts including game- and participant-specified events, audio input, and text chat, analyzes the highlight data to determine notable events (highlights) in the broadcasts, and extracts highlight segments from the broadcasts according to the determined events. Highlight reels may be created from the highlight segments according to one or more highlight selection criteria and/or spectator preferences. The highlight service may provide access to the highlights and highlight reels via a highlight user interface (UI). The spectators may selectively view the highlights or highlight reels via the highlight UI.

Claims

1. A system, comprising:

one or more computing devices configured to implement a game spectating service configured to stream a plurality of broadcasts received from a plurality of broadcaster devices to a plurality of spectator devices, wherein each broadcast includes video showing game play of a respective broadcaster participating as a player in a game executing on one of a plurality of game systems;

one or more computing devices configured to implement a highlight service configured to;

receive game event data from the plurality of game systems according to an application programming interface (API) to the highlight service, wherein the game event data received from a given game system indicates one or more notable events in a game executing on the respective game system as determined from the respective game event data, and wherein the game event data includes participant highlight tags that indicate one or more participant-specified notable events in the game;

determine a plurality of highlight segments from the plurality of broadcasts, wherein one or more of the plurality of highlight segments are determined according to the received game event data and the participant highlight tags, wherein each highlight segment includes a portion of a respective broadcast showing a notable event from a respective game, and wherein the one or more highlight segments are determined based at least in part on a corresponding participant highlight tag and satisfying a spectator voting threshold; and

provide access to at least some of the plurality of highlight segments for viewing by spectators via game spectating clients executing on the plurality of spectator devices.

2. The system as recited in claim 1, wherein the game event data comprises a plurality of game event records each including at least a game identifier and a game timestamp, and wherein, to determine the plurality of highlight segments from the plurality of broadcasts, the highlight service is configured to correlate the game event records to highlight segments in respective broadcasts according to the game identifiers and the game timestamps in the game event records.

3. The system as recited in claim 1, wherein the highlight service is further configured to:

receive audio input from the broadcaster devices and the spectator devices; and

analyze the audio input to generate audio analysis data for the plurality of broadcasts, wherein the audio analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from content of the audio input for the respective broadcast;

wherein at least one of the plurality of highlight segments is determined according to the audio analysis data.

4. The system as recited in claim 1, wherein the highlight service is further configured to:

receive text chat input from the broadcaster devices and the spectator devices; and

analyze the text chat input to generate chat analysis data for the plurality of broadcasts, wherein the chat analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from content of the text chat input for the respective broadcast;

wherein at least one of the plurality of highlight segments is determined according to the chat analysis data.

5. The system as recited in claim 1, wherein the highlight service is further configured to:

receive video input from the broadcaster devices, wherein the video input from a given broadcaster device shows a respective broadcaster when participating as a player in a respective game; and

analyze the video input to generate video analysis data for the plurality of broadcasts, wherein the video analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from physical actions of the respective broadcaster as shown in the respective video input;

wherein at least one of the plurality of highlight segments is determined according to the video analysis data.

6. The system as recited in claim 1, wherein the highlight service is further configured to:

receive participant highlight input from participants in the respective broadcasts via user interfaces on the participants'"'"' respective devices, wherein the participant highlight input for a given broadcast indicates one or more notable events in the respective broadcast as specified by one or more participants in the respective broadcast via the user interfaces on the respective devices;

wherein the one or more participant highlight tags are based at least in part on the participant highlight input.

7. A method, comprising:

performing, by a spectating system implemented on one or more computing devices;

streaming a plurality of broadcasts received from a plurality of broadcaster devices to a plurality of spectator devices, wherein each broadcast shows video of a respective broadcaster;

obtaining highlight data indicating notable events in the broadcasts, wherein the highlight data includes participant highlight tags that indicate participant-specified notable events in the broadcasts;

determining a plurality of highlight segments from the plurality of broadcasts according to the highlight data, wherein each highlight segment includes a portion of a respective broadcast showing a notable event from the broadcast determined according to respective highlight data and the respective participant highlight tags for the broadcast and satisfying a spectator voting threshold; and

providing access to at least some of the plurality of highlight segments for viewing on the plurality of spectator devices via a user interface (UI) to the spectating system.

8. The method as recited in claim 7, wherein at least one broadcast shows game play of a respective broadcaster participating as a player in a game executing on one of a plurality of game systems.

9. The method as recited in claim 8, wherein said obtaining highlight data indicating notable events in the broadcasts comprises receiving game event data from at least one of the plurality of game systems according to an application programming interface (API) to the spectating system, wherein the game event data received from a given game system indicates one or more notable events in a game executing on the respective game system.

10. The method as recited in claim 9, wherein the highlight data is received from the game systems as metadata included with video in the broadcasts received from the broadcaster devices.

11. The method as recited in claim 9, wherein the game event data comprises a plurality of game event records each including at least a game identifier and a game timestamp, and wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises:

correlating the game event records to the broadcasts according to the game identifiers; and

identifying the highlight segments in the broadcasts according to the game timestamps in the respective game event records.

12. The method as recited in claim 7, further comprising:

receiving audio input from the broadcaster devices and the spectator devices;

analyzing the audio input to generate audio analysis data for the plurality of broadcasts, wherein the audio analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from content of the audio input for the respective broadcast; and

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the audio analysis data.

13. The method as recited in claim 7, further comprising:

receiving text chat input from the broadcaster devices and the spectator devices;

analyzing the text chat input to generate chat analysis data for the plurality of broadcasts, wherein the chat analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from content of the text chat input for the respective broadcast;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the chat analysis data.

14. The method as recited in claim 7, further comprising:

receiving video inputs from one or more client devices, wherein the video input from a given client device shows a respective broadcaster or spectator when participating in the broadcast; and

analyzing the video inputs to generate video analysis data for the plurality of broadcasts, wherein the video analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from physical actions of participants in the broadcast as shown in the respective video inputs for the broadcast;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the video analysis data.

15. The method as recited in claim 14, wherein the video inputs show faces of respective participants, and wherein said analyzing the video inputs to generate video analysis data comprises analyzing facial expressions to detect emotions or states of respective participants.

16. The method as recited in claim 7, further comprising:

receiving spectator highlight input from spectators of the respective broadcasts via spectating user interfaces on the spectator devices, wherein the spectator highlight input for a given broadcast indicates one or more notable events in the respective broadcast as specified by one or more spectators of the respective broadcast via the spectating user interfaces on the respective spectator devices;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the spectator highlight input.

17. The method as recited in claim 7, further comprising:

receiving broadcaster highlight input from broadcasters of the respective broadcasts via broadcasting user interfaces on the broadcaster devices, wherein the broadcaster highlight input for a given broadcast indicates one or more notable events in the respective broadcast as specified by the broadcaster of the respective broadcast;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the broadcaster highlight input.

18. The method as recited in claim 7, further comprising:

applying a machine learning analysis technique to the broadcasts to detect highlight events in broadcast video, wherein the machine learning analysis technique compares attributes from previously determined highlight segments to attributes determined from the broadcast video to detect the highlight events;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the detected highlight events.

19. The method as recited in claim 7, further comprising:

applying a statistically improbable event analysis technique to metadata for the broadcasts to detect statistically improbable events in broadcast video, wherein the statistically improbable event analysis technique compares statistics from previous broadcasts to the metadata for the broadcasts to detect the statistically improbable events;

wherein said determining the plurality of highlight segments from the plurality of broadcasts comprises determining at least one of the plurality of highlight segments according to the detected statistically improbable events.

20. The method as recited in claim 7, wherein said determining a plurality of highlight segments from the plurality of broadcasts according to the highlight data comprises:

determining highlight events in the broadcasts; and

for each determined highlight event, determining a temporal window around the highlight event, wherein the windows provide context for the determined highlight events.

21. The method as recited in claim 7, wherein said determining a window around a highlight event comprises dynamically determining the window according to window attributes from one or more previously determined highlight segments.

22. The method as recited in claim 7, wherein said determining a window around a highlight event comprises determining the window according to specified window parameters.

23. The method as recited in claim 7, wherein said determining a window around a highlight event comprises determining the window according to spectator preferences.

24. The method as recited in claim 7, further comprising:

determining a set of two or more of the highlight segments according to one or more highlight selection criteria;

generating a highlight reel including the set of highlight segments; and

streaming the highlight reel to at least one of the plurality of spectator devices for display via the UI.

25. The method as recited in claim 24, wherein the one or more highlight selection criteria include game genres, game titles, broadcasters, and time periods.

26. The method as recited in claim 7, further comprising:

receiving highlight viewing preferences from spectators of the respective broadcasts via game spectating user interfaces on the spectator devices;

determining a set of two or more of the highlight segments according to the highlight viewing preferences of a particular spectator;

generating a highlight reel including the set of highlight segments; and

streaming the highlight reel to at least the spectator device of the particular spectator for display via the UI.

27. A non-transitory computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to implement a highlight service for a game spectating system, the highlight service configured to:

receive game event data from a plurality of game systems according to an application programming interface (API) to the highlight service, wherein the game event data received from a given game system indicates one or more notable events in a game executing on the respective game system as determined from the respective game event data, and wherein the game event data includes participant highlight tags that indicate one or more participant-specified notable events in the game;

determine a plurality of highlight segments from a plurality of broadcasts, wherein one or more of the plurality of highlight segments are determined according to the received game event data and the participant highlight tags and satisfying a spectator voting threshold, wherein each broadcast shows game play of a respective broadcaster participating as a player in a game executing on one of the plurality of game systems, wherein each highlight segment includes a portion of a respective broadcast showing a notable event from a respective game; and

present at least some of the plurality of highlight segments from the plurality of broadcasts for viewing on a plurality of spectator devices via a user interface (UI) to the highlight service.

28. The non-transitory computer-readable storage medium as recited in claim 27, wherein the highlight service is further configured to:

obtain highlight data indicating notable events in the plurality of broadcasts, wherein the highlight data includes one or more of broadcaster and spectator audio input to the broadcasts, broadcaster and spectator text chat input to the broadcasts, and broadcaster and spectator video input to the broadcasts; and

analyze the highlight data to generate analysis data for the plurality of broadcasts, wherein the analysis data for a given broadcast indicates one or more notable events in the respective broadcast as determined from the highlight data for the respective broadcast; and

determine at least one of the plurality of highlight segments according to the analysis data.