System and method for synchronizing and predicting game data from game video and audio data

• US 10,569,175 B2
• Filed: 02/02/2018
• Issued: 02/25/2020
• Est. Priority Date: 02/03/2017
• Status: Active Grant

• Current Assignee: Turnt Gaming LLC
• Original Assignee: Taunt Inc.
• Inventors: Kosai, Ryan T, Santosa, Ansel C. S., Robinson, Sean M., Crenshaw, John B
• Primary Examiner(s): McClellan, James S.
• Assistant Examiner(s): Iannuzzi, Peter J
• Application Number:
  US15/887,117
• Publication Number:
  US 20180221769A1
• Time in Patent Office: 753 Days
• Field of Search: None
• CPC Class Codes: A63F 13/493 Resuming a game, e.g. after pausing, malfunction or power failure
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/798 for assessing skills or for ranking players, e.g. for generating a hall of fame computing the game score A63F13/46
• CPC Class Codes: A63F 13/85 Providing additional services to players
• CPC Class Codes: A63F 13/86 Watching games played by other players
• CPC Class Codes: A63F 2300/535 for monitoring, e.g. of user parameters, terminal parameters, application parameters, network parameters
• CPC Class Codes: A63F 2300/6063 for sound processing
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G07F 17/32 for games, toys, sports, or amusements
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence

Abstract

Systems and methods for capturing and utilizing multimedia-based analytics to generate predictions about game-play in computer-based games and contests. In an embodiment, a video feed analyzes one or more video feeds frame by frame to determine one or more features within each video frame. Each feature may be collected in a central data collector and converted to one or more metrics about each collected parameter or feature. These metrics may be used to analyze and predict various aspects of the underlying game or contest, such as probability of winning by one side or the other. Further, the predictive data and analytical data may be displayed on a display separate from actual gameplay or may be overlaid upon the game play video feed. In this manner, the overlay video may be synchronized with the original video feed to present predictive data in conjunction with the video data used to influence the predictive data at the time.

Claims

1. A computer-based device, comprising:

a data receiver module for receiving video data from one or more external sources, the video data comprises gameplay video frames depicting views of gameplay of a computer-based game from one or more perspectives, the gameplay video frames of the video data containing underlying game data indicative of the computer-based game being played in sequential time;

a video analysis module coupled to the data receiver module and configured to extract the underlying game data from the gameplay video frames of the video data and further configured to store the extracted game data in a data store, the video analysis module being configured to analyze the video data on a frame-by-frame basis using one or more computer vision techniques on the gameplay video frames to extract the underlying game data from the gameplay video frames;

a prediction module coupled to the video analysis module and configured to analyze the extracted game data in sequential time and to generate predictions of one or more game outcomes corresponding to the computer-based game; and

an output module coupled to the prediction module and configured to generate an output having one or more aspects of the received video data and one or more aspects of the generated predictions.

2. The computer-based device of claim 1, further comprising a game data application programming interface coupled to the prediction module and configured to receive differing game data from the computer-based game that is different from the extracted game data extracted from the gameplay video frames of the video data, wherein the prediction module is configured to analyze the differing game data and the extracted game data to generate the predictions of the one or more game outcomes.

3. The computer-based device of claim 1, further comprising a synchronization module coupled to the prediction module and configured to synchronize the generated predictions with the output.

4. The computer-based device of claim 3, wherein the synchronization module further comprises an audio synchronization module configured to synchronize the generated predictions with the output based on audio cues in the output.

5. The computer-based system of claim 1, further comprising an audio input module coupled to the receiver module and configured to extract gameplay audio data from a feed that comprises the video data and further configured to send the extracted gameplay audio data to the prediction module, wherein the prediction module is configured to analyze the extracted gameplay audio data to generate the predictions of the one or more game outcomes.

6. The computer-based device of claim 1, further comprising an overlay video creation module coupled to the prediction module and configured to overlay the generated predictions onto the received video data for sending as the output.

7. The computer-based device of claim 1, further comprising one or more metrics conversion modules coupled to the prediction module and configured to convert a stream of computer vision features collected using the one or more computer vision techniques to metrics data, wherein the prediction module is configured to generate the predictions of the one or more game outcomes based on the metrics data.

8. A computer-based system, comprising:

a first gaming computer coupled to a computer network and having a game executable thereon;

a second gaming computer coupled to the computer network and having the game executable thereon simultaneous to the execution of the game on the first gaming computer; and

a server computer coupled to the computer network and configured to communicate with the first gaming computer and the second gaming computer while the game is executing at the first gaming computer and the second gaming computer, the server computer including;

a data receiver module for receiving video data from the first and second gaming computers, the video data comprises gameplay video frames depicting views of gameplay of the game generated by at least the first gaming computer and the second gaming computer, the gameplay video frames of the video data containing underlying game data indicative of the game executing in sequential time;

a video analysis module coupled to the data receiver module and configured to extract the underlying game data from the gameplay video frames of the video data and further configured to store the extracted game data in a data store, the video analysis module being configured to analyze the video data on a frame-by-frame basis using one or more computer vision techniques on the gameplay video frames to extract the underlying game data from the gameplay video frames;

a prediction module coupled to the video analysis module and configured to analyze the extracted game data in sequential time and to generate predictions of one or more game outcomes corresponding to the game; and

an output module coupled to the prediction module and configured to generate an output having one or more aspects of the received video data and one or more aspects of the generated predictions.

9. The computer-based system of claim 8, further comprising:

a third gaming computer coupled to the computer network and having the game executable thereon simultaneous to the execution of the game on the first gaming computer and the second gaming computer;

wherein the server computer is further configured to receive video data from the third gaming computer and configured to influence the predictions in response to video data received from the third gaming computer.

10. The computer-based system of claim 8, further comprising an observer computer coupled to the computer network, the observer computer configured to receive an output video feed simultaneous to the execution of the game at the first and second gaming computers and configured to display the output on a display.

11. The computer-based system of claim 8, wherein the server computer further comprises a video display configured to display an output video feed simultaneous to the execution of the game at the first and second gaming computers.

12. The computer-based system of claim 8, wherein the server computer further comprises a challenge management module configured to manage challenges related to the game simultaneous to the execution of the game on the first and second gaming computers.

13. A computer method, comprising:

receiving a multimedia signal from a remote computer system having a game being executed thereon, the game having an outcome based on gameplay, the multimedia signal comprises gameplay video frames depicting a view of the gameplay for the remote computer system;

extracting game data from the multimedia signal, the game data being extracted from the multimedia signal by preforming one or more computer vision techniques on the gameplay video frames of the multimedia signal on a frame-by-frame basis;

assimilating the extracted game data into a game analytics engine for analyzing the gameplay; and

generating a prediction of a game outcome in response to the analyzing.

14. The computer method of claim 13, further comprising generating a prediction video signal having aspects of the received multimedia signal and having aspects of the prediction.

15. The computer method of claim 14, further comprising synchronizing the aspects of the prediction with the received multimedia signal using extracted audio data from the received multimedia signal.

16. The computer method of claim 13, wherein the extracting game data further comprises identifying a graphical portion of a particular gameplay video frame included in the multimedia signal indicative of a game statistic by preforming the one or more computer vision techniques on the particular gameplay video frame and determining the game data from the identified graphical portion.

17. The computer method of claim 13, wherein the extracting game data further comprises identifying an alphanumeric portion of a particular gameplay video frame included in the multimedia signal indicative of a game statistic by performing the one or more computer vision techniques on the particular gameplay video frame and determining the game data from the identified alphanumeric portion.

18. The computer method of claim 13, wherein the extracting game data further comprises identifying an audio portion of the multimedia signal indicative of a game statistic and determining the game data from the identified audio portion.

19. The computer method of claim 13, further comprising:

receiving a second multimedia signal from a second remote computer system having the game being executed thereon, the second multimedia signal comprises second gameplay video frames depicting a second view of the gameplay for the second remote computer system;

extracting second game data from the second multimedia signal, the second game data being extracted from the second multimedia signal by preforming the one or more computer vision techniques on the second gameplay video frames of the second multimedia signal on a frame-by-frame basis;

assimilating the extracted second game data from the second multimedia signal into the game analytics engine for analyzing the gameplay; and

influencing the prediction of the game outcome in response to the analyzing of the extracted second game data from the second multimedia signal.

20. The computer method of claim 19, further comprising:

aggregating each multimedia signal and each prediction to generate an omniscient multimedia signal having aspects of each multimedia signal and each prediction; and

transmitting the omniscient multimedia signal to a third remote computer system.