Methods and systems of spatiotemporal pattern recognition for video content development

• US 10,460,176 B2
• Filed: 05/19/2017
• Issued: 10/29/2019
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Genius Sports SS LLC (Genius Sports Ltd.)
• Original Assignee: Second Spectrum, Inc. (Genius Sports Ltd.)
• Inventors: Chang, Yu-Han, Maheswaran, Rajiv, Su, Jeffrey Wayne, Hollingsworth, Noel
• Primary Examiner(s): Tran, Loi H
• Application Number:
  US15/600,355
• Publication Number:
  US 20170255826A1
• Time in Patent Office: 893 Days
• Field of Search: 386282, 386248
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4532 : involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4662 : characterized by learning algorithms
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications

Abstract

An enhanced video of an event in a first video feed, which is identified by a spatiotemporal pattern recognition algorithm that uses machine learning for understanding the event, is produced by including in the enhanced video an animation that characterizes a person'"'"'s motions that are derived from a machine learning-based understanding of an event in a second video.

Claims

1. A method for delivering personalized video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed of a professional game, wherein the understanding developed by the machine learning includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;

capturing 3D video of a non-professional player as a data feed;

developing an understanding using the machine learning of at least one event within the data feed relating to motion of the non-professional player; and

automatically, under computer control, providing an enhanced video feed that mixes video of the non-professional player with the at least one video feed of the professional game and represents the nonprofessional player as an animation having attributes based on the data feed relating to motion of the non-professional player playing within a context of the professional game based on the understanding of the at least one event within the at least one video feed of the professional game and the data feed relating to the motion of the non-professional player.

2. The method of claim 1, wherein using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

3. The method of claim 1, wherein using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.

4. The method of claim 3, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

5. The method of claim 3, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

6. The method of claim 1, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.