Methods and systems of spatiotemporal pattern recognition for video content development

• US 10,460,177 B2
• Filed: 12/21/2018
• Issued: 10/29/2019
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Genius Sports SS LLC
• Original Assignee: Second Spectrum, Inc. (Genius Sports Ltd.)
• Inventors: Chan, Edward Shek, Chang, Yu-Han, Maheswaran, Rajiv Tharmeswaran, Su, Jeffrey Wayne
• Primary Examiner(s): Tran, Loi H
• Application Number:
  US16/229,457
• Publication Number:
  US 20190114485A1
• Time in Patent Office: 312 Days
• Field of Search: 386280
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4532 : involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4662 : characterized by learning algorithms
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications

Abstract

A media system generally includes a memory device that stores an event datastore that stores a plurality of event records, each event record corresponding to a respective event and event metadata describing at least one feature of the event. The media system (a) receives a request to generate an aggregated clip comprised of one or more media segments, where each media segment depicts a respective event; (b) for each event record from at least a subset of the plurality of event records, determines an interest level of the event corresponding to the event record; (c) determines one or more events to depict in the aggregated clip based on the respective interest levels of the one or more events; (d) generates the aggregated clip based on the respective media segments that depict the one or more events; and (e) transmits the aggregated clip to a user device.

Claims

1. A method comprising:

receiving a first broadcast video teed capturing a filmed occurrence, the first broadcast video feed comprising a plurality of broadcast video frames captured by a broadcast camera, wherein the first broadcast video teed is a video feed that is consumable by a client device;

receiving a tracking camera video feed corresponding to the filmed occurrence, the tracking camera video feed comprising a plurality of tracking video frames and being captured by a tracking camera having a position that is calibrated to a frame of reference;

tracking one or more respective pixel locations of an object detected in one or more respective broadcast video frames of the first broadcast video feed;

tracking one or more respective spatial locations of the object based on one or more respective tracking video frames where the object is detected in the tracking video feed;

time-aligning the first broadcast video feed with the tracking video feed based on the one or more respective pixel locations and the one or more respective spatial locations;

generating a spatio-temporal index corresponding to the filmed occurrence based on the time-alignment of the first broadcast video feed with the tracking video feed, wherein the spatio-temporal index indexes spatio-temporal information relating to objects detected in the broadcast video feed and/or the tracking video feed;

spatially aligning an augmentation item with respect to the object in a subset of the one or more broadcast video frames based on the spatio-temporal index; and

generating an augmented video stream having one or more augmented video frames based on the subset of the one or more broadcast video frames and the spatial alignment of the augmentation item with respect to the object, wherein the augmentation item and the object are spatially aligned in the augmented video stream.

2. The method of claim 1, wherein the filmed occurrence is a sporting competition, and the object is a participant in the sporting competition and the augmentation item is a statistic relating to the participant that is obtained from a data feed corresponding to the sporting competition that is time aligned to the broadcast video feed.

3. The method of claim 1, further comprising:

associating an advertisement with a type of event that is detectable in the subset of broadcast video frames; and

wherein generating the augmented video stream further comprises;

detecting an event depicted in a set of broadcast video frames of the broadcast video feed that is of the type of event associated with the advertisement; and

in response to detecting the event, augmenting at least one broadcast video frame with the advertisement.

4. The method of claim 1, wherein the augmentation item is an advertisement, and the advertisement is spatially associated with the object that is detected in the subset of broadcast video frames.

5. The method of claim 1, further comprising transmitting the augmented video stream to a client device.