METHODS AND SYSTEMS OF SPATIOTEMPORAL PATTERN RECOGNITION FOR VIDEO CONTENT DEVELOPMENT

• US 20170255826A1
• Filed: 05/19/2017
• Published: 09/07/2017
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Genius Sports SS LLC (Genius Sports Ltd.)
• Original Assignee: Second Spectrum, Inc. (Genius Sports Ltd.)
• Inventors: Chang, Yu-Han, Maheswaran, Rajiv, Su, Jeffrey Wayne, Hollingsworth, Noel Grant
• Granted Patent: US 10,460,176 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4532 : involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4662 : characterized by learning algorithms
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications

Abstract

Presenting event-specific video content that conforms to a user selection of an event type includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed, extracting the video content displaying the at least one event and associating the understanding with the video content in a video content data structure. A user interface is configured to permit a user to indicate a preference for at least one event type that is used to retrieve and provide corresponding extracted video content with the data structure in a new video feed.

Claims

1. A method for enabling a user to express preferences relating to display of video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;

automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure;

providing a user interface configured to permit a user to indicate a preference for at least one event type;

upon receiving an indication of the preference by the user, retrieving the at least one video content data structure that was determined by the machine learning to be associated with the at least one event type indicated by the user; and

providing the user with a video feed containing the video content and including the at least one video content data structure.

2. The method of claim 1, wherein using machine learning to develop the understanding of the at least one event further comprises using events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

3. The method of claim 1, wherein using machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the at least one event, one or more feeds of tracking video for the at least one event, and one or more play-by-play data feeds of the at least one event.

4. The method of claim 3, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.

5. The method of claim 3, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds.

6. The method of claim 1, further comprising at least one of validating the understanding and modifying the understanding developed by the machine learning using a hierarchy involving at least two of at least one algorithm, at least one human operator, and at least one input feed where at least one item in the hierarchy used for validating the understanding is based on a nature of the one or more input feeds.

7. The method of claim 1, wherein the user interface is of at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, a touch screen device, a virtual reality or augmented reality headset, and a smart phone.

8. The method of claim 1, wherein the user interface further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.

9. The method of claim 1, wherein the understanding developed by the machine learning includes a context for the at least one of event and the context is stored with the at least one video content data structure.

10. The method of claim 9, wherein the user interface further comprises an element configured to permit a user to indicate a preference for the context.

11. The method of claim 10, further comprising retrieving portion of the video content corresponding to the context and displaying the portion of the video content corresponding to the context to the user upon receiving an indication of the preference for the context.

12. The method of claim 11, wherein the context comprises at least one of a presence of a preferred player in the at least one video feed, a preferred matchup of players in the at least one video feed, a preferred team in the at least one video feed, and a preferred matchup of teams in the at least one video feed.

13. The method of claim 1, wherein the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein the at least one metric is based at least in part on the understanding developed by the machine learning.

14. The method of claim 1, wherein extracting the content displaying the at least one event includes automatically extracting a cut from the video feed using a combination of the understanding of events developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

15. The method of claim 14, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

16. A method for enabling a mobile application that allows user interaction with video content, the method comprising:

taking a video feed;

processing the video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of an event within the video feed, wherein the understanding includes identifying context information relating to the event and an entry in a relationship library at least detailing a relationship between two visible features of the video feed;

automatically, under computer control, extracting content displaying the event and associating the extracted content with the context information;

producing a video content data structure that includes the context information; and

automatically, under computer control, producing a story that includes the video content data structure, wherein a portion of the story is based on a user preference, the context information, and the video content data structure.

17. The method of claim 16, wherein extracting the content displaying the event includes automatically extracting a cut from the video feed using a combination of the understanding of the event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

18. The method of claim 17, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

19. The method of claim 16, further comprising using a combination of the understanding developed with the machine learning of the event from the video feed and an understanding developed with the machine learning of another input feed selected from a group consisting of a broadcast video feed, an audio feed, and a closed caption feed to at least one of edit the video cut and combine the video cut with other content.

20. The method of claim 19, wherein the understanding developed with the machine learning of the other input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

21. The method of claim 16, further comprising automatically developing a semantic index of a video feed based on the understanding developed with the machine learning of at least one event in the video feed indicative of a time of the event in the video feed and a location of a display of the event in the video feed.

22. The method of claim 21, wherein the location of the display of the event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.

23. The method of claim 21, further comprising providing the semantic index of the video feed with the video feed in combination to enable augmentation of the video feed.

24. The method of claim 23, wherein the augmentation of the video feed includes adding content based on to the location of the display and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.

25. The method of claim 16, wherein using machine learning to develop the understanding of the event further comprises using events in position tracking data over time obtained from at least one of the video feed and a chip-based player tracking system, and wherein the understanding developed with machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

26. The method of claim 16, wherein using machine learning to develop the understanding of the event further comprises aligning multiple unsynchronized input feeds related to the event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

27. The method of claim 26, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

28. The method of claim 26, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on a nature of the input feeds.

29. The method of claim 16, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

30. The method of claim 16, wherein the user preference for a type of content is based on at least one of a user expressed preference and a preference that is inferred based on user interaction with an item of content.

31. A system for enabling a user to express preferences relating to display of video content, comprising:

a machine learning facility that uses at least one spatiotemporal pattern recognition algorithm for developing an understanding of at least one event within at least one video feed to determine at least one event type, wherein the understanding includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;

a video production facility for automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in at least one video content data structure; and

a server for serving data to a user interface that is configured to permit a user to indicate a preference for at least one event type, that retrieves the at least one video content data structure that was determined by the machine learning to have a event type preferred by the user, and that provides the user with a video feed containing the event type preferred by the user.

32. The system of claim 31, wherein the user interface that is configured to permit the user to indicate the preference for at least one event type is at least one of a mobile application, a browser, a desktop application, a remote control device, a tablet, and a smart phone.

33. The system of claim 31, wherein the user interface that is configured to permit the user to indicate the preference for at least one event type further comprises an element configured to permit a user to indicate a preference as to how content will be presented to the user.

34. The system of claim 31, wherein the understanding developed by the machine learning facility includes a context for the at least one event and the context is stored with the at least one video content data structure.

35. The system of claim 34, wherein the user interface further comprises an element configured to permit a user to indicate a preference for at least one context.

36. The system of claim 35, wherein the server for serving data to the user interface retrieves video content corresponding to the preference for at least one context and displays the video content to the user upon receiving an indication of the preference for the at least one context.

37. The system of claim 36, wherein the at least one context comprises at least one of a presence of a preferred player in the video feed, a preferred matchup of players in the video feed, a preferred team in the video feed, and a preferred matchup of teams in the video feed.

38. The system of claim 31, wherein the user interface is configured to permit a user to select at least one of a metric and a graphic element to be displayed on the video feed, wherein the metric is based at least in part on the understanding developed with the machine learning facility.

39. The system of claim 31, wherein the machine learning facility to develop the understanding of the at least one event uses events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

40. The system of claim 31, wherein the machine learning facility to develop the understanding of the at least one event aligns multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

41. The system of claim 40, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the at least one event.

42. The system of claim 40, wherein the video production facility at least one of validates and modifies the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

43. The system of claim 31, wherein the video production facility at least one of validates the understanding and modifies the understanding developed by the machine learning facility using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

44. The system of claim 31, wherein the video production facility for automatically, under computer control, extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning facility and an understanding developed with the machine learning facility of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

45. The system of claim 44, wherein the understanding developed with the machine learning facility of the another input feed includes at least one of a portion of content of a broadcast commentary and a change in camera view in the at least one video feed.

46. A method for delivering personalized video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed to determine at least one event type, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;

automatically, under computer control, extracting the video content displaying the at least one event and associating the understanding developed with the machine learning of the event type with the video content in a video content data structure;

developing a personal profile for a user based on at least one of expressed preferences of the user, information about the user, and information collected about actions taken by the user with respect to at least one event type; and

upon receiving an indication of the user profile, retrieving at least one video content data structure that was determined by the machine learning to have a event type likely to be preferred by the user based on the personal profile for the user.

47. The method of claim 46, wherein using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed by the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

48. The method of claim 46, wherein using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.

49. The method of claim 48, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

50. The method of claim 48, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

51. The method of claim 46, further comprising at least one of validating and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

52. The method of claim 51, wherein extracting the video content displaying the at least one event includes automatically extracting a cut from the at least one video feed using a combination of the understanding of the at least one event developed with the machine learning and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

53. The method of claim 52, wherein the understanding of the other input feed developed with the machine learning includes at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

54. A method for delivering personalized video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of at least one event within the at least one video feed of a professional game, wherein the understanding developed by the machine learning includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed

develop an understanding using the machine learning of at least one event within a data feed relating to motion of a non-professional player; and

automatically, under computer control, providing an enhanced video feed that represents the non-professional player playing within a context of the professional game based on the understanding of the at least one event within the at least one video feed of the professional game and a data feed relating to the motion of the non-professional player.

55. The method of claim 54, further comprising providing a facility having cameras for capturing 3D motion data, and capturing video of a non-professional player to provide the data feed for the non-professional player.

56. The method of claim 55, wherein the non-professional player is represented by mixing video of the non-professional player with at least one video of the professional game.

57. The method of claim 56, wherein the non-professional player is represented as an animation having attributes based on the data feed relating to motion of the non-professional player.

58. The method of claim 54, wherein using the machine learning to develop the understanding of the at least one event further comprises using the at least one event in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

59. The method of claim 54, wherein using the machine learning to develop the understanding of the at least one event further comprises aligning multiple unsynchronized input feeds related to the at least one event using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from a group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the at least one event.

60. The method of claim 59, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

61. The method of claim 59, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

62. The method of claim 54, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.