METHODS AND SYSTEMS OF SPATIOTEMPORAL PATTERN RECOGNITION FOR VIDEO CONTENT DEVELOPMENT

• US 20170238055A1
• Filed: 05/04/2017
• Published: 08/17/2017
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Genius Sports SS LLC (Genius Sports Ltd.)
• Original Assignee: Second Spectrum, Inc. (Genius Sports Ltd.)
• Inventors: Chang, Yu-Han, Maheswaran, Rajiv, Su, Jeffrey Wayne, Hollingsworth, Noel Grant
• Granted Patent: US 10,521,671 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/60 Generating or modifying game content before or while executing the game program, e.g. authoring tools specially adapted for game development or game-integrated level editor
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4345 Extraction or processing of SI, e.g. extracting service information from an MPEG stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4532 : involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4662 : characterized by learning algorithms
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 : Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications

Abstract

Providing enhanced video content includes processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events and to determine at least one event type for each of the plurality of events. The event type includes an entry in a relationship library detailing a relationship between two visible features. Extracting and indexing a plurality of video cuts from the video feed is performed based on the at least one event type determined by the understanding that corresponds to an event in the plurality of events detectable in the video cuts. Lastly, automatically and under computer control, an enhanced video content data structure is generated using the extracted plurality of video cuts based on the indexing of the extracted plurality of video cuts.

Claims

18. A method, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the video feed;

extracting a plurality of video cuts from the at least one video feed;

indexing the plurality of video cuts based on the at least one event type determined by the understanding developed by machine learning; and

providing a mobile application having a user interface configured to permit a user to find the extracted plurality of video cuts based on the indexing of the extracted plurality of video cuts with the mobile application.

19. The method of claim 18, wherein the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern selected from the group consisting of relative motion of two visible features toward each other for at least a duration threshold, acceleration of motion of at least two visible features with respect to each other for acceleration of at least an acceleration threshold, rate of motion of two visible features toward each other, projected point of intersection of the two visible features, and separation distance between the two visible features being less than a separation threshold.

20. The method of claim 18, the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern recognized by adjusting at least one of an input feature and a weight within a machine learning system, wherein the input type is selected from the group consisting of relative direction of motion of at least two visible features, duration of relative motion of visible features with respect to each other, rate of motion of at least two visible features with respect to each other, acceleration of motion of at least two visible feature with respect to each other, projected point of intersection of at least two visible features with respect to each other and separation distance between at least two visible features with respect to each other

21. The method of claim 18, wherein the understanding developed by the machine learning includes generating at least one metric associated with at least one event of the plurality of events and wherein the user interface of the mobile application is configured to permit the user to enhance a video cut from the plurality of video cuts by selecting the metric to be included in the video cut.

22. The method of claim 18, wherein the user interface of the mobile application is configured to permit the user to share an edited video via the mobile application.

23. The method of claim 18, wherein using the machine learning to develop the understanding of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

24. The method of claim 18, wherein using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

25. The method of claim 24, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event of the plurality of events.

26. The method of claim 24, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

27. The method of claim 18, further comprising at least one of validating the understanding and modifying the understanding using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validating is based on a nature of the input feed.

28. The method of claim 18, wherein extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

29. The method of claim 28, wherein the understanding of the another input feed includes an understanding developed with the machine learning of at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

30. A method for providing enhanced video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events;

extracting a plurality of video cuts from the at least one video feed;

indexing the extracted plurality of video cuts based on the at least one event type determined by the understanding developed by machine learning;

determining at least one pattern relating to the extracted plurality of video cuts; and

indexing at least a portion of the plurality of the extracted video cuts with an indicator of the pattern.

1. A method for providing enhanced video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events and to determine at least one event type for each of the plurality of events within the at least one video feed, wherein the at least one event type includes an entry in a relationship library at least detailing a relationship between two visible features of the at least one video feed;

extracting a plurality of video cuts from the at least one video feed;

indexing the extracted plurality of video cuts based on the at least one event type determined by the understanding developed by the machine learning that corresponds to an event in the plurality of events detectable in the plurality of video cuts; and

automatically, under computer control, generating an enhanced video content data structure using the extracted plurality of video cuts based on the indexing of the extracted plurality of video cuts.

2. The method of claim 1, wherein the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern selected from the group consisting of relative motion of two visible features toward each other for at least a duration threshold, acceleration of motion of at least two visible features with respect to each other for acceleration of at least an acceleration threshold, rate of motion of two visible features toward each other, projected point of intersection of the two visible features, and separation distance between the two visible features being less than a separation threshold.

3. The method of claim 1, wherein the at least one spatiotemporal pattern recognition algorithm is based on at least one pattern recognized by adjusting at least one of an input feature and a weight within a machine learning system, wherein the input type is selected from the group consisting of relative direction of motion of at least two visible features, duration of relative motion of visible features with respect to each other, rate of motion of at least two visible features with respect to each other, acceleration of motion of at least two visible feature with respect to each other, projected point of intersection of at least two visible features with respect to each other and separation distance between at least two visible features with respect to each other.

4. The method of claim 1, wherein automatically generating the enhanced video content data structure includes combining the understanding of the plurality of events with an understanding of at least one of a broadcast video event or a broadcast audio event developed with the machine learning.

5. The method of claim 1, wherein generating the enhanced video content data structure is based at least in part on at least one of a user preference and a user profile for a user for which the enhanced video content data structure is generated.

6. The method of claim 1, further comprising providing a user interface for display on a mobile device, wherein the user interface includes at least one of a search option and a filtering option to allow a user to at least one of specify and select a description of a type of event, wherein the enhanced video content data structure is generated to match the description.

7. The method of claim 1, wherein using the machine learning to develop the understanding of the event further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed by the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

8. The method of claim 1, wherein using the machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

9. The method of claim 8, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two event types related to the event.

10. The method of claim 8, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

11. The method of claim 1, further comprising at least one of validating the understanding and modifying the understanding developed by the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

12. The method of claim 1, further comprising automatically developing a semantic index of the at least one video feed based on the understanding developed with the machine learning of at least one event of the plurality of events in the video feed to indicate a game time of the at least one event in the video feed and a location of a display of the at least one event in the video feed.

13. The method of claim 12, wherein the location of the display of the at least one event in the video feed includes at least one of a pixel location, a voxel location, a raster image location.

14. The method of claim 12, further comprising providing the semantic index of the video feed with the video feed configured to enable semantic-based augmentation of the video feed.

15. The method of claim 14, wherein augmentation of the video feed includes adding content based on the identified location and enabling at least one of a touch interface feature and a mouse interface feature based on the identified location.

16. The method of claim 15, wherein extracting the plurality of video cuts includes automatically extracting a cut from the video feed using a combination of the machine learning developed understanding of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

17. The method of claim 16, wherein the understanding developed with the machine learning of the another input feed includes understanding at least one of a portion of content of a broadcast commentary and a change in camera view in the another input feed.

31. The method of claim 30, wherein the at least one pattern is developed by applying machine learning.

32. The method of claim 31, wherein the understanding developed by the machine learning includes identifying at least one player involved in an event, and wherein indexing of the extracted plurality of video cuts includes identifying at least one player represented in at least one of the video cuts from the plurality of the video cuts.

33. The method of claim 31, wherein the at least one pattern relates to a series of same event types involving a same player over time.

34. The method of claim 30, wherein the plurality of video cuts includes a player during multiple, identical event types over time.

35. The method of claim 30, further comprising providing an enhanced video feed that shows a player during the plurality of events over time, wherein the enhanced video feed is at least one of a simultaneous, superimposed video of the player involved in multiple, identical event types and a sequential video of the player involved in the same event type.

36. The method of claim 30, wherein determining at least one pattern includes identifying sequences of events that predict a given action that is likely to follow.

37. The method of claim 30, wherein determining the at least one pattern includes identifying similar sequences of events across the plurality of video feeds.

38. The method of claim 30, further comprising providing a user interface that enables a user to at least one of view and interact with the at least one pattern.

39. The method of claim 38, wherein the at least one pattern and the interaction options are personalized based on at least one of a user preference and a user profile.

40. The method of claim 30, wherein the at least one pattern relates to an anticipated outcome of at least one of a game and an event within a game.

41. The method of claim 40, further comprising providing a user with at least one of a statistic, trend information and a prediction based on the at least one pattern.

42. The method of claim 41, wherein the at least one the statistic, the trend information or the prediction is based on at least one of a user preference and a user profile.

43. The method of claim 30, wherein the at least one pattern relates to play of an athlete.

44. The method of claim 43, further comprising providing a comparison of the play of the athlete with another athlete based on a similarity of at least one of the extracted plurality of video cuts and the at least one pattern.

45. The method of claim 44, wherein the comparison is between a professional athlete and a non-professional user.

46. The method of claim 44, wherein the comparison is based on a similarity of a playing style of the professional athlete, as determined by the understanding developed with the machine learning of at least one event of the plurality of events and the at least one pattern, with at least one feature of the playing style of the non-professional user.

47. The method of claim 30, wherein the understanding developed with the machine learning of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

48. The method of claim 30, wherein the understanding developed with the machine learning of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators, wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

49. The method of claim 48, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

50. The method of claim 48, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

52. The method of claim 30, further comprising at least one of validating the understanding and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

53. The method of claim 52, wherein the understanding of the other input feed includes an understanding developed with the machine learning of at least one of a portion of content of a broadcast commentary and a change in camera view in the input feed.

52-1. The method of claim 30, wherein extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

54. A method for providing enhanced video content, comprising:

processing at least one video feed through at least one spatiotemporal pattern recognition algorithm that uses machine learning to develop an understanding of a plurality of events within at least one video feed to determine at least one event type for each of the plurality of events;

extracting a plurality of video cuts from the at least one video feed;

indexing the extracted plurality of video cuts based on at least one event type determined by the understanding developed by machine learning; and

automatically, under computer control, delivering the extracted plurality of video cuts to at least one user based on at least one of a user profile and a user preference.

55. The method of claim 54, wherein the at least one of the user preference and the user profile is continuously updated based on a user'"'"'s indication of at least one of liking and disliking at least one of video cuts of the plurality of extracted video cuts.

56. The method of claim 54, wherein the understanding developed with the machine learning is based on human-identified video alignment labels for semantic events.

57. The method of claim 54, wherein the at least one spatiotemporal pattern recognition algorithm uses time-aligned content from multiple input sources to develop the understanding with the machine learning.

58. The method of claim 57, wherein the at least one spatiotemporal pattern recognition algorithm uses a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds to handle the multiple input sources.

59. The method of claim 54, wherein using machine learning to develop the understanding of the plurality of events further comprises using the plurality of events in position tracking data over time obtained from at least one of the at least one video feed and a chip-based player tracking system, and wherein the understanding developed with the machine learning is based on at least two of spatial configuration, relative motion, and projected motion of at least one of a player and an item used in a game.

60. The method of claim 54, wherein using machine learning to develop the understanding of the plurality of events further comprises aligning multiple unsynchronized input feeds related to an event of the plurality of events using at least one of a hierarchy of algorithms and a hierarchy of human operators, and wherein the unsynchronized input feeds are selected from the group consisting of one or more broadcast video feeds of the event, one or more feeds of tracking video for the event, and one or more play-by-play data feeds of the event.

61. The method of claim 60, wherein the multiple unsynchronized input feeds include at least three feeds selected from at least two types related to the event.

62. The method of claim 60, further comprising at least one of validating and modifying the alignment of the unsynchronized input feeds using a hierarchy involving at least two of one or more algorithms, one or more human operators, and one or more input feeds.

63. The method of claim 54, further comprising at least one of validating the understanding and modifying the understanding developed with the machine learning using a hierarchy involving at least two of one or more algorithms, one or more human operators and one or more input feeds where at least one algorithm in the hierarchy for validation is based on the nature of the input feed.

64. The method of claim 54, wherein extracting the plurality of video cuts from the at least one video feed includes automatically extracting a cut from the at least one video feed using a combination of the understanding developed with the machine learning of the plurality of events and an understanding developed with the machine learning of another input feed selected from the group consisting of a broadcast video feed, an audio feed, and a closed caption feed.

65. The method of claim 64, wherein the understanding developed with the machine learning of the other input feed includes an understanding developed with machine learning of at least one video cut of the plurality of video cuts of a broadcast commentary and a change in camera view in the input feed.