System and Method for Identifying, Analyzing, and Reporting on Players in a Game from Video

• US 20170154222A1
• Filed: 11/25/2016
• Published: 06/01/2017
• Est. Priority Date: 11/26/2015
• Status: Active Grant

• Current Assignee: Robert Zakaluk
• Original Assignee: Robert Zakaluk
• Inventors: Zakaluk, Robert
• Granted Patent: US 9,977,966 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program

Abstract

The invention is a system and method used for detection, analysis, and reporting on player metrics of a sporting event using video images. The main components of the invention are: grabbing a digital video image frame or grabbing an analog video image frame and converting the analog video image frame to a digital video image frame, extracting field of play markings from the video image frame(s) for use as image control points, creating a perspective projection registration model based upon the pairing of image control points with a set of user-defined control points representative of the field of play, extracting player image locations on each team from the video image frames, and applying the perspective projection transformation model to register participant image locations to their respective planimetric coordinate position on the user defined field of play. In the case of processing multiple video image frames, to correct errors across perspective projection transformation models developed on each image frame, a final bundle adjustment affine transformation model is applied to correct participant locations. Analysis of players use network analysis, pattern analysis, spatial interpolation, hypothesis testing, or forecast modeling can then be performed with reports in tabular, chart and graphic, or cartographic formats.

Claims

1. A method for improved automated processing of a single image frame or multiple image frames acquired from a video source, said processing being performed by one or more computers and comprising the following computer-implemented steps of:

a) selecting an image frame from said single image frame or from said multiple image frames,

b) creating multispectral ratios, multispectral indices, and multispectral transformations from the wavebands of said selected image frame from step (a),

c) producing a field of play mask using polychotomous classification of said multispectral ratios, multispectral indices, and multispectral transformations from step (b),

d) identifying field of play marking object candidates from said selected image frame from step (a) using polychotomous classification of said multispectral ratios, multispectral indices, and multispectral transformations from step (b) under said field of play mask from step (c),

e) creating univariate object shape measures of said field of play marking object candidates from step (d),

f) creating field of play markings with topology from said field of play marking object candidates from step (d) using conditional expressions applied to said multispectral ratios, multispectral indices, and multispectral transformations from step (b) and said univariate object shape measures from step (e), or ANN modelling inputs of said multispectral ratios, multispectral indices, and multispectral transformations from step (b) and said univariate object shape measures from step (e), or the Radon transform,

g) creating xy image control points from erosion of said field of play markings with topology from step (f),

h) creating field of play control lines with topology using linear regression modelling from said field of play markings with topology from step (f),

i) creating xy image control points with topology from intersections of said field of play control lines with topology from step (h),

j) building a point to point file by combining said image control points from steps (g) and (i) and pairing said image control points from steps (g) and (i) with the predefined field of play reference control points,

k) creating a perspective projection image to field of play georeferencing transformation model from said point to point file from step (j),

l) identifying player object candidates from said selected image frame using polychotomous classification of said multispectral ratios, multispectral indices, and multispectral transformations from step (b) under said field of play mask from step (c),

m) creating univariate object shape measures of the said player object candidates from step (l),

n) outputting and labelling player object fragments by team from said player object candidates from step (l) using ANN models of inputs containing said multispectral ratios, multispectral indices, and multispectral transformations from step (b) and said univariate object shape measures from step (m),

o) creating XY Cartesian player fragment locations by team, registered to the field of play, by applying the said transformation model from step (k) to said player object fragments by team from step (n),

p) aggregate said XY Cartesian player fragment locations by team from step (o) to identify single XY Cartesian player locations for each player on each team, and

q) save player locations, team identifier, the perspective projection to planimetric transformation model, control points, rink topology, and image frame identifier on a non-transitory computer readable medium.

2. The method of claim 1 wherein said multiple image frames are acquired from the video source, and at least some of said multiple image frames vary from one another in scale and oblique view angle.

3. The method of claim 1 comprising performing steps (a) to (q) on said multiple image frames, and producing an automatic final bundle adjustment model to correct participant location errors across said multiple image frames by performing the following additional computer-implemented steps of:

r) transforming said image control points from steps (g) and (i) of each of said multiple image frames using a perspective projection image frame georeferencing transformation model from step (k) and relating said transformed image control points to all predefined field of play reference control points,

s) building a final bundle adjustment affine transformation model using results from step (r),

t) applying said final bundle adjustment affine transformation model from step (s) on said single XY Cartesian player locations for each player on each team from step (p) from said multiple image frames,

u) storing said final bundle adjustment affine transformation model from step (s) and final adjusted player location results from step (t) in non-transitory computer readable memory.

4. The method of claim 1 comprising a further computer-implemented step of performing CG calculations and performing a network analysis based thereon, including calculation of flow speed, flow direction, distance between team paths, and/or dispersal of team paths.

5. The method of claim 1 comprising a further computer-implemented step of performing CG calculations and performing a pattern analysis based thereon, including calculation of:

player nearest neighbor distances, player zone coverage, team density, team adjacency, team connectivity, and/or player isolation.

6. The method of claim 1 comprising a further computer-implemented step of performing CG calculations and performing a spatial interpolation analysis based thereon, including performance of trend surface modelling, spline interpolations, inverse distance weighting, or kriging.

7. The method of claim 1 comprising a further computer-implemented step of performing of a hypothesis testing analysis based upon CG calculations, which is comprised of statistical analysis of a null and alternative hypothesis of measurements derived from player locations.

8. The method of claim 1 comprising a further computer-implemented step of performing of a forecast analysis based upon CG calculations, which is comprised of ANN models, polynomial regression, multivariate regression, and exponential smoothing to predict future outcomes.

9. The method of claim 1 further comprising computer generation of tables, graphs, cartography, and alerts, and analog or digital presentation of at least one of said tables, graphs, cartography, and alerts.

10. The method of claim 9 wherein said tables are comprised of:

numeric, text, or alphanumeric data.

11. The method of claim 9 wherein said graphs are comprised of:

pie, bar/column, line, flowchart, or point graphs.

12. The method of claim 9 wherein said cartography is comprised of:

dasymetric mapping, choropleth mapping, isarithmic mapping, dot mapping, proportional symbol mapping, cartograms, or flow mapping cartography.

13. The method of claim 9 wherein said alerts are comprised of:

text, sound, image, or electronic signals.

14. The method of claim 1 comprising wide-cast electronic delivery of output data from one or more analyses performed in said method.

15. Non-transitory computer readable memory having stored thereon statements and instructions for execution by a computer processor to perform the method of claim 1.

16. A system for improved automated processing of a single image frame or multiple image frames acquired from a video source, the system comprising one or more computers having at least one processor and non-transitory computer readable memory connected to said at least one processor, said computer readable memory having stored thereon statements and instructions for execution by said at least one processor to perform the method of claim 1.

17. A method of producing an automatic final bundle adjustment model to correct participant location errors across multiple image frames that have been analyzed to detect and locate players from opposing teams among a plurality of sport participants in said multiple image frames, the method comprising the following computer-implemented steps of:

a) transforming image control points from each of said multiple image frames using a respective image frame georeferencing transformation model and relating said transformed image control points to all predefined field of play reference control points,

b) building a final bundle adjustment model using results from step (a),

c) applying said final bundle adjustment model from step (b) on single Cartesian XY player locations for each player on each team, and

d) storing said final bundle adjustment model from step (b) and final adjusted player location results from step (c) in non-transitory computer readable memory.

18. Non-transitory computer readable memory having stored thereon statements and instructions for execution by a computer processor to perform the method of claim 17.

19. A system for producing an automatic final bundle adjustment model to correct participant location errors across multiple image frames that have been analyzed to detect and locate players from opposing teams among a plurality of sport participants in said multiple image frames, the system comprising one or more computers having at least one processor and non-transitory computer readable memory connected to said at least one processor, said computer readable memory having stored thereon statements and instructions for execution by said at least one processor to perform the method of claim 17.