Systems and methods for enhancing images in a video recording of a sports event

• US 8,077,917 B2
• Filed: 06/22/2006
• Issued: 12/13/2011
• Est. Priority Date: 07/01/2005
• Status: Active Grant

• Current Assignee: Topgolf Sweden AB
• Original Assignee: Daniel Forsgren
• Inventors: Forsgren, Daniel
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): ANSARI, TAHMINA N
• Application Number:
  US11/917,741
• Publication Number:
  US 20080199043A1
• Time in Patent Office: 2,000 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: A63B 2024/0034 during flight
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: G01N 15/1433 using image recognition
• CPC Class Codes: G01N 15/1484 microstructural devices
• CPC Class Codes: G01N 2015/1486 Counting the particles
• CPC Class Codes: G01N 2021/0321 One time use cells, e.g. integrally moulded
• CPC Class Codes: G01N 2021/0325 Cells for testing reactions, e.g. containing reagents
• CPC Class Codes: G01N 2021/0346 Capillary cells; Microcells
• CPC Class Codes: G01N 21/03 Cuvette constructions
• CPC Class Codes: G01N 21/11 Filling or emptying of cuvettes
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30224 Ball; Puck
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

A video signal representing rapid ball movement is produced from a series of source images. An initial image position for the moving ball is identified by, for each image, producing a difference image between sequential images. In the difference image, image elements representing a contents alteration below a threshold are allocated a first value, and those representing a contents alteration above or equal to the threshold are allocated a second value. A set of candidates is then identified, where each candidate is represented by a group of neighboring image elements that all contain the second value. The group must fulfill a ball size criterion. A ball selection algorithm selects an initial image position from the set of ball candidates. The ball is tracked, and a composite image sequence is generated wherein a synthetic trace representing the path of the moving ball is shown as successively added image data.

Claims

1. A system for producing a video signal representing a sports scene wherein, during at least a period, a ball moves in front of an essentially static background, the system comprising:

an image recording means adapted to register a series of source images of the scene;

a tracking means adapted to track a path of the moving ball through at least a sub-set of the images in said series, the tracking involving a differencing between consecutively registered images wherein image areas are identified that have been updated in a later registered image in relation to an earlier registered image; and

a graphics generator adapted to, based on said source images, generate a composite image sequence wherein synthetic image elements are combined with the images in said series, such that a trace representing the path of the moving ball is shown in the form of added image data successively overlaid the image contents of the images in said series,

wherein the tracking means is adapted to automatically identify an initial image position for the moving ball by;

producing, for each image in said series, a difference image between the image and an image subsequent thereto, wherein in the difference image all image elements representing a contents alteration below a threshold level are allocated a first value, and all image elements representing a contents alteration above or equal to the threshold level are allocated a second value,

identifying a set of ball candidates in which each ball candidate is represented by a group of neighboring image elements which all contain the second value and said group fulfills a ball size criterion, and

selecting the initial image position from the set of ball candidates based on a ball selection algorithm,

wherein the ball selection algorithm implies a stroke estimation procedure wherein the ball candidates in the set of ball candidates are analyzed in three consecutively registered images, the stroke estimation procedure comprising;

defining a respective path candidate line extending straight between each ball candidate in a first registered image and each ball candidate in a third registered image,

discarding each ball candidate for which, in a second image registered between the first and third registered images, no ball candidate is found within a threshold range from the path candidate line, and

defining a tentative ball path for all other path candidate lines.

2. The system according to claim 1, wherein it comprises a digital processing unit to which the image recording means is connected, and the digital processing unit comprises the tracking means.

3. The system according to claim 2, wherein the digital processing unit comprises the graphics generator.

4. The system according to claim 2, wherein the image recording means comprises a main camera and an auxiliary camera, the cameras are co-located and mechanically fixed to one another, and the digital processing unit is adapted to accomplish a parallax adjustment in respect of said cameras.

5. The system according to claim 1, wherein the tracking means is adapted to receive a distance parameter to form a basis for the ball selection algorithm, the distance parameter reflecting a distance between the image recording means and an estimated initial ball position, and the distance parameter determining the ball size criterion.

6. The system according to claim 5, wherein the tracking means comprises a machine interface towards the image recording means which is adapted to receive the distance parameter in the form of a focal distance signal.

7. The system according to claim 5, wherein the tracking means comprises a user interface adapted to receive a manually entered distance parameter.

8. The system according to claim 4, wherein the digital processing unit comprises a buffer means adapted to buffer images registered by the main camera during a delay period, and

is adapted to, based on a set of images registered by the auxiliary camera during the delay period, generate the trace in respect of at least a sub set of the images registered during the delay period.

9. The system according to claim 1, wherein the image recording means comprises an image sensor containing an array of sensor elements, and the image sensor is adapted to enable read out of image data recorded by a selected sub set of the sensor elements.

10. A method of producing a video signal representing a sports scene wherein, during at least a period, a ball moves in front of an essentially static background, the method comprising:

registering a series of source images of the scene;

tracking a path of the moving ball through at least a sub-set of the images in said series, the tracking involving a differencing between consecutively registered images wherein image areas are identified that have been updated in a later registered image in relation to an earlier registered image; and

generating, based on said source images a composite image sequence wherein synthetic image elements are combined with the images in said series, such that a trace representing the path of the moving ball is shown in the form of added image data successively overlaid the image contents of the images in said series, wherein the method comprises automatically identifying an initial image position for the moving ball by;

producing, for each image in said series, a difference image between the image and an image subsequent thereto, wherein in the difference image all image elements representing a contents alteration below a threshold level are allocated a first value, and all image elements representing a contents alteration above or equal to the threshold level are allocated a second value,

identifying a set of ball candidates in which each ball candidate is represented by a group of neighboring image elements which all contain the second value and said group fulfills a ball size criterion, and

selecting the initial image position from the set of ball candidates based on a ball selection algorithm,

wherein the ball selection algorithm implies a stroke estimation procedure wherein the ball candidates in the set of ball candidates are analyzed in three consecutively registered images, the stroke estimation procedure comprising;

defining a respective path candidate line extending straight between each ball candidate in a first registered image and each ball candidate in a third registered image,

discarding each ball candidate for which, in a second image registered between the first and third registered images, no ball candidate is found within a threshold range from the path candidate line, and

defining a tentative ball path for all other path candidate lines.

11. The method according to claim 10, wherein the ball size criterion specifies a range of ball image areas equivalent to a particular interval of image elements.

12. The method according to claim 10, wherein the ball selection algorithm comprises a roundness estimation step wherein any ball candidates are discarded which fail to fulfill at least one of:

a rotation symmetry requirement, and

a requirement pertaining to a relationship between a maximum diameter of the group of neighboring image elements and a combined area of the image elements in the group.

13. The method according to claim 10, wherein the ball selection algorithm comprises a position estimation step wherein any ball candidates are discarded which are located outside a primary area of the source image, the primary area representing a lower two-thirds portion of the source image.

14. The method according to claim 13, wherein the primary area represents a 50%-portion of the source image around a central vertical line through the source image.

15. The method according to claim 10, wherein the stroke estimation procedure further comprises discarding any ball candidate whose tentative ball path has an angle relative to a vertical line exceeding a threshold angle.

16. The method according to claim 10, wherein the method exclusively generates the trace representing the path of the moving ball in the composite image sequence in respect of a ball candidate complying with the ball selection algorithm.

17. The method according to claim 10, wherein the method inhibits the identification of an initial image position for the moving ball after that at least one ball has been tracked throughout a first predefined number of images.

18. The method according to claim 10, wherein, after the three consecutively registered images, the tracking involves estimation of a movement vector predicting a ball position in a particular source image, the movement vector being based on:

a first image position for the ball in a first source image preceding the particular source image;

a second image position for the ball in a second source image subsequent to the first source image however preceding the particular source image; and

a third image position for the ball in a third source image subsequent to the second source image however preceding the particular source image.

19. The method according to claim 18, wherein the method, after an initial second number of registered images, the second number exceeding three, predicts the ball position in the particular source image on the basis of a linear relationship between positions for the ball in two images registered immediately prior to the particular image.

20. The method according to claim 10, wherein the method continues the tracking until no ball candidate can be found within a threshold distance from the predicted ball position.

21. A computer program directly loaded into a internal, non-transitory computer-readable memory of a computer, comprising software comprising computer-readable instructions for:

registering a series of source images of the scene;

tracking a path of the moving ball through at least a sub-set of the images in said series, the tracking involving a differencing between consecutively registered images wherein image areas are identified that have been updated in a later registered image in relation to an earlier registered image; and

generating, based on said source images a composite image sequence wherein synthetic image elements are combined with the images in said series, such that a trace representing the path of the moving ball is shown in the form of added image data successively overlaid the image contents of the images in said series, wherein the method comprises automatically identifying an initial image position for the moving ball by;

producing, for each image in said series, a difference image between the image and an image subsequent thereto, wherein in the difference image all image elements representing a contents alteration below a threshold level are allocated a first value, and all image elements representing a contents alteration above or equal to the threshold level are allocated a second value,

identifying a set of ball candidates in which each ball candidate is represented by a group of neighboring image elements which all contain the second value and said group fulfills a ball size criterion, and

selecting the initial image position from the set of ball candidates based on a ball selection algorithm,

wherein the ball selection algorithm implies a stroke estimation procedure wherein the ball candidates in the set of ball candidates are analyzed in three consecutively registered images, the stroke estimation procedure comprising;

defining a respective path candidate line extending straight between each ball candidate in a first registered image and each ball candidate in a third registered image,

discarding each ball candidate for which, in a second image registered between the first and third registered images, no ball candidate is found within a threshold range from the path candidate line, and

defining a tentative ball path for all other path candidate lines.

22. A non-transitory computer readable medium, having a program recorded thereon comprising computer-readable instructions for:

registering a series of source images of the scene;

tracking a path of the moving ball through at least a sub-set of the images in said series, the tracking involving a differencing between consecutively registered images wherein image areas are identified that have been updated in a later registered image in relation to an earlier registered image; and

generating, based on said source images a composite image sequence wherein synthetic image elements are combined with the images in said series, such that a trace representing the path of the moving ball is shown in the form of added image data successively overlaid the image contents of the images in said series, wherein the method comprises automatically identifying an initial image position for the moving ball by;

producing, for each image in said series, a difference image between the image and an image subsequent thereto, wherein in the difference image all image elements representing a contents alteration below a threshold level are allocated a first value, and all image elements representing a contents alteration above or equal to the threshold level are allocated a second value,

identifying a set of ball candidates in which each ball candidate is represented by a group of neighboring image elements which all contain the second value and said group fulfills a ball size criterion, and

selecting the initial image position from the set of ball candidates based on a ball selection algorithm,

wherein the ball selection algorithm implies a stroke estimation procedure wherein the ball candidates in the set of ball candidates are analyzed in three consecutively registered images, the stroke estimation procedure comprising;

defining a respective path candidate line extending straight between each ball candidate in a first registered image and each ball candidate in a third registered image,

discarding each ball candidate for which, in a second image registered between the first and third registered images, no ball candidate is found within a threshold range from the path candidate line, and

defining a tentative ball path for all other path candidate lines.