TRACKING OF HANDHELD SPORTING IMPLEMENTS USING COMPUTER VISION

• US 20170270354A1
• Filed: 03/16/2016
• Published: 09/21/2017
• Est. Priority Date: 03/16/2016
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportsvision Technology Corporation
• Inventors: Painter, James G.
• Granted Patent: US 9,846,805 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 40/25 Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2625 for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect sequence generated by event triggered capturing H04N7/188

Abstract

A path and/or orientation of at least a portion of a handheld sporting implement swung by an athlete is tracked using two or more cameras. At least two sets of video images of the handheld sporting implement being swung are obtained using at least two different cameras having different positions. Motion regions within video images are identified, and candidate locations in 2D space of an identifiable portion (e.g., a head) of the handheld sporting implement is/are identified within the motion region(s). Based thereon, a probable location in 3D space of the identifiable portion is identified, for each of a plurality of instants during which the handheld sporting implement was swung. A piecewise 3D trajectory of at least the identifiable portion (e.g., the head) of the sporting implement is approximated from the probable locations in 3D space of the head for multiple instants during which the sporting implement was swung.

Claims

1. A method for tracking at least a portion of a handheld sporting implement that is swung by an athlete, the handheld sporting implement having a shaft extending between a head and a knob, the method comprising:

(a) receiving two or more different sets of video images of a handheld sporting implement being swung by an athlete, wherein at least two of the different sets of video images are captured using different cameras having different positions;

(b) identifying one or more motion regions within each of a plurality of the video images in each of at least two of the different sets of video images captured using different cameras having different positions;

(c) identifying one or more candidate locations in two-dimensional (2D) space of an identifiable portion of the handheld sporting implement within the identified motion region(s) of the video image, for at least a subset of the video images included in at least two of the different sets of video images;

(d) identifying, based on the candidate locations in 2D space of the head of the handheld sporting implement, a probable location in three-dimensional (3D) space of the head of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete; and

(e) approximating a piecewise 3D trajectory of at least the identifiable portion of the handheld sporting implement from the probable locations in 3D space of the identifiable portion of the handheld sporting implement identified for the plurality of instants during which the handheld sporting implement was swung by the athlete.

2. The method of claim 1, wherein the handheld sporting implement comprises one of a bat, a club or a stick, and wherein the identifiable portion of the handheld sporting implement comprises the head of the handheld sporting implement.

3. The method of claim 1, wherein the receiving at step (a) includes two or more of the following:

(a.1) receiving a first set of video images, of the athlete swinging the handheld sporting implement, captured using a first camera having a first position;

(a.2) receiving a second set of video images, of the athlete swinging the handheld sporting implement, captured using a second camera having a second position; and

(a.3) receiving a third sets of video images, of the athlete swinging the handheld sporting implement, captured using a third camera having a third position;

wherein the first, second and third positions differ from one another.

4. The method of claim 3, wherein:

the sporting implement is a baseball bat;

the first position of the first camera is in foul territory between home plate and first base;

the second position of the second camera is in foul territory behind home plate; and

the third position of the third camera is foul territory between home plate and third base.

5. The method of claim 1, wherein at step (b) the one or more motion regions are identified within a said video image by comparing the said video image to at least a preceding or following one of the video images in a sequence of the video images.

6. The method of claim 1, wherein at step (c) the identifying one or more candidate locations in 2D space of the identifiable portion of the handheld sporting implement within the identified motion region(s) of the video image, for at least the subset of the video images of each of at least two of the different sets of video images captured using different cameras having different positions, comprises:

(c.1) identifying one or more candidate shafts of the handheld sporting implement within the identified motion region(s) of the video images;

(c.2) identifying an approximate center of elliptical arc motion of the handheld sporting implement; and

(c.3) identifying, based on the identified approximate center of elliptical arc motion of the handheld sporting implement and on the identified one or more candidate shafts, the one or more candidate locations in 2D space of the identifiable portion of the handheld sporting implement within the identified motion region(s) of the video images.

7. The method of claim 6, wherein the identifying one or more candidate shafts at (c.1) includes:

(c.1.i) outlining the regions of motion within each of a plurality of the video images in each of the sets of video images; and

(c.1.ii) identifying straight lines within the outlined regions of motion.

8. The method of claim 6, wherein the identifiable portion of the handheld sporting implement comprises the head of the handheld sporting implement, and wherein the identifying one or more candidate locations in 2D space of the head of the handheld sporting implement within the identified motion region(s) of the video images at (c.3) includes:

(c.3.i) identifying first and second ends of each of the one or more candidate shafts;

(c.3.ii) for each of the candidate shafts, identifying and filtering out the one of the first and second ends of the candidate shaft that is closer to the approximate center of elliptical arc motion of the handheld sporting implement, whereby the non-filtered out ones of the first and second ends remain as candidate locations in 2D space of the head of the handheld sporting implement within the identified motion region(s) of the video images; and

(c.3.iii) applying one or more heuristics to identify and filter out one or more of the candidate locations, remaining after the filtering at (c.3.ii), that is/are false positives.

9. The method of claim 1, wherein at step (d) the identifying the probable location in 3D space of the identifiable portion of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete, comprises, for each said instant:

(d.1) identifying different combinations of candidate locations in 2D space of the identifiable portion of the handheld sporting implement that were based on images captured using at least two different ones of the cameras;

(d.2) for each of at least two combinations included in the identified different combinations of candidate locations in 2D space of the identifiable portion of the handheld sporting implement,

determining, for each of at least two of the candidate locations of the identifiable portion of the handheld sporting implement in 2D space associated with the combination, a line of position in 3D space from the one of the cameras, that captured the image used to produce the candidate location, to the candidate location of the identifiable portion in 2D space, so that there are at least two lines of position associated with the combination; and

(d.3) determining, as the probable location in 3D space of the head of the handheld sporting implement for the said instance, a location in 3D space where distances between the location and at least two of the two or more lines of position are at a minima.

10. The method of claim 9, wherein the lines of position in 3D space are determined using a transformation matrix associated with the one of the cameras that captured the image that was used to produce the candidate location of the identifiable portion of the handheld sporting implement in 2D space.

11. The method of claim 1, wherein the approximating the piecewise 3D trajectory of the identifiable portion of the handheld sporting implement at step (e) is performed using a B-spline algorithm.

12. The method of claim 1, wherein the identifiable portion of the handheld sporting implement comprises the head of the handheld sporting implement, and further comprising:

identifying one or more candidate locations in 2D space of the knob of the handheld sporting implement within the identified motion region(s) of the video image, for at least a subset of the video images included in each of the different sets of video images;

identifying, from the candidate locations in 2D space of the knob of the handheld sporting implement, a probable location in 3D space of the knob of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete;

approximating a piecewise 3D trajectory of the knob of the handheld sporting implement from the probable locations in 3D space of the head of the handheld sporting implement identified for each of the plurality of instants during which the handheld sporting implement was swung by the athlete; and

for each of the plurality of instants extending a line between the head and the knob to represent a probable location in 3D space of the shaft of the handheld sporting implement for the said instant.

13. The method of claim 12, further comprising displaying a video of the athlete swinging the handheld sporting implement which is enhanced to visually show a path and/or orientation of the shaft of the handheld sporting implement.

14. The method of claim 1, further comprising displaying a video of the athlete swinging the handheld sporting implement which is enhanced to visually show a path and/or orientation of the identifiable portion of the handheld sporting implement.

15. A system for tracking at least a portion of a handheld sporting implement that is swung by an athlete, the handheld sporting implement having a shaft extending between a head and a knob, the system comprising:

at least first and second cameras that capture video images of a handheld sporting implement being swung by an athlete, wherein the first and second cameras have different positions;

at least one processor that

receives two or more different sets of video images of the handheld sporting implement being swung by the athlete, wherein one of the sets of video images is captured using the first camera and another one of the sets of video images is captured using the second camera;

identifies one or more motion regions within each of a plurality of the video images in each of at least two of the different sets of video images;

identifies one or more candidate locations in two-dimensional (2D) space of an identifiable portion of the handheld sporting implement within the identified motion region(s) of the video image, for at least a subset of the video images included in each of at least two of the different sets of video images;

identifies, based on the candidate locations in 2D space of the identifiable portion of the handheld sporting implement, a probable location in three-dimensional (3D) space of the identifiable portion of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete; and

approximates a piecewise 3D trajectory of at least the identifiable portion of the handheld sporting implement from the probable locations in 3D space of the identifiable portion of the handheld sporting implement identified for the plurality of instants during which the handheld sporting implement was swung by the athlete.

16. The system of claim 15, wherein:

the handheld sporting implement is a baseball bat for use in a baseball game played on a baseball field including a home plate, a first base, a second base and a third base;

the at least first and second cameras also includes a third camera;

the first camera is located in foul territory between the home plate and the first base and generally faces a base path between the home plate and the first base;

the second camera is located in foul territory behind the home plate and generally faces the home plate; and

the third camera is located in foul territory between the home plate and the third base and generally faces a base path between the home plate and the third base.

17. The system of claim 16, wherein the at least one processor:

receives a first set of video images of an athlete swinging a baseball bat that are captured by the first camera;

receives a second set of video images of the athlete swinging the baseball bat that are captured by the second camera; and

receives a third set of video images of the athlete swinging the baseball bat that are captured by the third camera.

18. The system of claim 15, wherein the at least one processor identifies the one or more motion regions within each of the plurality of the video images in each of the different sets of video images by comparing a said video image to at least one of a preceding or following one of the video images in a sequence of the video images captured using a same one of the cameras.

19. The system of claim 15, wherein the at least one processor identifies a respective approximate center of elliptical arc motion of the handheld sporting implement for each of at least two of the sets of video images captured using a different one of the cameras, and for each of the sets of video images uses the respective approximate center of elliptical arc motion of the handheld sporting implement to identify the one or more candidate locations in 2D space of the identifiable portion of the handheld sporting implement within the identified motion region(s) of the video images.

20. The system of claim 15, wherein the at least one processor identifies the probable location in 3D space of the identifiable portion of the handheld sporting implement, for each of the plurality of instants during which the handheld sporting implement was swung by the athlete, by:

identifying different combinations of candidate locations in 2D space of the identifiable location of the handheld sporting implement that were based on images captured using at least two different ones of the cameras for a said instant;

determining two or more lines of position for each of at least two of the combinations; and

determining, as the probable location in 3D space of the identifiable portion of the handheld sporting implement for the said instant, a location in 3D space where distances between the location and the two or more lines of position are at a minima.

21. The system of claim 15, wherein the at least one processor uses transformation matrices to determine the lines of position.

22. The system of claim 15, the at least one processor uses a curve fitting algorithm to approximate the piecewise 3D trajectory of the identifiable portion of the handheld sporting implement from the probable locations in 3D space of the identifiable portion of the handheld sporting implement identified for the plurality of instants during which the handheld sporting implement was swung by the athlete.

23. The system of claim 15, wherein the identifiable portion of the handheld sporting implement comprises the head of the handheld sporting implement, and wherein the at least one processor also:

identifies one or more candidate locations in 2D space of the knob of the handheld sporting implement within the identified motion region(s) of the video image, for at least a subset of the video images included in each of at least two of the different sets of video images;

identifies, from the candidate locations in 2D space of the knob of the handheld sporting implement, a probable location in 3D space of the knob of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete;

approximates a piecewise 3D trajectory of the knob of the handheld sporting implement from the probable locations in 3D space of the head of the handheld sporting implement identified for each of the plurality of instants during which the handheld sporting implement was swung by the athlete; and

extends a line between the head and the knob to represent a probable location in 3D space of the shaft of the handheld sporting implement, for each of the plurality of instants.

24. At least one processor readable storage device having processor readable code embodied thereon for programming at least one processor to perform a method for tracking at least a portion of a handheld sporting implement that is swung by an athlete, the method comprising:

receiving two or more different sets of video images of a handheld sporting implement being swung by an athlete, wherein at least two of the different sets of video images is captured using different cameras having different positions;

identifying one or more motion regions within each of a plurality of the video images in each of at least two of the different sets of video images;

identifying one or more candidate locations in two-dimensional (2D) space of a head of the handheld sporting implement within the identified motion region(s) of the video image, for at least a subset of the video images included in each of at least two of the different sets of video images;

identifying, based on the candidate locations in 2D space of the head of the handheld sporting implement, a probable location in three-dimensional (3D) space of the head of the handheld sporting implement, for each of a plurality of instants during which the handheld sporting implement was swung by the athlete; and

approximating a piecewise 3D trajectory of at least the head of the handheld sporting implement from the probable locations in 3D space of the head of the handheld sporting implement identified for the plurality of instants during which the handheld sporting implement was swung by the athlete.

25. A method, comprising:

receiving, from each of a plurality of different cameras, a separate set of two-dimensional (2D) video images of a handheld sporting implement being swung by an athlete, wherein at least two of the different cameras have different positions;

for each of a plurality of video images, within each of the different sets of video images

identifying one or more candidate locations in 2D space of a head of the handheld sporting implement and one or more candidate locations in 2D space of a knob of the handheld sporting implement;

identifying, based on the candidate locations in 2D space of the head of the handheld sporting implement and the candidate locations in 2D space of the knob of the handheld sporting implement, a probable location in three-dimensional (3D) space of the head of the handheld sporting implement and a probable location in 3D space of the knob of the handheld sporting implement;

extending a line between the probable locations in 3D space of the head and the knob of the handheld sporting implement, wherein the line is representative of a shaft of the handheld sporting implement; and

approximating swing trajectory of the handheld sporting implement from the probable locations in 3D space of the head and the knob of the handheld sporting implement and the lines extending therebetween.