Method for estimating a 3D trajectory of a projectile from 2D camera images

• US 10,380,409 B2
• Filed: 11/16/2017
• Issued: 08/13/2019
• Est. Priority Date: 11/16/2017
• Status: Active Grant

• Current Assignee: Newlight Capital, LLC
• Original Assignee: Blast Motion Incorporated
• Inventors: Thornbrue, James, Cherveny, Patrick, Haas, Juergen
• Primary Examiner(s): Mikeska, Neil R
• Application Number:
  US15/815,696
• Publication Number:
  US 20190147219A1
• Time in Patent Office: 635 Days
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30224 Ball; Puck
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A method that estimates the 3D trajectory of a projectile, such as a golf ball or baseball, by analyzing a sequence of images from a single-lens, 2D camera. Image analysis may be used to locate the projectile in the camera images, using motion detection and filters for the projectile'"'"'s expected size and shape. A physics model may be used to calculate the projectile'"'"'s trajectory as a function of its initial position and velocity (for example, just after impact with a golf club or bat). A camera projection transform may map this trajectory into predicted pixel locations, which may be compared to the observed projectile locations in the camera images. The projectile'"'"'s trajectory may be estimated by finding initial conditions that minimize differences between observed and predicted pixel locations, using a nonlinear least squares solver for example. The method may be extended to multiple cameras.

Claims

1. A method for estimating a 3D trajectory of a projectile from 2D camera images, comprising:

obtaining a sequence of 2D camera images from a camera, wherein one or more 2D camera images of said sequence of 2D camera images comprise an image of a projectile;

identifying a pixel location of said projectile in each 2D camera image of said one or more 2D camera images;

calculating a camera projection transform that maps points in a world reference frame into pixel positions in said sequence of 2D camera images;

calculating a model that maps projectile initial conditions into a modeled pixel position for said projectile in each 2D camera image of said one or more 2D camera images, wherein said projectile initial conditions comprise

an initial position vector in said world reference frame for said projectile; and

,

an initial velocity vector in said world reference frame for said projectile; and

,

calculating estimated projectile initial conditions as projectile initial conditions that minimize differences between said modeled pixel position for said projectile and said pixel location of said projectile across said one or more 2D camera images.

2. The method of claim 1, wherein said projectile comprises one or more of a golf ball, a baseball, a softball, a soccer ball, a football, a hockey puck, a tennis ball, a table tennis ball, a squash ball, a racket ball, a shuttlecock, a handball, a lacrosse ball, a field hockey ball, a volleyball, a kickball, a horseshoe, and a lawn dart.

3. The method of claim 1, further comprising:

calculating an estimated projectile trajectory in said world reference frame based on said estimated projectile initial conditions and on a projectile physics model.

4. The method of claim 3, wherein said projectile physics model comprises a model of forces on said projectile, wherein said forces comprise

gravity, and

drag.

5. The method of claim 4, wherein

said forces further comprise a Magnus effect force;

and,

said projectile physics model further comprises an estimate or calculation of an initial spin of said projectile.

6. The method of claim 3, further comprising:

calculating an estimated carry distance for said projectile as a horizontal distance between said initial position vector of said estimated projectile initial conditions and a point on said estimated projectile trajectory where said projectile contacts a horizontal plane.

7. The method of claim 1, wherein said calculating said camera projection transform comprises:

defining a camera reference frame fixed to said camera;

calculating a transformation between said world reference frame and said camera reference frame; and

,

calculating a transformation between said camera reference frame and said pixel positions based on one or more camera parameters.

8. The method of claim 7, further comprising:

measuring a size in pixels of a camera image of an object of a known size at a known distance to determine one or more of said one or more camera parameters.

9. The method of claim 7, wherein said calculating a transformation between said world reference frame and said camera reference frame comprises:

measuring an orientation of said camera in said world reference frame using one or more sensors coupled to said camera; and

,

calculating said transformation between said world reference frame and said camera reference frame from said orientation of said camera in said world reference frame.

10. The method of claim 9, wherein

said one or more sensors coupled to said camera comprise a three-axis accelerometer;

said measuring said orientation of said camera in said world reference frame comprises obtaining a measured gravity vector using said three-axis accelerometer; and

,

said transformation between said world reference frame and said camera reference frame comprises a rotation that rotates a vertical gravity vector in said world reference frame to said measured gravity vector.

11. The method of claim 1, wherein said identifying a pixel location of said projectile in each 2D camera image comprises:

applying a three-frame difference algorithm to identify one or more moving objects in said one or more 2D camera images, yielding one or more difference frames;

applying a moving average filter to said one or more difference frames, wherein a size of said moving average filter matches a largest expected pixel size of said projectile, yielding one or more filtered images; and

,

identifying isolated peaks in said one or more filtered images as potential pixel locations of said projectile, wherein each isolated peak of said isolated peaks has a value that exceeds a detection threshold and is separated from other potential pixel locations of said potential pixel locations by a distance that exceeds a distance threshold.

12. The method of claim 11, wherein

said projectile is a ball;

and,

said identifying a pixel location of said projectile in each 2D camera image further comprises;

filtering said potential pixel locations of said projectile to retain only pixel locations of objects that are substantially round.

13. The method of claim 12, wherein said filtering said potential pixel locations of said projectile to retain only pixel locations of objects that are substantially round comprises:

for each potential pixel location of said potential pixel locations of said projectile,

identifying candidate pixels within a region surrounding said potential pixel location that have a pixel value exceeding a value threshold;

determining a centroid of said candidate pixels;

calculating an average distance between said candidate pixels and said centroid;

calculating a disc having a center at said centroid and a radius of 1.5 times said average distance;

determining the fraction of pixels within said disc that are candidate pixels; and

,

retaining said potential pixel location when said fraction exceeds a density threshold.

14. The method of claim 1, further comprising:

fixing one coordinate of said initial position vector in said world reference frame based on a measured or assumed distance between said camera and said initial position vector; and

,

obtaining or calculating an initial time for motion of said projectile, wherein

said initial position vector corresponds to a position of said projectile at said initial time, and

said initial velocity vector corresponds to a velocity of said projectile at said initial time.

15. The method of claim 14, wherein said calculating estimated projectile initial conditions comprises:

obtaining a nonlinear least-squares solution that minimizes the sum of squared distances between said modeled pixel position for said projectile and said pixel location of said projectile across said one or more 2D camera images, wherein parameters of said model comprise two unknown coordinates for said initial position vector and three unknown coordinates for said initial velocity vector.

16. The method of claim 15, wherein said calculating said model comprises:

calculating or obtaining a time for each 2D camera image of said one or more 2D camera images;

mapping said time for each 2D camera image into a projectile location in said world reference frame based on

said projectile initial conditions, and

a physics model for an initial period of said 3D trajectory of said projectile;

mapping said projectile location in said world reference frame into said modeled pixel position based on said camera projection transform.

17. The method of claim 16, wherein said physics model for said initial period of said 3D trajectory comprises inertial motion starting at said initial position vector with a uniform velocity equal to said initial velocity vector.

18. The method of claim 16, wherein said physics model for said initial period of said 3D trajectory comprises a model of forces on said projectile, wherein said forces comprise gravity.

19. The method of claim 18, wherein said forces further comprise drag.

20. The method of claim 19, wherein

said forces further comprise a Magnus effect force;

and,

said physics model for said initial period of said 3D trajectory further comprises an estimate or calculation of an initial spin of said projectile.