True space tracking of axisymmetric object flight using image sensor

• US 9,697,617 B2
• Filed: 12/22/2014
• Issued: 07/04/2017
• Est. Priority Date: 04/03/2013
• Status: Active Grant

• Current Assignee: Pillar Vision Incorporated
• Original Assignee: Pillar Vision Incorporated
• Inventors: Carter, John, Marty, Alan W.
• Primary Examiner(s): Moyer, Andrew
• Application Number:
  US14/579,916
• Publication Number:
  US 20150103168A1
• Time in Patent Office: 925 Days
• Field of Search: 382103, 382154
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30224 Ball; Puck
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/51 Housings

Abstract

Methods and apparatus for determining a trajectory of a axisymmetric object in 3-D physical space using a digital camera which records 2-D image data are described. In particular, based upon i) a characteristic length of the axisymmetric object, ii) a physical position of the camera determined from sensors associated with the camera (e.g., accelerometers) and iii) captured 2-D digital images from the camera including a time at which each image is generated relative to one another, a position, a velocity vector and an acceleration vector can be determined in three dimensional physical space for axisymmetric object objects as a function of time. In one embodiment, the method and apparatus can be applied to determine the trajectories of objects in games which utilize axisymmetric object objects, such as basketball, baseball, bowling, golf, soccer, rugby or football.

Claims

1. An electronic device comprising:

a housing;

a wireless network interface;

a camera lens coupled to an exterior of the housing;

a digital image sensor disposed within an interior of the housing and positioned to receive light which passes through the camera lens; and

a processor communicatively coupled to the digital image sensor, the processor configured to

     1) receive a plurality of 2-D digital images from the digital image sensor,

     2) select a portion of the plurality of 2-D digital images wherein the portion is selected based upon at least whether an axisymmetric object in free flight used in a play of a game is identifiable,

     3) determine an orientation of the camera lens in a first reference frame;

     4) based upon the orientation of the camera lens in the first reference frame, transform first pixel data in each of the portion of the plurality of 2-D digital images from the first reference frame associated with a current orientation of the camera lens to second pixel data associated with a second reference frame;

     5) determine a plurality of first pixel lengths across the axisymmetric object in multiple directions of the second pixel data in each of the portion of the plurality of 2-D digital images to account for motion blurring and light blooming effects;

     6) based upon the plurality of first pixel lengths, determine a first characteristic pixel length of the axisymmetric object in each of the portion of the plurality of 2-D digital images;

     7) based upon first characteristic pixel length, determine distances from the camera lens to the axisymmetric object in physical space in each of the portion of the plurality of 2-D digital images;

     8) determine a center of the axisymmetric object in the second pixel data in each of the 2-D digital images;

     9) determine a plurality of second pixel lengths which measure changes in the center of the axisymmetric object between each of the portion of the plurality of 2-D digital images;

     10) based upon the plurality of second pixel lengths and the distances from the camera lens, determine a 3-D trajectory of the axisymmetric object as a function of time in physical space,

     11) determine one or more parameters of the 3-D trajectory, and

     12) output to a display information indicative of the one or more parameters of the 3-D trajectory.

2. The electronic device of claim 1, wherein the axisymmetric object is a spherically shaped object.

3. The electronic device of claim 1, wherein the processor is further configured to determine a second characteristic pixel length of the axisymmetric object perpendicular to the first characteristic pixel length in each of the portion of the plurality of 2-D digital images;

select one of the first characteristic pixel length or the second characteristic pixel length and based upon the selected one of the first characteristic pixel length or the second characteristic pixel length, determine the distance to the axisymmetric object.

4. The electronic device of claim 3, wherein the axisymmetric object is a frisbee.

5. The electronic device of claim 1, wherein the processor is further configured to identify one or more secondary objects associated with a play of the game captured in the plurality of the 2-D digital images.

6. The electronic device of claim 1, wherein the processor is further configured to output a first image of the plurality of 2-D images to the display, which includes a secondary object, and receive input designating a portion of first image where the secondary object appears.

7. The electronic device of claim 6, wherein the processor is further configured to search in the designated portion of the first image to identify the secondary object.

8. The electronic device of claim 6, wherein, after the secondary object is identified, the processor is further configured to determine a second distance to secondary object from the camera lens.

9. The electronic device of claim 1, wherein the processor is further configured to receive a pixel resolution of the 2-D digital images.

10. The electronic device of claim 9, wherein the distances are determined based upon the received pixel resolution.

11. The electronic device of claim 9, wherein the processor is further configured to receive or retrieve a characteristic length of the axisymmetric object measured in units associated with 3-D physical space.

12. The electronic device of claim 1, wherein the axisymmetric object in one of the 2-D digital images is identified while partially obscured.

13. The electronic device of claim 1, wherein the axisymmetric object in one of the 2-D digital images is identified while being held in a person'"'"'s hands.

14. The electronic device of claim 1, wherein the axisymmetric object in one of the 2-D digital images is identified while rolling along a surface or in contact with a surface.

15. The electronic device of claim 1, wherein the function of the time is based upon a time associated with each of the 2-D digital images.

16. The electronic device of claim 1, wherein the processor is further configured to receive a recording speed associated with the digital image sensor and to determine the function of the time based upon the recording speed.

17. The electronic device of claim 1, wherein the processor is further configured to determine a velocity vector and magnitude of the velocity vector as the function of time of the axisymmetric object along the 3-D trajectory.

18. The electronic device of claim 1, wherein processor is further configured to render one or more of a graphical representation of one or more of a playing field, playing field markings or a secondary object associated with the game in the new 2-D image.

19. The electronic device of claim 1, wherein only a first portion of a playing field, a first portion of a playing field markings or a first portion of a secondary object is captured in the one of the plurality of 2-D images and wherein the processor is further configured to render a second portion of the playing field, a second portion the playing field markings and/or a second portion of a secondary object in the new 2-D image such that the first portion and the second portion form a complete playing field, a complete playing field markings or a complete secondary object in the new 2-D image.

20. The electronic device of claim 1, wherein the processor is further configured to generate a simulation of the 3-D trajectory of the axisymmetric object using data determined from the 2-D digital images as initial conditions for the simulation.

21. The electronic device of claim 20, wherein the simulation is used to extrapolate the 3-D trajectory of the axisymmetric object beyond what is captured in the plurality of 2-D images.

22. The electronic device of claim 1, wherein information derived from the 3-D trajectory is used as input for a video game.

23. The electronic device of claim 1, wherein, based upon the 3-D trajectory, the processor is further configured to output training recommendations for improving performance of a participant in the game played using the axisymmetric object.

24. The electronic device of claim 1, further comprising a bracket coupled to the housing configured to mount the housing to a wall.

25. The electronic device of claim 1, wherein the electronic device is one of a tablet computer or a phone.

28. The electronic device of claim 1, wherein the processor is configured to output to the display the one or more parameters of the 3-D trajectory.

29. The electronic device of claim 1, wherein the processor is configured to output to the display an image showing a path of the 3-D trajectory based on the one or more parameters.

26. An electronic device comprising:

a housing;

a wireless network interface;

a camera lens coupled to an exterior of the housing;

a digital image sensor disposed within an interior of the housing and positioned to receive light which passes through the camera lens; and

a processor communicatively coupled to the digital image sensor, the processor configured to

     1) receive a plurality of 2-D digital images from the digital image sensor,

     2) select a portion of the plurality of 2-D digital images wherein the portion is selected based upon at least whether a football in free flight is identifiable,

     3) determine an orientation of the camera lens in a first reference frame;

     4) based upon the orientation of the camera lens in the first reference frame, transform first pixel data in each of the portion of the plurality of 2-D digital images from the first reference frame associated with a current orientation of the camera lens to second pixel data associated with a second reference frame;

     5) determine a plurality of first pixel lengths across the football in multiple directions of the second pixel data in each of the portion of the plurality of 2-D digital images to account for motion blurring and light blooming effects;

     6) based upon the plurality of first pixel lengths, determine a first characteristic pixel length of the football in each of the portion of the plurality of 2-D digital images;

     7) based upon first characteristic pixel length, determine distances from the camera lens to the football in physical space in each of the portion of the plurality of 2-D digital images;

     8) determine a position associated with the football in the second pixel data in each of the 2-D digital images;

     9) determine a plurality of second pixel lengths which measure changes in the position of the football between each of the portion of the plurality of 2-D digital images;

     10) based upon the plurality of second pixel lengths and the distances from the camera lens, determine a 3-D trajectory of the football as a function of time in physical space,

     11) determine one or more parameters of the 3-D trajectory, and

     12) output to a display information indicative of the one or more parameters of the 3-D trajectory.

27. An electronic device comprising:

a housing;

a wireless network interface;

a camera lens coupled to an exterior of the housing;

a digital image sensor disposed within an interior of the housing and positioned to receive light which passes through the camera lens; and

a processor communicatively coupled to the digital image sensor, the processor configured to

     1) receive a plurality of 2-D digital images from the digital image sensor,

     2) select a portion of the plurality of 2-D digital images wherein the portion is selected based upon at least whether a basketball in free flight is identifiable,

     3) determine an orientation of the camera lens in a first reference frame;

     4) based upon the orientation of the camera lens in the first reference frame, transform first pixel data in each of the portion of the plurality of 2-D digital images from the first reference frame associated with a current orientation of the camera lens to second pixel data associated with a second reference frame;

     5) determine a plurality of first pixel lengths across the basketball in multiple directions of the second pixel data in each of the portion of the plurality of 2-D digital images to account for motion blurring and light blooming effects;

     6) based upon the plurality of first pixel lengths, determine a first characteristic pixel length of the basketball in each of the portion of the plurality of 2-D digital images;

     7) based upon first characteristic pixel length, determine distances from the camera lens to the basketball in physical space in each of the portion of the plurality of 2-D digital images;

     8) determine a position associated with the basketball in the second pixel data in each of the 2-D digital images;

     9) determine a plurality of second pixel lengths which measure changes in the position of the basketball between each of the portion of the plurality of 2-D digital images;

     10) based upon the plurality of second pixel lengths and the distances from the camera lens, determine a 3-D trajectory of the basketball as a function of time in physical space,

     11) determine one or more parameters of the 3-D trajectory, and

     12) output to a display information indicative of the one or more parameters of the 3-D trajectory.