Systems and methods for monitoring objects in athletic playing spaces

• US 9,734,405 B2
• Filed: 10/05/2015
• Issued: 08/15/2017
• Est. Priority Date: 10/05/2015
• Status: Active Grant

• Current Assignee: Pillar Vision Incorporated
• Original Assignee: Pillar Vision Incorporated
• Inventors: Lucey, Simon, Marty, Alan W., Carter, John
• Primary Examiner(s): FUJITA, KATRINA R
• Application Number:
  US14/874,555
• Publication Number:
  US 20170098125A1
• Time in Patent Office: 680 Days
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30224 Ball; Puck
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/42 of sport video content

Abstract

A system uses a camera, a depth sensor, and a processing module to track a moving object in an athletic playing space over time. The processing module receives image frames from the camera and depth data from the depth sensor. The processing module analyzes the depth data to identify which of the pixels are in a plane of a sufficiently large size to indicate that the identified pixels correspond to a playing surface of the athletic playing space. The processing module then determines that gravity is in a direction normal to the identified plane. By finding the direction of gravity within the images, the algorithm for calculating performance parameters based on the trajectory of a moving object within the athletic playing space can be greatly simplified, thereby decreasing the processing burdens associated with calculating the performance parameters and providing more accurate trajectory estimations with fewer measured trajectory points.

Claims

1. A system for monitoring objects in athletic playing spaces, comprising:

a camera positioned to capture images of an object in an athletic playing space, wherein the camera is configured to provide image data defining frames of the captured images, and wherein the image data is in a format relative to a first coordinate system;

a depth sensor positioned to sense depths for objects in the athletic playing space, wherein for one of the frames the depth sensor is configured to provide a depth map having a plurality of depth pixels corresponding with a plurality of Image pixels of the one frame, and wherein each of the depth pixels indicates a sensed depth for at least one corresponding image pixel of the one frame; and

at least one processor configured to identify a playing surface plane defined by the depth pixels and to determine a direction of gravity in the one frame based on the identified playing surface plane, the at least one processor further configured to calculate at least one performance parameter based on the image data and the determined direction of gravity, wherein the playing surface plane corresponds to a playing surface of the athletic playing space in which athletes participate during a sporting event.

2. The system of claim 1, wherein the camera and depth sensor are mounted on an arm for supporting a basketball goal.

3. The system of claim 1, wherein the camera and the depth sensor are coupled to an aerial vehicle.

4. The system of claim 1, wherein the playing surface is a floor of a basketball court.

5. The system of claim 1, further comprising an accelerometer, wherein the at least one processor is configured to determine the direction of gravity based on the accelerometer and the identified playing surface plane.

6. The system of claim 1, wherein the at least one processor Is configured to determine an altitude of the athletic playing space and to determine a magnitude of gravitational pull based on the altitude, and wherein the at least one processor is configured to calculate the at least one performance parameter based on the determined magnitude.

7. The system of claim 1, further comprising an output device communicatively coupled to the at least one processor, the output device configured to provide an output based on the performance parameter.

8. The system of claim 7, wherein the output device is configured to communicate with the at least one processor wirelessly.

9. The system of claim 7, wherein the output device is configured to display the performance parameter.

10. The system of claim 1, wherein the at least one processor is configured to convert the image data into a format relative to a gravity-based coordinate system based on the determined direction of gravity.

11. The system of claim 10, wherein the at least one processor is configured to identify an image of a hoop in the depth map and to orient the gravity-based coordinate system based on the identified hoop.

12. The system of claim 11, wherein the at least one processor is configured to determine a distance from the depth sensor to the hoop based on a size of the image of the hoop in the depth map.

13. The system of claim 1, wherein the at least one processor is configured to track a trajectory of a basketball during a shot of the basketball toward a basketball goal.

14. The system of claim 13, wherein the at least one processor is configured to determine locations of the basketball in three-dimensional space at a plurality of points along the trajectory based on the image data and depth data from the depth sensor, and wherein the at least one processor Is configured to define a trajectory curve for the basketball based on the determined locations and the determined direction of gravity.

15. The system of claim 1, wherein the at least one processor is configured to account for a slope of the playing surface for determining the direction of gravity such that the determined direction of gravity is at an oblique angle relative to the playing surface plane.

16. The system of claim 15, wherein the at least one processor is configured to store surface data indicative of a surface topology of the playing surface, and wherein the at least one processor is configured to determine the direction of gravity based on the surface data.

17. The system of claim 1, wherein the at least one processor is configured to determine the direction of gravity based on an angle of gravity relative to the identified playing surface plane.

18. The system of claim 17, wherein the at least one processor is configured to determine the angle based on a slope of the identified playing surface plane.

19. The system of claim 18, wherein the at least one processor is configured to store surface data indicative of the slope, and wherein the at least one processor is configured to determine the angle based on the surface data.

20. A method for monitoring objects in athletic playing spaces, comprising:

capturing images of an object in an athletic playing space with a camera, thereby defining frames of image data, wherein the image data is in a format relative to a first coordinate system;

sensing depths for objects in the athletic playing space with a depth sensor,

providing from the depth sensor a depth map for at least one of the frames, wherein the depth map has a plurality of depth pixels corresponding with a plurality of image pixels of the one frame, and wherein each of the depth pixels indicates a sensed depth for at least one corresponding image pixel of the one frame;

identifying a playing surface plane defined by the depth pixels with at least one processor, wherein the playing surface plane corresponds to a playing surface on which athletes participate in a sporting event occurring in the athletic playing space;

analyzing the playing surface plane defined by the depth pixels with the at least one processor;

determining a direction of gravity for the one frame with the at least one processor based on the analyzing; and

calculating, with the at least one processor, at least one performance parameter based on the image data and the determined direction of gravity.

21. The method of claim 20, wherein the playing surface is a floor of a basketball court.

22. The method of claim 20, further comprising displaying an output with an output device based on the performance parameter.

23. The method of claim 20, wherein the determining the direction of gravity is based on an accelerometer and the identified playing surface plane.

24. The method of claim 20, further comprising;

determining an altitude of the athletic playing space; and

determining a magnitude of gravitational pull based on the determined altitude, wherein the calculating is based on the determined magnitude.

25. The method of claim 20, further comprising converting, with the at least one processor, the image data into a format relative to a gravity-based coordinate system based on the determined direction of gravity.

26. The method of claim 25, further comprising:

identifying, with the at least one processor, an image of a hoop in the depth map; and

orienting the gravity-based coordinate system based on the identified hoop.

27. The method of claim 26, further comprising determining a distance from the depth sensor to the hoop based on a size of the image of the hoop In the depth map.

28. The method of claim 20, further comprising tracking, with the at least one processor, a trajectory of a basketball during a shot of the basketball toward a basketball goal.

29. The method of claim 28, further comprising:

determining, with the at least one processor, locations of the basketball in three-dimensional space at a plurality of points along the trajectory based on the image data and depth data from the depth sensor; and

defining, with the at least one processor, a trajectory curve for the basketball based on the determined locations and the determine direction of gravity.

30. The method of claim 20, further comprising accounting for a slope of the playing surface with the at least one processor such that the determined direction of gravity is at an oblique angle relative to the playing surface plane.

31. The method of claim 20, further comprising storing in memory surface data indicative of a surface topology of the playing surface, wherein the determining the direction of gravity is based on the surface data.

32. The method of claim 20, wherein the determining comprises determining that the direction of gravity is at a predefined angle relative to the playing surface plane defined by the depth pixels.

33. A system for monitoring objects in athletic playing spaces, comprising:

a camera positioned to capture images of an object in an athletic playing space, wherein the camera is configured to provide image data defining frames of the captured images, and wherein the image data is in a format relative to a first coordinate system;

a depth sensor positioned to sense depths for objects in the athletic playing space, wherein for one of the frames the depth sensor is configured to provide a depth map having a plurality of depth pixels corresponding with a plurality of image pixels of the one frame, and wherein each of the depth pixels indicates a sensed depth for at least one corresponding image pixel of the one frame; and

at least one processor configured to identify a playing surface plane defined by the depth pixels and to analyze the playing surface plane defined by the depth pixels for determining a direction of gravity for the one frame, the at least one processor further configured to calculate at least one performance parameter based on the image data and the determined direction of gravity, wherein the playing surface plane corresponds to a playing surface of the athletic playing space in which athletes participate during a sporting event.

34. The system of claim 33, wherein the at least one processor is configured to determine that the direction of gravity is at a predefined angle relative to the playing surface plane defined by the depth pixels.

35. The system of claim 33, wherein the playing surface is a floor of a basketball court.

36. The system of claim 33, further comprising an output device communicatively coupled to the at least one processor, the output device configured to provide an output based on the performance parameter.

37. The system of claim 33, wherein the at least one processor is configured to convert the image data into a format relative to a gravity-based coordinate system based on the determined direction of gravity.

38. The system of claim 37, wherein the at least one processor is configured to identify an image of a hoop in the depth map and to orient the gravity-based coordinate system based on the identified hoop.

39. The system of claim 38, wherein the at least one processor is configured to determine a distance from the depth sensor to the hoop based on a size of the image of the hoop in the depth map.

40. The system of claim 33, wherein the at least one processor is configured to track a trajectory of a basketball during a shot of the basketball toward a basketball goal.

41. The system of claim 40, wherein the at least one processor is configured to determine locations of the basketball in three-dimensional space at a plurality of points along the trajectory based on the image data and depth data from the depth sensor, and wherein the at least one processor is configured to define a trajectory curve for the basketball based on the determined locations and the determined direction of gravity.