METHODS AND SYSTEMS FOR GENERATING SPORTS ANALYTICS WITH A MOBILE DEVICE

• US 20190366153A1
• Filed: 05/28/2019
• Published: 12/05/2019
• Est. Priority Date: 06/01/2018
• Status: Active Grant

• Current Assignee: NEX Team Inc.
• Original Assignee: NEX Team Inc.
• Inventors: Zhang, Qi, Chan, Wing Hung, Lee, Keng Fai, Ng, Wang Fai
• Granted Patent: US 11,638,854 B2
• Time in Patent Office: Days
• CPC Class Codes: A63B 2024/0025 Tracking the path or location of one or more users, e.g. players of a game
• CPC Class Codes: A63B 2024/0028 Tracking the path of an object, e.g. a ball inside a soccer pitch
• CPC Class Codes: A63B 24/0021 Tracking a path or terminating locations
• CPC Class Codes: A63B 24/0062 Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 69/0071 for basketball
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

Methods and systems for real-time generation of ball shot analytics are disclosed. The methods and systems perform steps for ball and posture detection, ball and posture flow generation, shot event identification and classification, and shot analytics generation based on identified shot events and shooter posture flow. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps. Also disclosed are benefits of the new methods, and alternative embodiments of implementation.

Claims

1. A method for generating ball shot analytics using a single mobile computing device, comprising:

receiving an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;

detecting one or more balls and player postures from the input video;

generating one or more ball flows and one or more posture flows by grouping the detected balls and the detected player postures along a time line;

identifying a generated player posture flow as a shooter posture flow, based on the input location of the shooter;

identifying a generated ball flow as related to the shooter posture flow;

determining a ball-from-shooter time by backtracking the related ball flow from a shot attempt;

determining a shot event occurring before the ball-from-shooter time; and

generating one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.

2. The method of claim 1, wherein the detecting of the one or more balls and player postures from the input video comprises using one or more Convolutional Neural Network (CNN) modules.

3. The method of claim 1, wherein the generating of the one or more shot analytics comprises determining a shot type of the shot attempt.

4. The method of claim 1, wherein the ball game is a basketball game.

5. The method of claim 1, wherein the input video is streamed.

6. The method of claim 1, wherein the detecting of balls and player postures is applied on a skip frame basis.

7. The method of claim 1, wherein the detecting of balls and player postures is limited to an image area close to the shooter.

8. The method of claim 1, wherein the generating of ball flows and posture flows comprises applying bipartite matching to detected balls and player postures, respectively, to existing ball flows and posture flows, wherein the bipartite matching of a detected ball and an existing ball flow comprises computing a matching score between the detected ball and the existing ball flow, and wherein the computing of the matching score comprises:

generating a predicted ball comprising a next ball location and a next ball size based on the existing ball flow; and

computing the matching score based on a location difference and a size difference between the predicted ball and the detected ball.

9. The method of claim 1, wherein the shooter posture flow is closest to the shooter location in the shooter identification frame when compared to other posture flows.

10. The method of claim 1,

wherein identifying the related ball flow comprises applying non-max-suppression to all generated ball flows,

wherein the related ball flow has a score against the shooter posture flow,

wherein the score is computed based on shooter movements, a distance to the shooter, and a confidence value, and

wherein the score is above a pre-defined threshold.

11. The method of claim 1, further comprising declaring the shot attempt by determining whether the related ball flow is thrown from the shooter'"'"'s upper body upward.

12. The method of claim 1, wherein the shot event occurs within a pre-defined time period before the ball-from-shooter time.

13. The method of claim 1, wherein the shot event is selected from the group consisting of dribble event, jump event, catch-ball event, ball-leave-hand event, one-two leg jump, and wherein the shot type is selected from the group consisting of layup, regular shot, dribble-pull-up, off-the-move, and catch-and-shoot.

14. The method of claim 1, wherein the shot analytics is selected from the group consisting of release time, back angle, leg bend ratio, leg power, moving speed, moving direction, and height of jump.

15. The method of claim 2, wherein each CNN module has been trained using one or more prior input videos.

16. The method of claim 1, wherein the input video is received from a single mobile device camera.

17. A system for generating ball shot analytics using a single mobile computing device, comprising:

at least one processor on the mobile computing device; and

a non-transitory physical medium for storing program code and accessible by the processor, the program code when executed by the processor causes the processor to;

receive an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;

detect one or more balls and player postures from the input video;

generate one or more ball flows and one or more posture flows by grouping the detected balls and the detected player postures along a time line;

identify a generated player posture flow as a shooter posture flow, based on the input location of the shooter;

identify a generated ball flow as related to the shooter posture flow;

determine a ball-from-shooter time by backtracking the related ball flow from a shot attempt;

determine a shot event occurring before the ball-from-shooter time; and

generate one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.

18. The system of claim 17, wherein the detection of the one or more balls and player postures from the input video comprises using one or more Convolutional Neural Network (CNN) modules.

19. A non-transitory physical medium for generating ball shot analytics, the storage medium comprising program code stored thereon, the program code when executed by the processor causes the processor to:

receive an input video of a ball game, and a location of a shooter in a shooter identification frame of the input video;

detect one or more balls and player postures from the input video;

generate one or more ball flows and one or more posture flows by grouping the detected balls and the detected player postures along a time line;

identify a generated player posture flow as a shooter posture flow, based on the input location of the shooter;

identify a generated ball flow as related to the shooter posture flow;

determine a ball-from-shooter time by backtracking the related ball flow from a shot attempt;

determine a shot event occurring before the ball-from-shooter time; and

generate one or more shot analytics based on the shot event, the shooter posture flow, and the related ball flow.

20. The non-transitory physical medium of claim 19, wherein the detecting of the one or more balls and player postures from the input video comprises using one or more Convolutional Neural Network (CNN) modules.