Jump shot and athletic activity analysis system

• US 9,589,207 B2
• Filed: 11/21/2013
• Issued: 03/07/2017
• Est. Priority Date: 11/21/2013
• Status: Active Grant

• Current Assignee: Mo' Motion Ventures
• Original Assignee: Mo' Motion Ventures
• Inventors: Holohan, Maureen
• Primary Examiner(s): Park, Soo Jin
• Application Number:
  US14/086,554
• Publication Number:
  US 20150139502A1
• Time in Patent Office: 1,202 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

Techniques where a computer or mobile device performs video analysis of a person performing a physical activity are described. The computer or mobile device performs video analysis based on one or more reference skeletons. The reference skeleton may be superimposed over captured video of the person performing the physical activity or compared to a generated player skeleton based on the captured video to determine derivation from the reference skeleton in the physical activity performed by the person.

Claims

1. A computer program product for video analysis of a basketball jump shot, the computer program product comprising:

a plurality of processor-executable instructions configured for processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, the instructions being resident on a non-transitory computer-readable storage medium and being configured, upon execution by a processor, to;

determine a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

process pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

receive input indicative of a size and shooting hand for the person;

define a reference skeleton for the basketball jump shot phase in response to the received input such that the reference skeleton is scaled for the person'"'"'s size and shooting hand;

compare the determined body position in the determined frame with the reference skeleton; and

determine a deviation between the determined body position and the reference skeleton based on the comparison operation.

2. The computer program product of claim 1 wherein the reference skeleton is representative of a desired body position with respect to a fundamental aspect of the basketball jump shot phase, and wherein the deviation is representative of a difference in the fundamental aspect as between the determined body position and the desired body position of the reference skeleton.

3. The computer program product of claim 2, wherein the plurality of processor-executable instructions are further configured to:

perform the frame determination operation, the processing operation, the comparison operation, and the deviation determination operation with respect to a plurality of the basketball jump shot phases.

4. The computer program product of claim 3 wherein the phases comprise at least two members of the group consisting of (1) the base phase, (2) the frame phase, (3) the release phase, (4) the follow-through phase, and (5) the finish phase.

5. The computer program product of claim 3 wherein the phases comprise (1) the base phase, (2) the frame phase, (3) the release phase, (4) the follow-through phase, and (5) the finish phase.

6. The computer program product of claim 2, wherein the plurality of processor-executable instructions are further configured to:

compute an arc of a basketball for the basketball jump shot based on the video data.

7. The computer program product of claim 2, wherein the plurality of processor-executable instructions are further configured to:

compute data indicative of a release rate for the basketball jump shot based on the video data.

8. The computer program product of claim 1, wherein the plurality of processor-executable instructions are further configured to:

repeat the frame determination operation, the processing operation, the comparison operation, and the deviation determination operation for a plurality of basketball jump shots by the person; and

maintain data indicative of whether the basketball jump shots went through a basketball hoop;

build a database associated with the basketball jump shots, the database including the determined deviations for the basketball jump shots and data indicative of which of the basketball jump shots went through the basketball hoop.

9. The computer program product of claim 8, wherein the plurality of processor-executable instructions are further configured to:

analyze the database to generate metrics about the person'"'"'s basketball jump shots; and

communicate the generated metrics for presentation to a user.

10. The computer program product of claim 8, wherein the plurality of processor-executable instructions are further configured to:

determine a plurality of deviations with respect to a plurality of the basketball jump shots based on the repeated operations; and

analyze the database to correlate the determined deviations for the basketball jump shots with respect to the data indicative of whether the basketball jump shots went through the basketball hoop.

11. The computer program product of claim 10, wherein the plurality of processor-executable instructions are further configured to:

identify which determined deviations exhibit a correlation beyond a threshold with respect to basketball jump shots that did not go through the basketball hoop.

12. The computer program product of claim 1 wherein the plurality of processor-executable instructions are further configured to:

search a plurality of the frames of the video data to identify a frame that exhibits a plurality of characteristics that are representative of the basketball jump shot phase, wherein the identified frame serves as the determined frame.

13. The computer program product of claim 1, wherein the plurality of processor-executable instructions are further configured to:

communicate a result based on the determined deviation.

14. The computer program product of claim 1 wherein the plurality of processor-executable instructions are further configured to:

superimpose the reference skeleton over the determined body position.

15. The computer program product of claim 1 wherein the video data exhibits a perspective with respect to the person.

16. The computer program product of claim 15 wherein the perspective comprises a member of the group consisting of a front view and a side view.

17. The computer program product of claim 1 wherein the plurality of processor-executable instructions are further configured to:

capture the video data with a camera resident on a mobile computing device.

18. The computer program product of claim 1 wherein the determined frame comprises a plurality of frames that depict the person'"'"'s body position in the basketball jump shot phase.

19. The computer program product of claim 1, wherein the reference skeleton comprises an image that depicts a plurality joints and lines connecting the joints that are representative of a player in the basketball jump shot phase.

20. The computer program product of claim 1, wherein the reference skeleton comprises a plurality of data points indicating locations of a plurality of joints and limbs relative to each other that are representative of a player in the basketball jump shot phase.

21. The computer program product of claim 1, wherein the plurality of processor-executable instructions are further configured to:

calculate body edge coordinates and a body size for the person in the basketball jump shot phase based on the processed pixels of the determined frame;

determine joint coordinates for the person in the basketball jump shot phase based on the body edge coordinates and the body size; and

perform the comparison based on the determined joint coordinates relative to the reference skeleton.

22. The computer program product of claim 1, wherein the received user input further comprises data indicative of a gender for the person, and wherein the plurality of processor-executable instructions are further configured to scale the reference skeleton for the person'"'"'s gender.

23. A method for video analysis of a basketball jump shot, the method comprising:

processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, wherein the processing step comprises;

a processor determining a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

the processor processing pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

the processor comparing the determined body position in the determined frame with a reference skeleton for the phase of the basketball jump shot phase;

the processor determining a deviation between the determined body position and the reference skeleton based on the comparing step;

maintaining a database that associates a plurality of different types of deviations for the basketball jump shot phases with data representative of an explanation regarding each deviation type;

classifying the determined deviation as a deviation type from among a plurality of different types of deviations;

analyzing the database to map the determined deviation to the explanation data associated with the classified deviation type for the determined deviation; and

communicating the associated explanation data to a user to facilitate coaching of the person'"'"'s basketball jump shot.

24. An apparatus for video analysis of a basketball jump shot, the apparatus comprising:

a processor configured for processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, wherein the processor is configured to;

determine a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

process pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

compare the determined body position in the frame with a reference skeleton for the basketball jump shot phase;

determine a deviation between the determined body position and the reference skeleton based on the comparison operation;

repeat the frame determination operation, the processing operation, the comparison operation, and the deviation determination operation for a plurality of basketball jump shots by the person such that a plurality of deviations are determined with respect to a plurality of the basketball jump shots based on the repeated operations, wherein the deviations correspond to a plurality of different types of deviations;

maintain data indicative of whether the basketball jump shots went through a basketball hoop;

build a database associated with the basketball jump shots, the database including the determined deviations for the basketball jump shots and data indicative of which of the basketball jump shots went through the basketball hoop;

analyze the database to correlate the determined deviations for the basketball jump shots with respect to the data indicative of whether the basketball jump shots went through the basketball hoop; and

classify at least one of the determined deviation types as being an outcome-determinative deviation type based on the correlations of the determined deviations with the data indicative of whether the basketball jump shots went through the basketball hoop.

25. The apparatus of claim 24 wherein the processor is resident on a mobile computing device.

26. The apparatus of claim 24 wherein the processor is resident on a server system.

27. A system comprising:

a camera configured to generate a plurality of image frames of a person performing a basketball jump shot;

a memory; and

a processor for cooperation with the camera and the memory, the processor configured to;

receive the image frames from the camera;

process pixels of the image frames to determine and identify which of the image frames is representative of a crest of a jump by the player for the basketball jump shot in the release phase;

process pixels in the identified image frame to determine a body position for the person within the identified image frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the identified frame;

compare the determined body position to a release phase reference body position;

determine whether the person demonstrated a deviation from the release phase reference body position while performing the basketball jump shot as a result of the comparison operation; and

in response to a determination that the person demonstrated the deviation, store data representative of the deviation in the memory.

28. The system of claim 27, wherein the processor is further configured to:

process pixels of the identified release phase image frame to determine whether the basketball has already left a shooting hand of the person; and

in response to a determination that the basketball had already left the shooting hand, (1) determine that an error corresponding to a release point being too low for the person exists, and (2) communicate the determined error to a user to facilitate coaching of the person'"'"'s basketball jump shot.

29. The system of claim 27 wherein the camera comprises (1) a first camera configured to generate image frames corresponding to a front view of the person performing the basketball jump shot, and (2) a second camera configured to generate image frames corresponding to a side view of the person performing the basketball jump shot, and wherein the identified image frame includes a front view image frame corresponding to the release phase and a side view image frame corresponding to the release phase.

30. A system comprising:

a camera configured to generate a plurality of image frames of a person performing a basketball jump shot;

a memory; and

a processor for cooperation with the camera and the memory, wherein the processor is configured to;

receive the image frames from the camera;

process pixels of the image frames to determine and identify which of the image frames is representative of the player in a frame phrase of the basketball jump shot and preparing to execute the basketball jump shot such that the player is on his or her toes but not yet in the air;

process pixels in the identified image frame to determine a body position for the person within the identified image frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the identified frame;

compare the determined body position with a frame phase reference body position;

determine whether the person demonstrated a deviation from the frame phase reference body position while performing the basketball jump shot based on the comparison operation; and

in response to a determination that the person demonstrated the deviation, store data representative of the deviation in the memory.

31. A system comprising:

a camera configured to generate a plurality of image frames of a person performing a basketball jump shot;

a memory; and

a processor for cooperation with the camera and the memory, wherein the processor is configured to;

receive the image frames from the camera;

process pixels of the image frames to identify an image frame that depicts the person in a frame phase of the basketball jump shot;

process pixels in the identified image frame to determine a body position for the person within the identified image frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the identified frame;

compare the determined body position with a frame phase reference body position;

determine based on the comparison whether the elbow corresponding to the player'"'"'s shooting hand is flared out relative to the frame phase reference body position; and

in response to a determination that the elbow corresponding to the player'"'"'s shooting hand is flared out relative to the frame phase reference body position, (1) determine that an error corresponding to an elbow flare out exists, (2) store data representative of the error in the memory, and communicate the determined error to a user to facilitate coaching of the person'"'"'s basketball jump shot.

32. The system of claim 31 wherein the camera comprises (1) a first camera configured to generate image frames corresponding to a front view of the person performing the basketball jump shot, and (2) a second camera configured to generate image frames corresponding to a side view of the person performing the basketball jump shot, and wherein the identified image frame includes a front view image frame corresponding to the frame phase and a side view image frame corresponding to the frame phase.

33. A computer program product for video analysis of a basketball jump shot, the computer program product comprising:

a plurality of processor-executable instructions configured for processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, the instructions being resident on a non-transitory computer-readable storage medium and being configured, upon execution by a processor, to;

determine a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

process pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

compare the determined body position in the determined frame with a reference skeleton for the basketball jump shot phase;

determine a deviation between the determined body position and the reference skeleton based on the comparison operation;

maintain a database that associates a plurality of different types of deviations for the basketball jump shot phases with data representative of an explanation regarding each deviation type;

classify the determined deviation as a deviation type from among a plurality of different types of deviations;

analyze the database to map the determined deviation to the explanation data associated with the classified deviation type for the determined deviation; and

communicate the associated explanation data to a user to facilitate coaching of the person'"'"'s basketball jump shot.

34. The computer program product of claim 33 wherein the instructions are configured for execution by a mobile computing device.

35. The computer program product of claim 33 wherein the instructions are configured for execution by a server system.

36. The computer program product of claim 33, wherein the plurality of processor-executable instructions are further configured to:

perform the frame determination operation, the processing operation, the comparison operation, and the deviation determination operation with respect to first video data from a first perspective and second video data from a second perspective.

37. The computer program product of claim 33 wherein the processor-executable instructions comprise a first plurality of the processor-executable instructions configured for execution by a first processor and a second plurality of the processor-executable instructions configured for execution by a second processor.

38. The computer program product of claim 33 wherein the deviation types comprise a plurality of members of the group consisting of (1) a release point too low deviation type, (2) a too upright without enough knee bend and squat deviation type, (3) an imbalance deviation type, (4) an elbow flare out deviation type, (5) a misplaced guide hand deviation type, (6) a falling backward deviation type, (7) a landing forward deviation type, and (8) a not square deviation type.

39. A computer program product for video analysis of a basketball jump shot, the computer program product comprising:

a plurality of processor-executable instructions configured for processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, the instructions being resident on a non-transitory computer-readable storage medium and being configured, upon execution by a processor, to;

determine a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

process pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

compare the determined body position in the determined frame with a reference skeleton for the basketball jump shot phase; and

determine a deviation between the determined body position and the reference skeleton based on the comparison operation;

repeat the frame determination operation, the processing operation, the comparison operation, and the deviation determination operation for a plurality of basketball jump shots by the person such that a plurality of deviations are determined with respect to a plurality of the basketball jump shots based on the repeated operations, wherein the deviations correspond to a plurality of different types of deviations;

maintain data indicative of whether the basketball jump shots went through a basketball hoop;

build a database associated with the basketball jump shots, the database including the determined deviations for the basketball jump shots and data indicative of which of the basketball jump shots went through the basketball hoop;

analyze the database to correlate the determined deviations for the basketball jump shots with respect to the data indicative of whether the basketball jump shots went through the basketball hoop; and

classify at least one of the determined deviation types as being an outcome-determinative deviation type based on the correlations of the determined deviations with the data indicative of whether the basketball jump shots went through the basketball hoop.

40. A computer program product for video analysis of a basketball jump shot, the computer program product comprising:

a plurality of processor-executable instructions configured for processing video data, the video data comprising a plurality of frames of a person performing the basketball jump shot, the instructions being resident on a non-transitory computer-readable storage medium and being configured, upon execution by a processor, to;

determine a frame of the video data that depicts a body position of the person in a phase of the basketball jump shot, wherein the phase comprises a member of the group consisting of (1) a base phase, (2) a frame phase, (3) a release phase, (4) a follow-through phase, and (5) a finish phase;

process pixels in the determined frame to determine a body position for the person within the determined frame, wherein the determined body position comprises data indicative of an arrangement of joints and limbs for the person within the determined frame;

receive input indicative of a skill level for the person;

compare the determined body position in the determined frame with a reference skeleton for the basketball jump shot phase;

compute data representative of a difference between the body position in the frame and the reference skeleton in response to the comparison between the body position in the frame and the reference skeleton for the basketball jump shot phase;

compare the computed data with a threshold that is scaled based on the skill level input; and

in response to a determination that the computed data exceeds the threshold, classify the difference as a deviation between the determined body position and the reference skeleton.