Action Detection and Activity Classification

• US 20130190903A1
• Filed: 02/21/2012
• Published: 07/25/2013
• Est. Priority Date: 01/19/2012
• Status: Active Grant

• Current Assignee: Nike, Inc.
• Original Assignee: Nike, Inc.
• Inventors: Rice, Jordan M., Walker, Steven H., Goodwin, Aaron K., Mullin, James M., Schmitt, John M., Balakrishnan, Santosh, Carroll, Adam S., Dow-Hygelund, Corey C., Rattenbury, Tye L., Rooke-Ley, Joshua M.
• Granted Patent: US 9,352,207 B2
• Time in Patent Office: Days
• US Class Current: 700/91
• CPC Class Codes: A41D 1/002 adapted to accommodate electronic equipment patients' garments for medical monitoring A41D13/1281; luminous ornaments A41D27/085
• CPC Class Codes: A43B 3/34 with electrical or electronic arrangements
• CPC Class Codes: A61B 2503/10 Athletes
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0252 Load cells
• CPC Class Codes: A61B 2562/029 Humidity sensors
• CPC Class Codes: A61B 2562/08 Sensors provided with means for identification, e.g. barcodes or memory chips
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/0004 characterised by the type of physiological signal transmitted
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0059 using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6807 Footwear
• CPC Class Codes: A61B 5/681 Wristwatch-type devices
• CPC Class Codes: A61B 5/7246 : using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7257 : using Fourier transforms
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7282 : Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/742 : using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/7475 : User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: A63B 2024/0065 : Evaluating the fitness, e.g. fitness level or fitness index
• CPC Class Codes: A63B 24/0003 : Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0062 : Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 24/0075 : Means for generating exercise programs or schemes, e.g. computerized virtual trainer, e.g. using expert databases
• CPC Class Codes: A63B 71/06 : Indicating or scoring devices for games or players , or for other sports activities
• CPC Class Codes: G09B 19/003 : Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 19/0038 : Sports
• CPC Class Codes: G09B 5/00 : Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 5/02 : with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09B 5/125 : the stations being mobile
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

Activities, actions and events during user performance of physical activity may be detected using various algorithms and templates. Templates may include an arrangement of one or more states that may identify particular event types and timing between events. Templates may be specific to a particular type of activity (e.g., types of sports, drills, events, etc.), user, terrain, time of day and the like.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device, information identifying a plurality of events detected from one or more sensor signal streams during performance of a physical activity by a user;

evaluating, by the computing device, the plurality of events against one or more constraints of the first action template;

determining whether the plurality of events matches the first action template based on the evaluation of the plurality of events against the one or more constraints of the first action template; and

in response to determining that the plurality of events matches the one or more constraints of the first action template, registering user performance of a first type of action corresponding to the first action template.

2. The computer-implemented method of claim 1, further comprising:

determining whether one or more events of the plurality of events matches an exclusion state constraint defined in the first action template; and

in response to determining that the one or more events matches the exclusion state constraint, determining that the one or more events is not a match with the first action template or that the plurality of events are not a match with the first action template.

3. The computer-implemented method of claim 1, wherein comparing the plurality of events to a first action template includes:

selecting a first state from the first action template to evaluate; and

determining whether a first candidate event of the plurality of events meets one or more non-relative constraints of the first state.

4. The computer-implemented method of claim 3, wherein comparing the plurality of events to the first action template further includes:

determining whether the first candidate event meets one or more relative constraints of the first state, wherein the one or more relative constraints defines a required relationship between the first candidate event and a second candidate event.

5. The computer-implemented method of claim 4, wherein the one or more relative constraints further define at least one of:

a required relative timing between a current template matching analysis and one or more past template matching analyses of one or more other events, and a required relationship with statistics regarding prior occurrences of events of a type corresponding to the first state.

6. The computer-implemented method of claim 4, wherein determining whether the first candidate event meets the one or more relative constraints of the first state includes:

determining whether the second candidate event of the plurality of events meets the one or more relative constraints of the first event; and

in response to determining that the second candidate event meets the one or more relative constraints of the first event, determining whether the second candidate event meets one or more constraints of the second event specified in the first action template.

7. The computer-implemented method of claim 4, wherein upon determining that the first candidate event meets the non-relative and relative constraints of the first state, analyzing each of one or more remaining template states in a sequential order such that the analysis for each of the one or more remaining template state includes:

determining whether a candidate event from the plurality of events meets the one or more non-relative constraints of the template state being analyzed;

determining whether the candidate event from the plurality of events that satisfied the non-relative constraints of the template state being analyzed also satisfies one or more relative constraints of the template state being analyzed, wherein the one or more relative constraints depend on one or more template states already analyzed; and

in response to determining that the candidate event meets the one or more non-relative and relative constraints of the template state being analyzed, proceeding to analyze another remaining template state.

8. The computer-implemented method of claim 1, further comprising:

selecting the first action template based on a type of physical activity performed by the user.

9. The computer-implemented method of claim 1, further comprising:

register user performance of a second type of action based on the plurality of events matching a second action template, wherein each of the first and second action templates corresponds to a different type of action.

10. The computer-implemented method of claim 1, further comprising non-dimensionalizing one or more characteristics of the plurality of events.

11. The computer-implemented method of claim 1, further comprising:

determining a level of match between the plurality of events and the first action template; and

generating one or more instructions or recommendations for improving user performance of the first type of action based on one or more of;

the determined level of match, a location of the user performance of the first type of action and a time of the user performance of the first type of action.

12. The computer-implemented method of claim 1, further comprising:

determining that the plurality of events matches the first action template and a second action template;

selecting one of the first and second templates; and

defining the first type of action based on the selected one of the first and second templates.

13. The computer-implemented method of claim 12, wherein selecting the one of the first and second action templates includes:

determining a first level of match between the plurality of events and the first action template;

determining a second level of match between the plurality of events and the second action template; and

selecting the one of the first and second templates having a greater level of match.

14. The computer-implemented method of claim 1, wherein the first action template includes a match tolerance.

15. One or more non-transitory computer readable media storing computer readable instructions that, when executed, cause an apparatus to process a physical activity action template, the physical activity action template storing:

a plurality of required states; and

for each of the plurality of required states, a state-specific set of one or more constraints defining match characteristics of a corresponding required event,

wherein a state-specific set of one or more constraints for a first state of the plurality of required states includes a relative constraint defining a required relationship between the first state and a second state of the plurality of required states.

16. The one or more non-transitory computer readable media of claim 15, wherein the state-specific set of one or more constraints further includes a scoring distribution for determining a match score for the one or more constraints.

17. The one or more non-transitory computer readable media of claim 16, wherein the state-specific set of one or more constraints further includes a first parameter indicating whether a first characteristic of the corresponding event is to be scored.

18. The one or more non-transitory computer readable media of claim 17, wherein the state-specific set of one or more constraints further includes a second parameter indicating whether a second characteristic of the corresponding event is to be scored.

19. The one or more non-transitory computer readable media of claim 15, wherein the state-specific set of one or more constraints includes at least one of:

an event occurrence time stamp constraint, an event generation time stamp constraint, an event value constraint and an event duration constraint.

20. The one or more non-transitory computer readable media of claim 19, wherein the state-specific set of one or more constraints includes different scoring distributions for each of the at least one of:

the event occurrence time stamp constraint, the event value constraint, the event generation time stamp constraint and the event duration constraint.

21. The one or more non-transitory computer readable media of claim 15, wherein the physical activity action template further stores one or more exclusion state constraints configured to exclude a candidate event from being identified as potential matches with one or more of the plurality of required states when the candidate event is determined to match the one or more exclusion state constraints.

22. The one or more non-transitory computer readable media of claim 15, wherein the physical activity action template further includes an optional state to be matched.

23. The one or more non-transitory computer readable media of claim 22, wherein the optional state is defined in the physical activity action template to provide a user with a bonus for performance of an action represented by the physical activity action template.

24. An apparatus comprising:

a processor; and

memory storing computer readable instructions that, when executed, cause the apparatus to;

receive information identifying a plurality of events detected from one or more sensor signal streams during performance of a physical activity by a user;

evaluate the plurality of events against one or more constraints of the first action template;

determine whether the plurality of events matches the first action template based on the evaluation of the plurality of events against the one or more constraints of the first action template; and

in response to determining that the plurality of events matches the one or more constraints of the first action template, register user performance of a first type of action corresponding to the first action template.