Action detection and activity classification

• US 9,962,111 B2
• Filed: 04/28/2017
• Issued: 05/08/2018
• Est. Priority Date: 01/19/2012
• Status: Active Grant

• Current Assignee: Nike, Inc.
• Original Assignee: Nike, Inc.
• Inventors: Balakrishnan, SantoshKumar, Rice, Jordan M., Walker, Steven H., Carroll, Adam S., Dow-Hygelund, Corey C., Goodwin, Aaron K., Mullin, James M., Rattenbury, Tye L., Rooke-Ley, Joshua M., Schmitt, John M.
• Primary Examiner(s): Renwick, Reginald
• Application Number:
  US15/581,108
• Publication Number:
  US 20170224255A1
• Time in Patent Office: 375 Days
• CPC Class Codes: A41D 1/002 adapted to accommodate electronic equipment patients' garments for medical monitoring A41D13/1281; luminous ornaments A41D27/085
• CPC Class Codes: A43B 3/34 with electrical or electronic arrangements
• CPC Class Codes: A61B 2503/10 Athletes
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0252 Load cells
• CPC Class Codes: A61B 2562/029 Humidity sensors
• CPC Class Codes: A61B 2562/08 Sensors provided with means for identification, e.g. barcodes or memory chips
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/0004 characterised by the type of physiological signal transmitted
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0059 using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6807 Footwear
• CPC Class Codes: A61B 5/681 Wristwatch-type devices
• CPC Class Codes: A61B 5/7246 : using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7257 : using Fourier transforms
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7282 : Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/742 : using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/7475 : User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: A63B 2024/0065 : Evaluating the fitness, e.g. fitness level or fitness index
• CPC Class Codes: A63B 24/0003 : Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0062 : Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 24/0075 : Means for generating exercise programs or schemes, e.g. computerized virtual trainer, e.g. using expert databases
• CPC Class Codes: A63B 71/06 : Indicating or scoring devices for games or players , or for other sports activities
• CPC Class Codes: G09B 19/003 : Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 19/0038 : Sports
• CPC Class Codes: G09B 5/00 : Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 5/02 : with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09B 5/125 : the stations being mobile
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

Activities, actions and events during user performance of physical activity may be detected using various algorithms and templates. Templates may include an arrangement of one or more states that may identify particular event types and timing between events. Templates may be specific to a particular type of activity (e.g., types of sports, drills, events, etc.), user, terrain, time of day and the like.

Claims

1. A system, comprising:

a sensor system including a plurality of sensors, at least a first processor and at least a first memory storing computer-executable instructions that, when executed, cause the sensor system to;

receive, from the plurality of sensors, raw athletic activity data associated with a potential corresponding event during performance of an athletic activity;

process the raw athletic activity data;

transmit, from the sensor system to an activity processing system, the processed athletic activity data, transmitting the processed athletic activity data to an activity processing system including controlling, by the sensor system, at least one of a type of data and an amount of data transmitted, wherein the type of data or amount of data transmitted is controlled based on a sensor subscription;

the activity processing system including at least a second processor and at least a second memory storing computer-executable instructions that, when executed, cause the activity processing system to;

receive, from the sensor system, the processed athletic activity data;

compare the received processed athletic activity data to a plurality of physical action templates to determine whether a physical activity of at least one physical action template of the plurality of physical action templates has occurred, the comparing including;

determining a plurality of required states associated with the physical activity of the at least one physical action template and forming a portion of the at least one physical action template;

for each of the plurality of required states, determine a state-specific set of one or more constraints defining match characteristics of a corresponding required event,

determine, based on the comparison, a match score for the one or more constraints;

determine, based on the match score, whether the physical activity has occurred; and

transmit a result of the determining whether the physical activity has occurred to one or more devices.

2. The system of claim 1, wherein the sensor subscription is determined based on identification of a type of athletic activity performed.

3. The system of claim 1, wherein a state-specific set of one or more constraints for a first state of the plurality of required states includes a relative constraint defining a required relationship between the first state and a second state of the plurality of required states, and

wherein the state-specific set of one or more constraints further includes a scoring distribution for determining the match score for the one or more constraints.

4. The system of claim 3, wherein the state-specific set of one or more constraints further includes a first parameter indicating whether a first characteristic of the corresponding event is to be scored.

5. The system of claim 1, wherein the state-specific set of one or more constraints further includes a second parameter indicating whether a second characteristic of the corresponding event is to be scored.

6. The system of claim 1, wherein the state-specific set of one or more constraints includes at least one of:

an event occurrence time stamp constraint, an event generation time stamp constraint, an event value constraint and an event duration constraint.

7. The system of claim 1, wherein the one or more devices include a remote activity tracking system.

8. The system of claim 1, further including instructions that, when executed, cause the first processor to:

responsive to determining, based on the match score, that the physical activity has occurred, displaying that a user has performed the physical activity corresponding to the at least one physical action template on a display of the one or more devices.

9. The system of claim 1, wherein the plurality of sensors are arranged in at least one of:

an article of apparel, an article of footwear, and a wrist-borne device.

10. One or more non-transitory computer readable media storing computer readable instructions that, when executed, cause a processor to:

receive, by a sensor system having a plurality of sensors and from the plurality of sensors, raw athletic activity data associated with a potential corresponding event during performance of the athletic activity;

process the raw athletic activity data;

transmit, from the sensor system to an activity processing system, the processed athletic activity data, transmitting the processed athletic activity data to the activity processing system including controlling, by the sensor system, at least one of a type of data and an amount of data transmitted, wherein the type of data or amount of data transmitted is controlled based on a sensor subscription;

receive, by the activity processing system, the processed athletic activity data;

compare, by the activity processing system, the received, processed athletic activity data to a plurality of physical action templates to determine whether a physical activity of at least one physical action template of the plurality of physical action templates has occurred, the comparing including;

determining a plurality of required states associated with the physical activity of the at least one physical action template and forming a portion of the at least one physical action template;

for each of the plurality of required states, determine a state-specific set of one or more constraints defining match characteristics of a corresponding required event;

determining, based on the comparison, a match score for the one or more constraints;

determining, based on the match score, whether the physical activity has occurred; and

transmitting a result of the determining whether the physical activity has occurred to one or more devices.

11. The one or more non-transitory computer readable media of claim 10, wherein the sensor subscription is determined based on identification of a type of athletic activity performed.

12. The one or more non-transitory computer readable media of claim 10, wherein a state-specific set of one or more constraints for a first state of the plurality of required states includes a relative constraint defining a required relationship between the first state and a second state of the plurality of required states,

wherein the state-specific set of one or more constraints includes at least one of:

an event occurrence time stamp constraint, an event generation time stamp constraint, an event value constraint and an event duration constraint, and

wherein the state-specific set of one or more constraints includes different scoring distributions for each of the at least one of;

the event occurrence time stamp constraint, the event value constraint, the event generation time stamp constraint and the event duration constraint.

13. The one or more non-transitory computer readable media of claim 10, wherein the physical activity action template further stores one or more exclusion state constraints configured to exclude the potential corresponding event from being identified as potential matches with one or more of the plurality of required states when the potential corresponding event is determined to match the one or more exclusion state constraints.

14. The one or more non-transitory computer readable media of claim 10, wherein the one or more devices include a remote activity tracking system.

15. The one or more non-transitory computer readable media of claim 10, further including instructions that, when executed, cause the processor to:

responsive to determining, based on the match score, that the physical activity has occurred, displaying that a user has performed the physical activity corresponding to the at least one physical action template on a display of the one or more devices.

16. The one or more non-transitory computer readable media of claim 10, wherein the plurality of sensors are arranged in at least one of:

an article of apparel, an article of footwear, and a wrist-borne device.

17. A method, comprising:

receiving, by a sensor system having a plurality of sensors and from the plurality of sensors, raw athletic activity data associated with a potential corresponding event during performance of the athletic activity;

processing, by the sensor system, the raw athletic activity data;

transmitting, from the sensor system to an activity processing system, the processed athletic activity data, transmitting the processed athletic activity data to the activity processing system including controlling, by the sensor system, at least one of a type of data and an amount of data transmitted, wherein the type of data or amount of data transmitted is controlled based on a sensor subscription;

receiving, by the activity processing system, the processed athletic activity data;

comparing, by the activity processing system, the received, processed athletic activity data to a plurality of physical action templates to determine whether a physical activity of at least one physical action template of the plurality of physical action templates has occurred, the comparing including;

determining a plurality of required states associated with the physical activity of the at least one physical action template and forming a portion of the at least one physical action template;

for each of the plurality of required states, determining a state-specific set of one or more constraints defining match characteristics of a corresponding required event,

determining, based on the comparison, a match score for the one or more constraints;

determining, based on the match score, whether the physical activity has occurred; and

transmitting a result of the determining whether the physical activity has occurred to one or more devices.

18. The method of claim 17, wherein the one or more devices include a remote activity tracking system.