Action detection and activity classification

• US 10,293,239 B2
• Filed: 05/21/2018
• Issued: 05/21/2019
• Est. Priority Date: 01/19/2012
• Status: Active Grant

• Current Assignee: Nike, Inc.
• Original Assignee: Nike, Inc.
• Inventors: Balakrishnan, Santoshkumar, Rice, Jordan M., Walker, Steven H., Carroll, Adam S., Dow-Hygelund, Corey C., Goodwin, Aaron K., Mullin, James M., Rattenbury, Tye L., Rooke-Ley, Joshua M., Schmitt, John M.
• Primary Examiner(s): Renwick, Reginald A
• Application Number:
  US15/984,732
• Publication Number:
  US 20180333611A1
• Time in Patent Office: 365 Days
• CPC Class Codes: A41D 1/002 adapted to accommodate electronic equipment patients' garments for medical monitoring A41D13/1281; luminous ornaments A41D27/085
• CPC Class Codes: A43B 3/34 with electrical or electronic arrangements
• CPC Class Codes: A61B 2503/10 Athletes
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0252 Load cells
• CPC Class Codes: A61B 2562/029 Humidity sensors
• CPC Class Codes: A61B 2562/08 Sensors provided with means for identification, e.g. barcodes or memory chips
• CPC Class Codes: A61B 5/0002 Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network A61B5/07 takes precedence; transmission systems for measured values G08C; transmission H04B; transmission of digital information H04L; wireless communication networks H04W
• CPC Class Codes: A61B 5/0004 characterised by the type of physiological signal transmitted
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0059 using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/02438 with portable devices, e.g. worn by the patient
• CPC Class Codes: A61B 5/1112 Global tracking of patients, e.g. by using GPS
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/1123 Discriminating type of movement, e.g. walking or running A61B5/1116, A61B5/112 take precedence
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6807 Footwear
• CPC Class Codes: A61B 5/681 Wristwatch-type devices
• CPC Class Codes: A61B 5/7246 : using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7257 : using Fourier transforms
• CPC Class Codes: A61B 5/7264 : Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7282 : Event detection, e.g. detecting unique waveforms indicative of a medical condition A61B5/4094, A61B5/4818, A61B5/349, A61B5/0823 take precedence
• CPC Class Codes: A61B 5/742 : using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/7475 : User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: A63B 2024/0065 : Evaluating the fitness, e.g. fitness level or fitness index
• CPC Class Codes: A63B 24/0003 : Analysing the course of a movement or motion sequences during an exercise or trainings sequence, e.g. swing for golf or tennis
• CPC Class Codes: A63B 24/0062 : Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 24/0075 : Means for generating exercise programs or schemes, e.g. computerized virtual trainer, e.g. using expert databases
• CPC Class Codes: A63B 71/06 : Indicating or scoring devices for games or players , or for other sports activities
• CPC Class Codes: G09B 19/003 : Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 19/0038 : Sports
• CPC Class Codes: G09B 5/00 : Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 5/02 : with visual presentation of the material to be studied, e.g. using film strip
• CPC Class Codes: G09B 5/125 : the stations being mobile
• CPC Class Codes: G16H 20/30 : relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/67 : for remote operation

Abstract

Activities, actions and events during user performance of physical activity may be detected using various algorithms and templates. Templates may include an arrangement of one or more states that may identify particular event types and timing between events. Templates may be specific to a particular type of activity (e.g., types of sports, drills, events, etc.), user, terrain, time of day and the like.

Claims

1. One or more non-transitory computer readable media storing computer readable instructions that, when executed, cause at least one computing device to:

determine a type of physical activity associated with an athletic activity performed by a user;

based on the determined type of physical activity performed by the user, identify a plurality of activity templates associated with the physical activity;

receive raw sensor data associated with the athletic activity performed by the user, the raw sensor data being received from one or more sensing devices into a buffer;

analyze the received raw sensor data in the buffer, further comprising;

mean centering and then determining a mean of an absolute value of acceleration in each of one or more subsections of the buffer;

delay further analysis of the raw sensor data in the buffer until the raw sensor data in the one or more subsections has a threshold acceleration magnitude;

calculate a Fast Fourier Transform of raw sensor data to determine one or more events performed during the athletic activity of the user;

compare the determined one or more events to a plurality of activity templates, further comprising;

determine whether the raw sensor data indicates that a foot contact duration is within a threshold range of an average duration of one or more previously matched foot contact events of the plurality of activity templates;

determine whether the determined one or more events match one or more activity templates of the plurality of activity templates;

responsive to determining that the determined one or more events match one or more activity templates, classify the events as performance of a first action of a first type and communicate sound feedback to the user based on the first action; and

responsive to determining that the determined one or more events do not match one or more activity templates, storing the received sensor data.

2. The one or more non-transitory computer readable media of claim 1, further including instructions that, when executed, cause the at least one computing device to:

determine a quality of the first action;

determine whether the quality of the first action is below a predetermined threshold; and

responsive to determining that the quality of the first action is below the predetermined threshold, provide coaching information to the user.

3. The one or more non-transitory computer-readable media of claim 2, wherein determining the quality of the first action further includes instructions that, when executed, cause the computing device to:

determine a strength of a match between the determined one or more events and the one or more activity templates.

4. The one or more non-transitory computer-readable media of claim 2, wherein determining the quality of the first action further includes instructions that, when executed, cause the computing device to:

identify one or more optional events associated with the one or more matching activity templates;

determine whether a match exists between the one or more optional events associated with the one or more matching activity templates and the determined one or more events.

5. The one or more non-transitory computer-readable media of claim 2, wherein providing coaching information to the user further includes instructions that, when executed, cause the computing device to:

provide recommendations for improving the quality of the first type of action.

6. The one or more non-transitory computer-readable media of claim 2, wherein the coaching information provided to the user is specific to at least one of the one or more determined events.

7. The one or more non-transitory computer-readable media of claim 1, wherein at least one sensor of the one or more sensing devices is arranged in an article of footwear of the user.

8. A system comprising:

a sensor system including a plurality of sensors, the sensor system further including;

a first processor; and

memory storing computer readable instructions that, when executed, cause the first processor to;

receive raw sensor data associated with athletic activity of a user into a buffer;

determine, based on a signature of a signal received from the plurality of sensors, a category of athletic activity being performed; and

an activity processing system, including;

a second processor; and

memory storing computer-readable instructions that, when executed, cause the second processor to;

receive the determined category of athletic activity being performed;

analyze the received raw sensor data in the buffer, further comprising;

mean centering and then determining a mean of an absolute value of acceleration in each of one or more subsections of the buffer;

delay further analysis of the raw sensor data in the buffer until the raw sensor data in the one or more subsections has a threshold acceleration magnitude;

calculate a Fast Fourier Transform of raw sensor data to determine one or more events performed during athletic activity of a user;

select a plurality of athletic activity templates for comparison, the plurality of athletic activity templates being selected based on the determined category of athletic activity being performed;

compare the determined one or more events to the selected plurality of activity templates, further comprising;

determine whether the raw sensor data indicates that a foot contact duration is within a threshold range of an average duration of one or more previously matched foot contact events of the plurality of activity templates;

determine whether the determined one or more events match one or more activity templates of the selected plurality of activity templates; and

responsive to determining that the determined one or more events match one or more activity templates, classify the events of the user as performance of a first action of a first type and communicate sound feedback to the user based on the first action.

9. The system of claim 8, the activity processing system further including instructions that, when executed, cause the second processor to:

determine a quality of the first action;

determine whether the quality of the first action is below a predetermined threshold; and

responsive to determining that the quality of the first action is below the predetermined threshold, provide coaching information to the user.

10. The system of claim 9, wherein determining the quality of the first action further includes instructions that, when executed, cause the second processor to:

determine a strength of a match between the determined one or more events and the one or more activity templates.

11. The system of claim 9, wherein determining the quality of the first action includes instructions that, when executed, cause the second processor to:

identify one or more optional events associated with the one or more matching activity templates;

determine whether a match exists between the one or more optional events associated with the one or more matching activity templates and the determined one or more events.