Devices, systems, and methods for monitoring and assessing gait, stability, and/or balance of a user

• US 10,842,415 B1
• Filed: 10/25/2019
• Issued: 11/24/2020
• Est. Priority Date: 10/25/2019
• Status: Active Grant

• Current Assignee: Plethy, Inc.
• Original Assignee: Plethy, Inc.
• Inventors: Jagannathan, Ravi, Sundaram, Raja, Harikrishnan, Hari
• Primary Examiner(s): Gonzales, Vincent
• Application Number:
  US16/664,379
• Time in Patent Office: 396 Days
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0223 Magnetic field sensors
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/02055 Simultaneously evaluating both cardiovascular condition and temperature
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/053 Measuring electrical impedance or conductance of a portion of the body
• CPC Class Codes: A61B 5/1116 Determining posture transitions
• CPC Class Codes: A61B 5/112 Gait analysis
• CPC Class Codes: A61B 5/1121 Determining geometric values, e.g. centre of rotation or angular range of movement
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/14542 for measuring blood gases A61B5/14551 takes precedence
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: A61B 5/6801 specially adapted to be attached to or worn on the body surface
• CPC Class Codes: A61B 5/721 using a separate sensor to detect motion or using motion information derived from signals other than the physiological signal to be measured
• CPC Class Codes: A61B 5/7221 Determining signal validity, reliability or quality A61B5/7207 and A61B5/7217 take precedence
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A63B 24/0062 : Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: G06N 20/20 : Ensemble learning
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A method for assessing movement of a body portion includes, via one or more machine learning models, analyzing a sensor signal indicative of movement of the body portion to determine a movement of the body portion; determining a sensor confidence level based, at least in part, on a characteristic of the sensor signal; receiving a series of images indicative of movement of the body portion; measuring an angle of movement of the body portion; determining a vision confidence level based, at least in part, on a quality of an identification the body portion; selecting the sensor signal, the measured angle of movement, or a combination thereof as an input into a machine learning model based on the sensor confidence level and the vision confidence level, respectively; analyzing the input to determine a movement pattern of the body portion; and outputting the movement pattern to a user.

Claims

1. A system for assessing movement of one or more body portions, comprising:

one or more inertial sensors positioned on one or more body portions;

an image sensor;

a processor communicatively coupled to the one or more inertial sensors and the image sensor; and

a memory, coupled to the processor, configured to store program instructions, wherein, when executed by the processor, the program instructions cause the processor to perform a method comprising;

via a first machine learning model;

receiving a sensor signal using the one or more inertial sensors, the sensor signal being indicative of movement of the one or more body portions over a time period,

analyzing the sensor signal of the one or more body portions to determine a movement of the one or more body portions, and

determining a sensor confidence level based, at least in part, on a characteristic of the sensor signal from the one or more inertial sensors over the time period;

via a second machine learning model;

receiving a series of images using the image sensor, the series of images comprising the one or more body portions depicted therein and indicative of movement of the one or more body portions over the time period,

identifying the one or more body portions in each of the series of images,

measuring an angle of movement of the one or more body portions in at least a subset of the series of images over the time period, and

determining a vision confidence level based, at least in part, on an ability of the second machine learning model to identify the one or more body portions in the at least a subset of the series of images; and

via a third machine learning model;

selecting the sensor signal from the first machine learning model based on the sensor confidence level, the measured angle of movement from the second machine learning model based on the vision confidence level, or a weighted combination thereof as an input into the third machine learning model,

analyzing the input to determine a movement pattern of the one or more body portions, and

outputting the movement pattern to a user.

2. The system of claim 1, wherein the characteristic of the sensor signal comprises a smoothness of the sensor signal over the time period.

3. The system of claim 2, wherein the smoothness comprises a substantially sinusoidal pattern.

4. The system of claim 1, wherein the characteristic comprises a degree of rotation sensed by the one or more inertial sensors when positioned in parallel to a plane of movement.

5. The system of claim 1, wherein the sensor confidence level is further based on an expected movement of the one or more body portions versus an actual movement of the one or more body portions.

6. The system of claim 1, wherein the vision confidence level is further based on one or more attributes of the identified body portions in the series of images.

7. The system of claim 6, wherein the one or more attributes comprise one or more of:

a shape of the identified one or more body portions, a relative size of the identified one or more body portions, a location in each image of the identified one or more body portions, an acutance in each image of the identified one or more body portions, a comparative length of symmetrical body portions, and a combination thereof.

8. The system of claim 1, wherein the vision confidence level is further based on a binary classification of each pixel in each of the series of images.

9. The system of claim 1, wherein the vision confidence level is further based on an expected number of body portions in each image equaling an actual number of body portions in each image.

10. The system of claim 1, wherein the movement detected by the one or more inertial sensors comprises movement in three-dimensions.

11. The system of claim 1, wherein the movement detected by the one or more inertial sensors comprises movement in nine degrees-of-freedom.

12. The system of claim 1, wherein one or more of the first machine learning model, the second machine learning model, and the third machine learning model is a neural network.

13. A system for assessing movement of one or more body portions, comprising:

an inertial sensor positioned on one or more body portions;

an image sensor;

a processor communicatively coupled to the inertial sensor and the image sensor; and

a memory, coupled to the processor, configured to store program instructions, wherein, when executed by the processor, the program instructions cause the processor to perform a method comprising;

via a first machine learning model;

receiving a sensor signal using the inertial sensor, the sensor signal being indicative of movement of the one or more body portions over a time period,

analyzing the sensor signal of the one or more body portions to determine a movement of the one or more body portions, and

determining a sensor confidence level based, at least in part, on a characteristic of the sensor signal from the inertial sensor over the time period;

via a second machine learning model;

receiving a series of images using the image sensor, the series of images comprising the one or more body portions depicted therein and indicative of movement of the one or more body portions over the time period,

identifying the one or more body portions in each of the series of images,

measuring an angle of movement of the one or more body portions in at least a subset of the series of images over the time period, and

determining a vision confidence level based, at least in part, on an ability of the second machine learning model to identify the one or more body portions in the at least a subset of the series of images; and

via a third learning model;

selecting the sensor signal from the first machine learning model based on the sensor confidence level, the measured angle of movement from the second machine learning model based on the vision confidence level, or a weighted combination thereof as an input into the third machine learning model, wherein selection between the sensor signal, the measured angle of movement, and the weighted combination thereof is further based on one or more of;

a movement of two or more joints compared at a defined or predetermined time instant in the first and second machine learning models, a size of movement, a dimension of movement, a direction of movement, a speed of movement, a number of image sensors required, a background color, and a combination thereof,

analyzing the input to determine a movement pattern of the one or more body portions, and

outputting the movement pattern to a user.

14. The system of claim 13, further comprising a mobile computing device comprising the image sensor, the processor, and the memory.

15. The system of claim 13, wherein the vision confidence level is further based on one or more attributes of the identified body portions in the series of images.

16. The system of claim 15, wherein the one or more attributes comprise one or more of:

a shape of the identified one or more body portions, a relative size of the identified one or more body portions, a location in each image of the identified one or more body portions, an acutance in each image of the identified one or more body portions, a comparative length of symmetrical body portions, and a combination thereof.

17. The system of claim 13, wherein the vision confidence level is further based on a binary classification of each pixel in each of the series of images.

18. The system of claim 13, wherein the vision confidence level is further based on an expected number of body portions in each image equaling an actual number of body portions in each image.

19. The system of claim 13, wherein the second machine learning model is a convolutional neural network.

20. The system of claim 13, wherein identifying the one or more body portions in the at least a subset of the series of images comprises identifying a first white pixel as an image is analyzed from top to bottom, bottom to top, left to right, and right to left.