Activity analysis, fall detection and risk assessment systems and methods

• US 9,408,561 B2
• Filed: 04/26/2013
• Issued: 08/09/2016
• Est. Priority Date: 04/27/2012
• Status: Active Grant

• Current Assignee: Curators of the University of Missouri (University of Missouri)
• Original Assignee: Curators of the University of Missouri (University of Missouri)
• Inventors: Stone, Erik Edward, Skubic, Marjorie
• Primary Examiner(s): Dougherty, Sean
• Assistant Examiner(s): CERIONI, DANIEL LEE
• Application Number:
  US13/871,816
• Publication Number:
  US 20130289449A1
• Time in Patent Office: 1,201 Days
• Field of Search: 600/595
• US Class Current: 1/1
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/004 adapted for image acquisition of a particular organ or body part A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062
• CPC Class Codes: A61B 5/112 Gait analysis

Abstract

Aspects of the present disclosure include methods and corresponding systems for performing health risk assessments for a patient in the home environment. In various aspects, depth image data for a person may be obtained and subsequently processed to generate one or more parameters, such as temporal and spatial gait parameters. Subsequently, the generated parameters may be processed with other medical information related to the patient, such as electronic health records, to perform various health risk assessments.

Claims

1. A method comprising:

receiving, by at least one processor, depth image data from at least one depth camera, wherein the depth image data comprises a plurality of frames that depict a person walking through a home environment over time, the frames comprising a plurality of pixels;

performing, by the at least one processor, segmentation on the pixels of the frames;

in response to the segmentation, (1) generating, by the at least one processor, a three-dimensional (3D) data object based on the depth image data, and (2) tracking, by the at least one processor, the 3D data object over a plurality of frames of the depth image data, wherein the tracked 3D data object comprises time-indexed spatial data that represents the person walking through the home environment over time;

identifying, by the at least one processor, a walking sequence from the tracked 3D data object, wherein the identifying step comprises;

the at least one processor determining a speed for the tracked 3D data object over a time frame;

the at least one processor comparing the determined speed with a speed threshold;

in response to the comparison indicating that the determined speed is greater than the speed threshold, the at least one processor assigning a state indicative of walking to the tracked 3D data object;

while the tracked 3D data object is in the assigned walking state;

the at least one processor determining a walk straightness for the tracked 3D data object;

the at least one processor determining a walk length for the tracked 3D data object;

the at least one processor determining a walk duration for the tracked 3D data object;

the at least one processor saving the tracked 3D data object in memory as the identified walking sequence when (i) the determined walk straightness exceeds a straightness threshold, (ii) the determined walk length exceeds a walk length threshold, and (iii) the determined walk duration exceed a walk duration threshold;

the at least one processor excluding from the identified walking sequence in the memory the time-indexed spatial data from the tracked 3D data object corresponding to a time period where the determined walk straightness is less than the walk straightness threshold;

the at least one processor repeating the speed determining step and the comparing step for the tracked 3D data object while the tracked 3D data object is in the assigned walking state; and

the at least one processor assigning a state indicative of not walking to the tracked 3D data object in response to a determination that the speed of the tracked 3D data object in the walking state has fallen below the speed threshold;

analyzing, by the at least one processor, the time-indexed spatial data from the identified walking sequence to generate one or more gait parameters; and

performing, by the at least one processor, at least one health risk assessment based on the one or more gait parameters to determine a health risk assessment score for the person.

2. The method of claim 1, wherein generating the 3D data object comprises:

for each frame within the plurality of frames, the at least one processor (1) segmenting pixels within that frame into background pixels and foreground pixels according to at least one background model, (2) converting the

foreground pixels into a set of three-dimensional 3D points, (3) segmenting the set of 3D points to generate

at least one 3D object for tracking, the generated 3D data object being representative of an object within the space and comprising a plurality of 3D points that define a spatial position of the represented object, (4) maintaining a list that identifies each 3D data object for tracking, (5) comparing each generated 3D data object with each tracked 3D data object on the list, (6) for each generated 3D data object that matches a tracked 3D data object on the list, updating the matching tracked 3D data object based on the matching generated 3D data object, wherein the updated tracked 3D data object comprises a plurality of

     3) points that define the spatial position of the represented object over time, and (7) for each generated 3D data object that does not match any tracked 3D data object on the list, adding the non-matching 3D data object to the list as a new 3D data object for tracking.

3. The method of claim 1, wherein the depth image data comprises three-dimensional motion tracking data captured at thirty frames per second.

4. The method of claim 1, wherein performing the health risk assessment comprises mapping the one or more parameters to a standard clinical measure to generate the health risk assessment score.

5. The method of claim 1, wherein the at least one depth image camera comprises a Microsoft Kinect.

6. The method of claim 1, wherein the one or more gait parameters comprises at least one of walking speed, stride time, and stride length.

7. A system comprising:

a memory configured to (1) store a model of walk characteristics data for a person of interest and (2) store a plurality of walk sequence data sets in association with the person of interest; and

at least one processor for cooperation with the memory, the at least one processor configured to receive and process the depth image data to populate the memory with the walk sequence data sets associated with the person of interest;

wherein the at least one processor is further configured to;

receive depth image data from at least one depth camera, wherein the depth image data comprises a plurality of frames that depict a space over time, the frames comprising a plurality of pixels;

process the pixels within the frames to generate and track a plurality of three-dimensional (3D) data objects that represent a plurality of objects that are moving within the space over time, each tracked 3D data object comprising a plurality of 3D points that define a spatial position of its represented object over time;

process the 3D points of each tracked 3D data object to make a plurality of determinations as whether any tracked 3D data object is indicative of a person walking;

identify a plurality of walking sequences in response to the walking determinations, each identified walking sequence corresponding to a tracked 3D data object;

analyze the 3D points of the tracked 3D data objects corresponding to the identified walking sequences to generate data indicative of a plurality of walk characteristics for the identified walking sequences;

save a walking sequence data set for each identified walking sequence, each walking sequence data set comprising the walk characteristics data for its corresponding walking sequence;

cluster each saved walk sequence data set and compare the clustered walk sequence data sets with the stored model;

based on the comparison, determine whether any of the clustered walk sequence data sets are attributable to the person of interest;

in response to a determination that a clustered walk sequence data set is attributable to the person of interest, store that walk sequence data set in the memory in association with the person of interest;

perform at least one health risk assessment based on at least one of the stored walk sequence data sets associated with the person of interest; and

communicate a result of the health risk assessment for display.

8. The system of claim of claim 7, wherein, as part of the pixel processing operation, the at least one processor is further configured to:

for each frame within the plurality of frames, (1) segment pixels within that frame into background pixels and foreground pixels according to at least one background model, (2) convert the

foreground pixels into a set of three-dimensional (3D) points, (3) segment the set of 3D points to generate

at least one 3D data object for tracking, the generated 3D data object being representative of an object within the space and comprising a plurality of 3D points that define a spatial position of the represented object, (4) maintain a list that identifies each 3D data object for tracking, (5) compare each generated 3D data object with each tracked 3D data object on the list, (6) for each generated 3D data object that matches a tracked 3D data object on the list, update the matching tracked 3D data object based on the matching generated 3D data object, wherein the updated tracked 3D data object comprises a plurality of 3D points that define the spatial position of the represented object over time, and (7) for each generated 3D data object that does not match any tracked 3D data object on the list, add the non-matching 3D data object to the list as a new 3D data object for tracking.

9. The system of claim 8, wherein the depth image data comprises three-dimensional motion tracking data captured at thirty frames per second.

10. The system of claim 8, wherein the at least one processor, as part of the health risk assessment, is configured to map the walk characteristics data for at least one of the stored walking sequence data sets associated with the person of interest to a standard clinical measure to generate a health risk assessment score.

11. The system of claim 8, wherein the at least one depth image camera comprises a Microsoft Kinect.

12. The system of claim 8, wherein the walk characteristics data comprises at least one of walking speed, stride time, and stride length.

20. The system of claim 7, wherein the at least one processor, as part of the 3D points processing operation and the walking sequences identification operation, is further configured to, for each of a plurality of the tracked 3D data objects:

determine a speed for the tracked 3D object over a time frame;

compare the determined speed with a speed threshold;

in response to the speed comparison indicating that the determined speed is greater than the speed threshold, assign a state indicative of walking to the tracked 3D data object

while the tracked 3D data object is in the assigned walking state;

determine a walk straightness for the tracked 3D data object;

determine a walk length for the tracked 3D data object;

determine a walk duration for the tracked 3D data object;

save the tracked 3D data object in the memory as an identified walking sequence when (i) the determined walk straightness exceeds a straightness threshold, (ii) the determined walk length exceeds a walk length threshold, and (iii) the determined walk duration exceeds a walk duration threshold;

exclude from the identified walking sequence in the memory the 3D points from the tracked 3D object corresponding to a time period where the determined walk straightness is less than the walk straightness threshold;

repeat the speed determination operation and the speed comparison operation for the tracked 3D data object while the tracked 3D data object is in the assigned walking state; and

assign a state indicative of not walking to the tracked 3D data object in response to a determination that the speed of the tracked 3D data object in the walking state has fallen below the speed threshold.

13. A system comprising:

at least one depth camera configured to capture depth image data of an area within a field of view of the at least one depth camera;

a memory configured to (1) store a model of walk characteristics data for a person of interest and (2) store a plurality of walk sequence data sets in association with the person of interest;

at least one processor for cooperation with the at least one depth camera and the memory, the at least one processor configured to receive and process the depth image data to populate the memory with the walk sequence data sets associated with the person of interest;

wherein the at least one processor is further configured to;

receive depth image data from at least one depth camera, wherein the received depth image data comprises a plurality of frames that depict the space over time, the frames comprising a plurality of pixels;

process the pixels within the frames to generate and track a plurality of three-dimensional (3D) data objects that represent a plurality of objects that are moving within the space over time, each tracked 3D data object comprising a plurality of 3D points that define a spatial position of its represented object over time;

process the 3D points of each tracked 3D data object to make a plurality of determinations as whether any tracked 3D data object is indicative of a person walking;

identify a plurality of walking sequences in response to the walking determinations, each identified walking sequence corresponding to a tracked 3D data object;

analyze the 3D points of the tracked 3D data objects corresponding to the identified walking sequences to generate data indicative of a plurality of walk characteristics for the identified walking sequences;

save a walking sequence data set for each identified walking sequence, each walking sequence data set comprising the walk characteristics data for its corresponding walking sequence;

cluster each saved walk sequence data set and compare the clustered walk sequence data sets with the stored model;

based on the comparison, determine whether any of the clustered walk sequence data sets are attributable to the person of interest;

in response to a determination that a clustered walk sequence data set is attributable to the person of interest, store that walk sequence data set in the memory in association with the person of interest;

map at least one of the stored walk sequence data sets associated with the person of interest to a clinical measure indicative of a fall risk to thereby compute a fall risk score for the person of interest;

compare the compound fall risk score with a threshold; and

in response to the fall risk score comparison resulting in a determination that the computed fall risk score exceeds the threshold, generate an alert for display that is indicative of a high fall risk for the person of interest.

14. The system of claim of claim 13, wherein, as part of the pixel processing operation, the at least one processor is further configured to:

for each frame within the plurality of frames, (1) segment pixels within that frame into background pixels and foreground pixels according to at least one background model, (2) convert the

foreground pixels into a set of three-dimensional (3D) points, (3) segment the set of 3D points to generate

at least one 3D data object for tracking, the generated 3D data object being representative of an object within the space and comprising a plurality of 3D points that define a spatial position of the represented object, (4) maintain a list that identifies each 3D data object for tracking, (5) compare each generated 3D data object with each tracked 3D data object on the list, (6) for each generated 3D data object that matches a tracked 3D data object on the list, update the matching tracked 3D data object based on the matching generated 3D data object, wherein the updated tracked 3D data object comprises a plurality of 3D points that define the spatial position of the represented object over time, and (7) for each generated 3D data object that does not match any tracked 3D data object on the list, add the non-matching 3D data object to the list as a new 3D data object for tracking.

15. The system of claim 13, wherein the depth image data comprises three-dimensional motion tracking data captured at thirty frames per second.

16. The system of claim 13, wherein the clinical measure comprises a Timed-up-and-Go (TUG) test such that the computed fall risk score comprises a TUG time for the person of interest.

17. The system of claim 13, wherein the at least one depth image camera comprises a Microsoft Kinect.

18. The system of claim 13, wherein the walk characteristics data comprises at least one of walking speed, stride time, and stride length.

19. The system of claim 13, wherein the at least one processor, as part of the 3D points processing operation and the walking sequences identification operation, is further configured to, for each of a plurality of the tracked 3D data objects:

determine a speed for the tracked 3D object over a time frame;

compare the determined speed with a speed threshold;

in response to the speed comparison indicating that the determined speed is greater than the speed threshold, assign a state indicative of walking to the tracked 3D data object

while the tracked 3D data object is in the assigned walking state;

determine a walk straightness for the tracked 3D data object;

determine a walk length for the tracked 3D data object;

determine a walk duration for the tracked 3D data object;

save the tracked 3D data object in the memory as an identified walking sequence when (i) the determined walk straightness exceeds a straightness threshold, (ii) the determined walk length exceeds a walk length threshold, and (iii) the determined walk duration exceeds a walk duration threshold;

exclude from the identified walking sequence in the memory the 3D points from the tracked 3D object corresponding to a time period where the determined walk straightness is less than the walk straightness threshold;

repeat the speed determination operation and the speed comparison operation for the tracked 3D data object while the tracked 3D data object is in the assigned walking state; and

assign a state indicative of not walking to the tracked 3D data object in response to a determination that the speed of the tracked 3D data object in the walking state has fallen below the speed threshold.