Activity analysis, fall detection and risk assessment systems and methods

• US 9,597,016 B2
• Filed: 01/31/2014
• Issued: 03/21/2017
• Est. Priority Date: 04/27/2012
• Status: Active Grant

• Current Assignee: Curators of the University of Missouri (University of Missouri)
• Original Assignee: Curators of the University of Missouri (University of Missouri)
• Inventors: Stone, Erik Edward, Skubic, Marjorie, Rantz, Marilyn, Popescu, Mihail
• Primary Examiner(s): CERIONI, DANIEL LEE
• Application Number:
  US14/169,508
• Publication Number:
  US 20140148733A1
• Time in Patent Office: 1,145 Days
• Field of Search: 600/595
• US Class Current: 1/1
• CPC Class Codes: A61B 5/0013 Medical image data A61B1/00011, A61B6/56, A61B8/56 take precedence
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/004 adapted for image acquisition of a particular organ or body part A61B5/0082 takes precedence; arrangements for optical scanning A61B5/0062
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1117 Fall detection
• CPC Class Codes: A61B 5/112 Gait analysis
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/742 using visual displays A61B5/339 takes precedence
• CPC Class Codes: A61B 5/746 Alarms related to a physiological condition, e.g. details of setting alarm thresholds or avoiding false alarms alarms per se G08B
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 40/67 for remote operation
• CPC Class Codes: G16H 50/30 for calculating health indices; for individual health risk assessment
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

A method for performing health risk assessments for a patient in a home or medical facility is provided. In various embodiments, the method comprises compiling depth image data from at least one depth camera associated with a particular patient, and generating at least one three-dimensional object based on the depth image data. The method additionally includes identifying a walking sequence from the at least one three-dimensional object, analyzing the walking sequence to generate one or more parameters, and performing at least one health risk assessment based on the one or more parameters to determine a health risk assessment score. The method further comprises sending an alert message to at least one caregiver when the at least one health risk assessment indicates the occurrence of an incident (i.e., the recorded depth image data) denoting that the patient has fallen or that there exists a high risk of the patient falling.

Claims

1. A method for determining whether a person has fallen or is at a high risk of falling, and notifying a caregiver when the person has fallen or is at high risk of falling, the method comprising:

acquiring depth image data of a person, as the person walks within a particular area, utilizing at least one depth camera of a risk assessment system, the at least one depth camera located in the particular area;

receiving the depth image data from the at least one depth camera by at least one processor of a computer-based remote device of the risk assessment system, wherein the depth image data comprises a plurality of frames that depict the person walking through a home environment over time, the frames comprising a plurality of pixels, the remote device located remotely from the at least one depth camera, the remote device comprising electronic memory on which an image analysis application is electronically stored, and the at least one processor structured and operable to execute the image analysis application;

segmenting, by the at least one processor, the pixels of the frames of the depth image data;

in response to the segmenting, (1) generating, by the at least one processor, a three-dimensional data object based on the analyzed depth image data, and (2) tracking by the at least one processor, the three-dimensional data object over a plurality of frames of the depth images data, wherein the tracked three-dimensional object comprises time-indexed spatial data that represents the person walking through the home environment over time;

identifying, by the at least one processor, a walking sequence from the tracked three-dimensional object, wherein the identifying step comprises;

the at least one processor determining a speed for the tracked three-dimensional data object over a time frame;

the at least one processor comparing the determined speed with a speed threshold;

in response to the comparison indicating that the determined speed is greater than the speed threshold, the at least one processor assigning a state indicative of walking to the tracked three-dimensional data object;

while the tracked three-dimensional data object is in the assigned walking state;

the at least one processor determining a walk straightness for the tracked three-dimensional data object;

the at least one processor determining a walk length for the tracked three-dimensional data object;

the at least one processor determining a walk duration for the tracked three-dimensional data object;

the at least one processor saving the tracked three-dimensional data object in memory as the identified walking sequence when (i) the determined walk straightness exceeds a straightness threshold, (ii) the determined walk length exceeds a walk length threshold, and (iii) the determined walk duration exceeds a walk duration threshold;

the at least one processor excluding from the identified walking sequence in the memory the time-indexed spatial data from the tracked three-dimensional data object corresponding to a time period where the determined walk straightness is less than the walk straightness threshold;

the at least one processor repeating the speed determining step and the comparing step for the tracked three-dimensional data object while the tracked three-dimensional data object is in the assigned walking state; and

the at least one processor assigning a state indicative of not walking to the tracked three-dimensional data object in response to a determination that the speed of the tracked three-dimensional data object in the walking state has fallen below the speed threshold;

analyzing by the at least one processor, the time-indexed spatial data from the identified walking sequence to generate one or more gait parameters;

performing, by the at least one processor, at least one health risk assessment based on the one or more gait parameters to determine a health risk assessment score for the person, the health risk assessment score indicative of a level of risk at which the person is of falling;

analyzing, by the at least one processor, the health risk assessment score to determine whether an alert incident has occurred, the alert incident indicating at least one of;

the person has fallen; and

a high risk of the person falling is present; and

generating and sending, by the at least one processor, an alert message to at least one caregiver electronic device when it is determined that the health risk assessment score indicates an alert incident has occurred.

2. The method of claim 1, wherein the caregiver electronic device comprises at least one of a smart phone, a laptop computer, a desktop computer, a computer tablet, and a television, and wherein sending the alert message comprises sending the alert message to at least one of:

the smart phone of the caregiver,

the laptop computer of the caregiver,

the desktop computer of the caregiver,

the computer tablet of the caregiver, and

the television of the caregiver.

3. The method of claim 2, wherein sending the alert message comprises sending the alert message in the form of a visual alert message.

4. The method of claim 2, wherein sending the alert message comprises sending the alert message in the form of an audible alert message.

5. The method of claim 2, wherein sending the alert message comprising sending the alert message in the form of a tactile alert message.

6. The method of claim 1, wherein sending the alert message comprises sending the alert message that includes at least one of:

confidence of the risk assessment score information, time of the alert incident occurrence information, location of the alert incident occurrence information, and presence of another person at the time of the alert incident occurrence information.

7. The method of claim 1, wherein sending the alert message comprises sending the alert message including a hyperlink to video data containing the depth image data of the determined alert incident.

8. The method of claim 7, wherein the video data can be viewed using media player software and can be rewound, replayed and fast forwarded via the media player software.

9. A risk assessment system for determining whether a subject has fallen or is at a high risk of falling, and notifying a caregiver when the subject has fallen or is at high risk of falling, the risk assessment system comprising:

at least one depth camera located in the particular area in which a subject is present; and

a computer-based remote device communicatively connected with, and located remotely from, the at least one depth camera, the remote device comprising;

at least one electronic storage device having stored thereon an image analysis application and structured and operable to (1) store a model of walk characteristics data for the subject and (2) store a plurality of walk sequence data sets in association with the subject;

a display structured and operable to provide various graphical images; and

at least one processor structured and operable to;

receive and process depth image data received from at least one depth camera to populate the electronic storage device with the walk sequence data sets associated with the subject as the subject walks within the particular area, wherein the depth image data comprises a plurality of frames that depict a space over time, the frames comprising a plurality of pixels;

process the pixels within the frames to generate and track a plurality of three dimensional objects that represent a plurality of objects that are moving within the space over time, each tracked three-dimensional data object comprising a plurality of three-dimensional points that define a spatial position of the respective tracked three-dimensional data object over time;

process the three-dimensional points of each tracked three-dimensional data object to make a plurality of determinations as whether any tracked three-dimensional data object is indicative of a subject walking;

identify a plurality of walking sequences in response to the walking determinations, each identified walking sequence corresponding to a tracked three-dimensional data object;

analyze the three-dimensional points of the tracked three-dimensional data objects corresponding to the identified walking sequences to generate data indicative of a plurality of walk characteristics for the identified walking sequences;

save a walking sequence data set for each identified walking sequence, each walking sequence data set comprising the walk characteristics data for a corresponding walking sequence;

cluster each saved walk sequence data set and compare the clustered walk sequence data sets with the stored model;

based on the comparison, determine whether any of the clustered walk sequence data sets are attributable to the subject;

in response to a determination that a clustered walk sequence data set is attributable to the subject, store that walk sequence data set in the electronic storage device in association with the subject;

perform at least one health risk assessment score for the subject based on at least one of the stored walk sequence data sets associated with the subject, the health risk assessment indicative of a level of risk at which the subject is of falling;

display, the health risk assessment;

analyze the health risk assessment and determine whether an alert incident has occurred, the alert incident indicating at least one of;

the subject has fallen; and

a high risk of the subject falling is present; and

generate and send an alert message to at least one caregiver electronic device when the health risk assessment score indicates an alert incident has occurred.

10. A risk assessment system for determining whether a subject has fallen or is at a high risk of falling, and notifying a caregiver when the subject has fallen or is at high risk of falling, the risk assessment system comprising:

at least one depth camera located in a particular area in which a subject is present, the at least one depth camera structured and operable to capture depth image data within a field of view of the at least one depth camera within the particular area; and

a computer-based remote device communicatively connected with, and located remotely from, the at least one depth camera, the remote device comprising;

at least one electronic storage device having stored thereon an image analysis application and structured and operable to (1) store a model of walk characteristics data for a the subject and (2) store a plurality of walk sequence data sets in association with the subject;

at least one processor for cooperation with the at least one depth camera and the electronic storage device, and structured and operable to;

receive depth image data from the at least one depth camera and process the depth image data to populate the electronic storage device with the walk sequence data sets associated with the subject, wherein the received depth image data comprises a plurality of frames that depict a space within the particular area over time, the frames comprising a plurality of pixels;

process the pixels within the frames to generate and track a plurality of three-dimensional data objects that represent a plurality of objects that are moving within the space over time, each tracked three-dimensional data object comprising a plurality of three-dimensional points that define a spatial position of the respective tracked three-dimensional data object over time;

process the three-dimensional points of each tracked three-dimensional data object to make a plurality of determinations as to whether any tracked three-dimensional data object is indicative of a subject walking;

identify a plurality of walking sequences in response to the walking determinations, each identified walking sequence corresponding to a tracked three-dimensional data object; and

analyze the three-dimensional points of the tracked three-dimensional data objects corresponding to the identified walking sequences to generate data indicative of a plurality of walk characteristics for the identified walking sequences;

save a walking sequence data set for each identified walking sequence, each walking sequence data set comprising the walk characteristics data for a corresponding walking sequence;

cluster each saved walk sequence data set and compare the clustered walk sequence data sets with the stored model;

based on the comparison, determine whether any of the clustered walk sequence data sets are attributable to the subject;

in response to a determination that a clustered walk sequence data set is attributable to the subject, store that walk sequence data set in the electronic storage device in association with the subject;

map at least one of the stored walk sequence data sets associated with the subject to a clinical measure indicative health risk and

generate a health risk assessment score for the subject, the health risk assessment score indicative of a level of risk at which the subject is of falling;

compare the computed health risk score with a threshold to determine whether the health risk score exceeds the threshold indicating that an alert incident has occurred, the alert incident indicating at least one of;

the subject has fallen; and

a high risk of the subject falling is present; and

display the health risk assessment score; and

generate and send an alert message to at least one caregiver electronic device when the health risk assessment score indicates an alert incident has occurred.