Systems and methods for markerless tracking of subjects

• US 10,485,454 B2
• Filed: 05/21/2018
• Issued: 11/26/2019
• Est. Priority Date: 05/24/2017
• Status: Active Grant

• Current Assignee: NeuroPath SPRL
• Original Assignee: NeuroPath SPRL
• Inventors: Tas, Benoit Yvonne, Clark, Eric Michael
• Primary Examiner(s): Desire, Gregory M
• Application Number:
  US15/984,986
• Publication Number:
  US 20180338710A1
• Time in Patent Office: 554 Days
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 5/0035 adapted for acquisition of images from more than one imaging mode, e.g. combining MRI and optical tomography
• CPC Class Codes: A61B 5/0075 by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy A61B5/0071 takes precedence
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/4082 Diagnosing or monitoring movement diseases, e.g. Parkinson, Huntington or Tourette
• CPC Class Codes: A61B 5/7246 using correlation, e.g. template matching or determination of similarity
• CPC Class Codes: A61B 5/7278 Artificial waveform generation or derivation, e.g. synthesising signals from measured signals A61B5/327 takes precedence
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Markerless tracking systems and methods for markerless tracking of subjects. In one embodiment, the markerless tracking system includes an active 3D infrared camera, a memory, and an electronic processor. The electronic processor is configured to extract body motion data for a subject'"'"'s body from depth data captured by the active 3D infrared camera. The electronic processor is also configured to detect movements of the subject'"'"'s body using the body motion data. The electronic processor is further configured to determine attributes for the movements of the subject'"'"'s body using the body motion data. The electronic processor is also configured to assign a rating by comparing the determined attributes with a plurality of benchmarks included in a pre-stored movement profile in the memory. The electronic processor is further configured to create a session record for the subject. The session record includes the body motion data, the determined attributes, and the assigned rating.

Claims

1. A system for markerless tracking of a subject performing a motor exercise, comprising:

an active 3D infrared camera for capturing depth data of a body of the subject during the motor exercise;

a memory for pre-storing a movement profile including a plurality of benchmarks; and

an electronic processor configured to

extract body motion data for the subject'"'"'s body from the captured depth data, the body motion data including a set of 3D coordinates for a plurality of body joints for each frame of the captured depth data,

detect movements of the subject'"'"'s body using the body motion data,

determine a set of attributes for the detected movements of the subject'"'"'s body using the body motion data,

assign a rating for the motor exercise by comparing the set of attributes for the detected movements of the subject'"'"'s body with the plurality of benchmarks included in the pre-stored movement profile, and

create a session record for the subject including the body motion data, the set of attributes for the detected movements of the subject'"'"'s body, and the assigned rating for the motor exercise,

wherein the set of attributes is a first set of attributes, wherein the electronic processor is further configured to

retrieve a second set of attributes from the memory, the second set of attributes having been stored in the memory prior to determining the first set of attributes,

determine a difference between the first set of attributes and the second set of attributes

determine whether the subject is in an ON functional state or in an OFF functional state based on the determined difference between the first set of attributes and the second set of attributes, and

add subject state data to the session record for the subject, the subject state data indicating the determined functional state of the subject.

2. The system of claim 1, wherein the set of attributes for the detected movements of the subject'"'"'s body including at least one selected from a group consisting of speed, distance, and amplitude.

3. The system of claim 1, further comprising an RGB camera for capturing a sequence of color images of the subject performing the motor exercise, wherein the RGB camera is positioned at a predetermined location relative to the location of the active 3D infrared camera, wherein the RGB camera and the active 3D infrared camera have overlapping fields of view of the subject, wherein for each color image in the sequence of color images, the electronic processor is configured to

calculate 2D coordinates in the color image for each of the plurality of body joints using the body motion data and prior knowledge of the locations and the fields of view of the cameras,

alter the color image by superimposing a skeleton on the color image, the skeleton including a visual indicator at the calculated 2D coordinates for each of the plurality of body joints of the subject'"'"'s body, and

add the altered color image to the session record for the subject.

4. The system of claim 1, wherein a frame rate of the captured depth data is at least thirty frames per second.

5. The system of claim 1, wherein the motor exercise including at least one motor exercise of the unified Parkinson'"'"'s disease rating scale.

6. The system of claim 1, wherein the rating is a first rating, wherein the system further comprising a user interface, wherein the electronic processor is further configured to

receive a second rating for the motor exercise via the user interface,

determine a difference between the first rating for the motor exercise and the second rating for the motor exercise, and

adjust at least one of the plurality of benchmarks included in the pre-stored movement profile based on the determined difference between the first rating for the motor exercise and the second rating for the motor exercise.

7. The system of claim 1, wherein the electronic processor is further configured to

extract facial motion data for a face of the subject from the captured depth data, the facial motion data including a set of 3D coordinates for a plurality of facial points for each frame of the captured depth data, and

add the facial motion data to the session record for the subject.

8. The system of claim 1, wherein the electronic processor is further configured to

extract facial motion data for a face of the subject from the captured depth data, the facial motion data including a set of 3D coordinates for a plurality of facial points for each frame of the captured depth data,

determine identification data for the subject based on the facial motion data, and

add the determined identification data for the subject to the session record for the subject.

9. The system of claim 1, wherein each of the plurality of benchmarks define a set of conditions associated with a mobility level, wherein the electronic processor assigning the rating for the motor exercise by comparing the set of attributes for the detected movements of the subject'"'"'s body with the plurality of benchmarks included in the pre-stored movement profile including

selecting a lowest mobility level with at least one condition met by the set of attributes for the detected movements of the subject'"'"'s body, and

assigning the rating for the motor exercise to the selected mobility level.

10. A system for markerless tracking of subjects located in a space, comprising:

an active 3D infrared camera for capturing depth data in the space; and

an electronic processor configured to

extract body motion data for a body of a first subject located in the space from the captured depth data, the body motion data including a set of 3D coordinates for a plurality of body joints for each frame of the captured depth data,

detect movements of the first subject'"'"'s body using the body motion data,

determine a set of attributes for the detected movements of the first subject'"'"'s body using the body motion data,

extract facial motion data for a face of the first subject from the captured depth data, the facial motion data including a set of 3D coordinates for a plurality of facial points for each frame of the captured depth data,

determine identification data for the first subject using the facial motion data, and

create a session record for the first subject including the body motion data, the detected movements of the first subject'"'"'s body, the set of attributes, and the identification data for the first subject,

wherein the electronic processor is further configured to

extract second body motion data for a second body of a second subject located in the space from the captured depth data, the second body motion data including a set of 3D coordinates for a second plurality of body joints for each frame of the captured depth data,

detect movements of the second subject'"'"'s body using the second body motion data,

determine a second set of attributes for the detected movements of the second subject'"'"'s body based on the second body motion data,

extract second facial motion data for a second face of the second subject face from the captured depth data, the second facial motion data including a set of 3D coordinates for a second plurality of facial points for each frame of the captured depth data,

determine identification data for the second subject based on the second facial motion data, and

create a second session record for the second subject including the second body motion data, the detected movements of the second subject'"'"'s body, the second set of attributes, and the identification data for the second subject.

11. The system of claim 10, wherein the set of attributes for the detected movements of the first subject'"'"'s body including at least one selected from a group consisting of speed, distance, and amplitude.

12. The system of claim 10, further comprising an RGB camera for capturing a sequence of color images of the first subject, wherein the RGB camera is positioned at a predetermined location relative to the location of the active 3D infrared camera, wherein the RGB camera and the active 3D infrared camera have overlapping fields of view of the first subject, wherein for each color image in the sequence of color images, the electronic processor is configured to

calculate 2D coordinates in the color image for each of the plurality of body joints using the body motion data and prior knowledge of the locations and the fields of view of the cameras,

alter the color image by superimposing a skeleton on the color image, the skeleton including a visual indicator at the calculated 2D coordinates for each of the plurality of body joints, and

add the altered color image to the session record for the first subject.

13. The system of claim 10, wherein a frame rate of the captured depth data is at least thirty frames per second.

14. A method for markerless tracking of a subject performing a motor exercise, comprising:

pre-storing a movement profile in a memory, the pre-stored movement profile including a plurality of benchmarks;

capturing, with an active 3D infrared camera, depth data of a body of the subject during the motor exercise;

extracting, with an electronic processor, body motion data for the subject'"'"'s body from the captured depth data, the body motion data including a set of 3D coordinates for a plurality of body joints for each frame of the captured depth data;

detecting, with the electronic processor, movements of the subject'"'"'s body using the body motion data;

determining, with the electronic processor, a set of attributes for the detected movements of the subject'"'"'s body using the body motion data;

assigning, with the electronic processor, a rating for the motor exercise by comparing the set of attributes for the detected movements of the subject'"'"'s body with the plurality of benchmarks included in the pre-stored movement profile; and

creating, with the electronic processor, a session record for the subject including the body motion data, the set of attributes for the detected movements of the subject'"'"'s body, and the assigned rating for the motor exercise,

wherein the set of attributes is a first set of attributes, wherein the method further comprising

retrieving a second set of attributes from the memory, the second set of attributes having been stored prior to determining the first set of attributes;

determining, with the electronic processor, a difference between the first set of attributes and the second set of attributes;

determining, with the electronic processor, whether the subject is in an ON functional state or in an OFF functional state based on the determined difference between the first set of attributes and the second set of attributes; and

adding, with the electronic processor, subject state data to the session record for the subject, the subject state data indicating the determined functional state of the subject.

15. The method of claim 14, wherein the set of attributes for the detected movements of the subject'"'"'s body including at least one selected from a group consisting of speed, distance, and amplitude.

16. The method of claim 14, wherein the motor exercise including at least one motor exercise of the unified Parkinson'"'"'s disease rating scale.

17. The method of claim 14, wherein each of the plurality of benchmarks define a set of conditions associated with a mobility level, wherein assigning the rating for the motor exercise by comparing the set of attributes for the detected movements of the subject'"'"'s body with the plurality of benchmarks included in the pre-stored movement profile including

selecting a lowest mobility level with at least one condition met by the set of attributes for the detected movements of the subject'"'"'s body, and

assigning the rating for the motor exercise to the selected mobility level.