Anonymized video analysis methods and systems

• US 8,890,937 B2
• Filed: 06/01/2010
• Issued: 11/18/2014
• Est. Priority Date: 06/01/2009
• Status: Active Grant

• Current Assignee: Curators of the University of Missouri (University of Missouri)
• Original Assignee: Curators of the University of Missouri (University of Missouri)
• Inventors: Skubic, Marjorie, Keller, James M., Wang, Fang, Anderson, Derek T., Stone, Erik Edward, Luke, Robert H. III, Banerjee, Tanvi, Rantz, Marilyn J.
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): WALKER, JARED T
• Application Number:
  US12/791,496
• Publication Number:
  US 20100328436A1
• Time in Patent Office: 1,631 Days
• Field of Search: 348/47, 348/77, 348/135, 348/143, 340/573.1, 382/103
• US Class Current: 348/47
• CPC Class Codes: A61B 2503/08 Elderly
• CPC Class Codes: A61B 2505/07 Home care
• CPC Class Codes: A61B 2560/0242 adapted to measure environmental factors, e.g. temperature, pollution
• CPC Class Codes: A61B 5/002 Monitoring the patient using a local or closed circuit, e.g. in a room or building A61B5/0017 takes precedence
• CPC Class Codes: A61B 5/0022 Monitoring a patient using a global network, e.g. telephone networks, internet
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/021 Measuring pressure in heart or blood vessels A61B5/0205 takes precedence
• CPC Class Codes: A61B 5/024 Detecting, measuring or recording pulse rate or heart rate A61B5/0205, A61B5/021 take precedence
• CPC Class Codes: A61B 5/0816 Measuring devices for examining respiratory frequency measuring frequency of electric signals G01R23/00
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G08B 21/0423 detecting deviation from an expected pattern of behaviour or schedule
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

Methods and systems for anonymized video analysis are described. In one embodiment, a first silhouette image of a person in a living unit may be accessed. The first silhouette image may be based on a first video signal recorded by a first video camera. A second silhouette image of the person in the living unit may be accessed. The second silhouette image may be of a different view of the person than the first silhouette image. The second silhouette image may be based on a second video signal recorded by a second video camera. A three-dimensional model of the person in voxel space may be generated based on the first silhouette image, the second silhouette image, and viewing conditions of the first video camera and the second video camera. In some embodiments, information on falls, gait parameters, and other movements of the person living unit are determined. Additional methods and systems are disclosed.

Claims

1. A method comprising:

accessing a first silhouette image of a person in a living unit, the first silhouette image based on a first video signal recorded by a first video camera;

accessing a second silhouette image of the person in the living unit, the second silhouette image being of a different view of the person than the first silhouette image, the second silhouette image based on a second video signal recorded by a second video camera, the second video camera being a different video camera than the first video camera, the first video camera and the second video camera being positioned to record an area of the living unit from a different position in the living unit;

generating a three-dimensional model of the person in voxel space based on the first silhouette image, the second silhouette image, and viewing conditions of the first video camera and the second video camera;

accessing a third silhouette image of the person in the living unit, the third silhouette image based on the first video signal recorded by the first video camera, the third silhouette image associated with a different frame of the first video signal than the first silhouette image;

accessing a fourth silhouette image of the person in the living unit, the fourth silhouette image based on the second video signal recorded by the second video camera, the fourth silhouette image associated with a different frame of the second video signal than the second silhouette image;

analyzing a projection of a bottom section of the three-dimensional model of the person onto a ground plane of the living unit during the time period;

identifying a plurality of voxel intersections during a time period, a particular voxel intersection including an overlapping portion of the bottom section of the three-dimensional model of the person onto the ground plane on multiple frames, the third silhouette image being after the first silhouette image, the fourth silhouette image being after the second silhouette image;

calculating a voxel volume of the three-dimensional model of the person;

applying fuzzy set theory to the voxel volume of the three-dimensional model of the person based on an expected voxel volume of the person; and

generating a fuzzified version of the three-dimensional model of the person in the voxel space based on the first silhouette image, the second silhouette image, the viewing conditions of the first video camera and the second video camera, and application of fuzzy set theory.

2. The method of claim 1, further comprising:

extracting a plurality of features from the three-dimensional model of the person.

3. The method of claim 2, further comprising:

applying fuzzy logic to extraction of the plurality of features to identify a state of the person.

4. The method of claim 2, wherein the plurality of features include an eigen height of the three-dimensional model of the person, a centroid of the three-dimensional model of the person, and a ground plane normal similarity of the three-dimensional model of the person.

5. The method of claim 4, further comprising:

extracting a plurality of features from the three-dimensional model of the person based on the first silhouette image and the second silhouette image;

applying fuzzy logic to extraction of the plurality of features from the three-dimensional model of the person to identify a first state of the person;

extracting a plurality of features from an additional three-dimensional model of the person based on a third silhouette image of the person in the living unit and a fourth silhouette image of the person in the living unit, the third silhouette image associated with a different frame of the first video signal than the first silhouette image, the fourth silhouette image based on the second video signal recorded by the second video camera, the fourth silhouette image associated with a different frame of the second video signal than the second silhouette image;

extracting the plurality of features from the additional three-dimensional model of the person based on the third silhouette image and the fourth silhouette image;

applying the fuzzy logic to the extraction of the plurality of features from the additional three-dimensional model of the person to identify a second state of the person; and

identifying an activity of the person as a falling activity when the three-dimensional model of the person transitions from an upright state to an on-the-ground state between the first state and the second state and includes a certain transition from the three-dimensional model of the person to the additional three-dimensional model of the person for at least some of the plurality of features.

6. The method of claim 1, further comprising:

analyzing a centroid of the three-dimensional model of the person during the time period, the centroid being a particular feature of the plurality of features; and

calculating a speed of the person during the time period based on the analysis of the centroid.

7. The method of claim 1, further comprising:

analyzing a projection of a bottom section of the three-dimensional model of the person onto a ground plane of the living unit during the time period; and

calculating step length and step time during the time period based on the analysis of the projection.

8. The method of claim 1, further comprising:

forming a plurality of spatial clusters based on identification of the plurality of voxel intersections to identify a plurality of footsteps, a particular footstep of the plurality of footsteps associated with a particular spatial cluster of the plurality of spatial clusters.

9. The method of claim 8, further comprising:

determining a direction of motion of the three-dimensional model of the person based on the plurality of footsteps.

10. The method of claim 8, further comprising:

identifying a left footstep, a right footstep, a left step length, a right step length, a left step time, a right step time, and shuffling based on the plurality of footsteps.

11. The method of claim 1, further comprising:

accessing a centroid of the three-dimensional model of the person, the centroid being a particular feature of the plurality of features;

computing a trajectory of the three-dimensional model of the person over the time period;

generating a varying angle and an amplitude of the three-dimensional model of the person based on the certroid and computation of the trajectory; and

identifying lateral sway of the three-dimensional model of the person based on generation of the varying angle and the amplitude.

12. The method of claim 2, further comprising:

accessing a centroid of the three-dimensional model of the person;

determining a base of support of the three-dimensional model of the person;

generating a varying angle and an amplitude of the three-dimensional model of the person based on the certroid and the base of the support; and

identifying body sway of the three-dimensional model of the person based on generation of the varying angle and the amplitude.

13. The method of claim 1, further comprising:

extracting a first feature from the first silhouette image and the second silhouette image;

extracting a second feature from the three-dimensional model of the person; and

identifying sit to stand time based on extraction of the first feature and the second feature.

14. The method of claim 1, further comprising:

receiving a first video signal from a first video camera, the first video signal including a depiction of the person;

receiving a second video signal from a second video camera, the second video signal including a different depiction of the person, the different depiction of the person being of the person from a different view than the depiction of the person;

generating the first silhouette image of the person from the first video signal; and

generating the second silhouette image of the person from the second video signal.

15. The method of claim 1, further comprising:

generating a display of the three-dimensional model of the person, the display including a silhouetted depiction of the three-dimensional model in the living unit.

16. The method of claim 1, wherein the first video camera and the second video camera are approximately 90 degrees apart from each other in the living unit in reference to the person.

17. The method of claim 1, wherein the first silhouette image and the second silhouette image are associated with the person being in a same position in the living unit.

18. The method of claim 1, wherein the first silhouette image and the second silhouette image are associated with a similar moment in time.

19. The method of claim 1, wherein the viewing conditions include a location of the first video camera and the second video camera in the living unit, orientation of the first video camera and the second video camera, and a lensing effect of the first video camera and the second video camera.

20. The method of claim 1, further comprising:

using the three-dimensional model of the person in voxel space to monitor wellbeing of the person in the living unit.