Framework for analysis of body camera and sensor information

• US 10,043,085 B2
• Filed: 10/25/2016
• Issued: 08/07/2018
• Est. Priority Date: 10/25/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dejewski, Donald P., Flores, Romelia H., Kwan, Hung T.
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US15/334,018
• Publication Number:
  US 20180114081A1
• Time in Patent Office: 651 Days
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/15 Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
• CPC Class Codes: G06V 40/155 use of biometric patterns for forensic purposes
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

A method, computer program product, and system for initiating a responsive action to environmental factors, include a processor monitoring biometric data and movements of an individual obtained from one or more sensors proximate to the individual, to establish and continuously update, a baseline state for the individual. The processor obtains video data and audio data from a recording device worn by the individual and contemporaneous data from the one or more sensors. The processor determines, based on the contemporaneous sensor data and the baseline state, that the individual is experiencing a state inconsistent with the baseline state. The processor analyzes the video data and the audio data to identify environmental factors contributing to the state inconsistent with the baseline state. Based on identifying the environmental factors, the processor determines a responsive action to mitigate the environmental factors and initiates the responsive action.

Claims

1. A computer-implemented method, comprising:

monitoring, by one or more processors, biometric data and movements of an individual obtained from one or more sensors proximate to the individual, to establish and continuously update, a baseline state for the individual;

obtaining, by the one or more processors, video data and audio data from a recording device worn by the individual and contemporaneous data from the one or more sensors;

determining, by the one or more processors, based on the contemporaneous sensor data and the baseline state, that the individual is experiencing a state inconsistent with the baseline state;

analyzing, by the one or more processors, the video data and the audio data to identify environmental factors contributing to the state inconsistent with the baseline state;

based on identifying the environmental factors, determining, by the one or more processors, a responsive action to mitigate the environmental factors, wherein the determining the responsive action further comprises analyzing historical data in a memory to select the responsive action based on the environmental factors, wherein the historical data comprises data gathered during prior responsive actions with similar environmental factors to the environmental factors;

initiating, by the one or more processors, the responsive action;

monitoring, by the one or more processors, the responsive action, utilizing the recording device and the sensors; and

retaining, by the one or more processors, data gathered during the responsive action, wherein the retaining comprises indexing the data gathered during the responsive action in the memory and updating the historical data with the data gathered during the responsive action.

2. The computer-implemented method of claim 1, wherein the initiating comprises:

transmitting, by the one or more processors, over a network connection, to a client, a request for the responsive action; and

configuring, by the one or more processors, the request for display in a customized graphical user interface on the client.

3. The computer-implemented method of claim 1, further comprising:

identifying, by the one or more processors, based on the video data, the audio data, and the contemporaneous data, one or more additional individuals in a given vicinity of the individual, wherein the initiating comprises transmitting, by the one or more processors, over a network connection, to one or more clients associated with the one or more additional individual, a request for the responsive action.

4. The computer-implemented method of claim 1, wherein the biometric data comprises biometric data comprises one or more of:

heart rate of the individual, body temperature of the individual, or blood pressure of the individual.

5. The computer-implemented method of claim 1, wherein the baseline state comprises a resting state and the state inconsistent with the baseline state comprises a high stress state and the determining comprises ascertaining the high stress state based on the contemporaneous data exceeding a threshold established by the baseline state.

6. The computer-implemented method of claim 1, wherein the initiating comprises:

transmitting, by the one or more processors, over a network connection, to a client, a request for the responsive action; and

configuring, by the one or more processors, the request for display in a customized graphical user interface on the client, the computer-implemented method further comprising;

based on monitoring the responsive action, configuring and displaying, in the customized graphical user interface, data obtained during the response action from the recording device and the sensors.

7. The computer-implemented method of claim 1, wherein the determining the responsive action further comprises analyzing one or more of operational procedures to select the responsive action based on the environmental factors.

8. The computer-implemented method of claim 1, wherein the contemporaneous sensor data comprises additional biometric data and additional data characterizing movements of the individual, the determining that the individual is experiencing a state inconsistent with the baseline state comprising:

analyzing, by the one or more processors, the additional biometric data to determine, based on the baseline data, if at least one biometric value in the additional biometric data is inconsistent with the baseline data; and

based on determining that the at least one biometric value in the additional biometric data is inconsistent with the baseline data, analyzing, by the one or more processors, the additional data characterizing the movements to determine if the at least one biometric value is an outlier or if the at least one biometric value is indicative of the environmental factors.

9. A computer program product comprising:

a computer readable storage medium readable by one or more processors and storing instructions for execution by the one or more processors for performing a method comprising;

monitoring, by the one or more processors, biometric data and movements of an individual obtained from one or more sensors proximate to the individual, to establish and continuously update, a baseline state for the individual;

obtaining, by the one or more processors, video data and audio data from a recording device worn by the individual and contemporaneous data from the one or more sensors;

determining, by the one or more processors, based on the contemporaneous sensor data and the baseline state, that the individual is experiencing a state inconsistent with the baseline state;

analyzing, by the one or more processors, the video data and the audio data to identify environmental factors contributing to the state inconsistent with the baseline state;

based on identifying the environmental factors, determining, by the one or more processors, a responsive action to mitigate the environmental factors, wherein the determining the responsive action further comprises analyzing historical data in a memory to select the responsive action based on the environmental factors, wherein the historical data comprises data gathered during prior responsive actions with similar environmental factors to the environmental factors;

initiating, by the one or more processors, the responsive action;

monitoring, by the one or more processors, the responsive action, utilizing the recording device and the sensors; and

retaining, by the one or more processors, data gathered during the responsive action, wherein the retaining comprises indexing the data gathered during the responsive action in the memory and updating the historical data with the data gathered during the responsive action.

10. The computer program product of claim 9, wherein the initiating comprises:

transmitting, by the one or more processors, over a network connection, to a client, a request for the responsive action; and

configuring, by the one or more processors, the request for display in a customized graphical user interface on the client.

11. The computer program product of claim 9, further comprising:

identifying, by the one or more processors, based on the video data, the audio data, and the contemporaneous data, one or more additional individuals in a given vicinity of the individual, wherein the initiating comprises transmitting, by the one or more processors, over a network connection, to one or more clients associated with the one or more additional individual, a request for the responsive action.

12. The computer program product of claim 9, wherein the biometric data comprises biometric data comprises one or more of:

heart rate of the individual, body temperature of the individual, or blood pressure of the individual.

13. The computer program product of claim 9, wherein the baseline state comprises a resting state and the state inconsistent with the baseline state comprises a high stress state and the determining comprises ascertaining the high stress state based on the contemporaneous data exceeding a threshold established by the baseline state.

14. The computer program product of claim 9, wherein the initiating comprises:

transmitting, by the one or more processors, over a network connection, to a client, a request for the responsive action; and

configuring, by the one or more processors, the request for display in a customized graphical user interface on the client, the computer-implemented method further comprising;

based on monitoring the responsive action, configuring and displaying, in the customized graphical user interface, data obtained during the response action from the recording device and the sensors.

15. The computer program product of claim 9, wherein the determining the responsive action further comprises analyzing one or more of operational procedures to select the responsive action based on the environmental factors.

16. A system comprising:

a memory;

one or more processors in communication with the memory; and

program instructions executable by the one or more processors via the memory to perform a method, the method comprising;

monitoring, by the one or more processors, biometric data and movements of an individual obtained from one or more sensors proximate to the individual, to establish and continuously update, a baseline state for the individual;

obtaining, by the one or more processors, video data and audio data from a recording device worn by the individual and contemporaneous data from the one or more sensors;

determining, by the one or more processors, based on the contemporaneous sensor data and the baseline state, that the individual is experiencing a state inconsistent with the baseline state;

analyzing, by the one or more processors, the video data and the audio data to identify environmental factors contributing to the state inconsistent with the baseline state;

based on identifying the environmental factors, determining, by the one or more processors, a responsive action to mitigate the environmental factors, wherein the determining the responsive action further comprises analyzing historical data in a memory to select the responsive action based on the environmental factors, wherein the historical data comprises data gathered during prior responsive actions with similar environmental factors to the environmental factors;

initiating, by the one or more processors, the responsive action;

monitoring, by the one or more processors, the responsive action, utilizing the recording device and the sensors; and

retaining, by the one or more processors, data gathered during the responsive action, wherein the retaining comprises indexing the data gathered during the responsive action in the memory and updating the historical data with the data gathered during the responsive action.