Predicting adverse behaviors of others within an environment based on a 3D captured image stream

• US 9,208,678 B2
• Filed: 02/09/2012
• Issued: 12/08/2015
• Est. Priority Date: 01/12/2007
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Albertson, Jacob C., Arnold, Kenneth C., Goldman, Steven D., Paolini, Michael A., Sessa, Anthony J.
• Primary Examiner(s): Murray, Daniel C
• Application Number:
  US13/370,004
• Publication Number:
  US 20120140042A1
• Time in Patent Office: 1,398 Days
• Field of Search: 348/152, 348/143, 348/161, 340/506, 340/573.1, 382/103, 382/107, 382/190, 706/14
• US Class Current: 1/1
• CPC Class Codes: G01V 5/20 Detecting prohibited goods, e.g. weapons, explosives, hazardous substances, contraband or smuggled objects
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: Y10S 706/933 Law, law enforcement, or government

Abstract

A computer-implemented method, system, and program includes a behavior processing system for capturing a three-dimensional movement of a monitored user within a particular environment monitored by a supervising user, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the monitored user. The behavior processing system identifies a three-dimensional object properties stream using the captured movement. The behavior processing system identifies a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with multiple adverse behavior definitions. In response to identifying the particular defined adverse behavior from among the multiple adverse behavior definitions, the behavior processing system triggers a warning system to notify the supervising user about the particular defined adverse behavior of the monitored user through an output interface only detectable by the supervising user.

Claims

1. A computer-implemented method for warning a supervising user of behaviors of a monitored user representative of adverse behavior, comprising:

monitoring a current location of a supervising user, wherein the supervising user is actively mobile;

identifying a particular type of physical environment from among the plurality of types of physical environments comprising the current location;

responsive to detecting the supervising user conducting law enforcement activity within the particular type of environment comprising a law enforcement environment, triggering at least one image capture device within the particular environment to capture the three-dimensional movement of a monitored user within the particular environment to capture a focus area surrounding the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

identifying a three-dimensional object properties stream using the captured movement;

identifying a particular type of the interface accessible to the supervising user from among a plurality of types of interfaces;

accessing a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database and for at least one adverse behavior definition specified for the particular type of interface in the behavior database from among a plurality of adverse behavior definitions each specified for a separate one of the plurality of types of interfaces;

identifying a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, notifying the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface of the particular type of interface only detectable by the supervising user by;

sending the particular defined adverse behavior to a warning, system for controlling warning signals output to the supervising user;

translating, at the warning system, the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement; and

controlling output of the feedback control signal to at least one tactile feedback device detectable b the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

capturing a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface;

predicting whether the response indicates that the particular defined adverse behavior is correctly identified;

prompting the supervising user, within the interface, to indicate whether the prediction of whether the response indicates that the particular defined adverse behavior is correctly identified is correct; and

responsive to the supervising user indicating the prediction is correct, adjusting at least one definition in the selection of adverse behavior definitions specified for the particular type of environment and the particular type of interface in the behavior database to reflect the response.

2. The method according to claim 1, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.

3. The method according to claim 1, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using at least one stereoscopic image device and at least one sensor enabled device to detect at least one additional characteristic of a detected object in the three-dimensional movement.

4. The method according to claim 1, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse.

5. The method according to claim 1, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of adverse behavior definitions further comprises identifying from the three-dimensional object properties stream one of a suspicious activity by the monitored user, an impaired condition of the monitored user, a concealed weapon carried by the monitored user, an aggressive stance by the monitored user, an angered facial expression of the monitored user, and a nervous condition of the monitored user.

6. A computer system for warning a supervising user of behaviors of a monitored user representative of adverse behavior, comprising:

one or more processors;

a behavior system for execution by at least one of the one or more processors and operative to monitor a current location of a supervising user, wherein the supervising user is actively mobile;

the behavior system operative to identify a particular type of physical environment from among the plurality of types of physical environments comprising the current location;

the behavior system, responsive to detecting the supervising user conducting law enforcement activity within the particular type of environment comprising a law enforcement environment, operative to trigger at least one image capture device within the particular environment to capture the three-dimensional movement of a monitored user within the particular environment to capture a focus area surrounding the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

the behavior system operative to identify a three-dimensional object properties stream using the captured movement;

the behavior system operative to identify a particular type of the interface accessible to the supervising user from among a plurality of types of interfaces;

the behavior system operative to access a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database and for at least one adverse behavior definition specified for the particular type of interface in the behavior database from among a plurality of adverse behavior definitions each specified for a separate one of the plurality of types of interfaces;

the behavior system operative to identify a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

the behavior system, in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, operative to notify the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface of the particular type of interface only detectable by the supervising user by;

sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user;

translating, at the warning system, the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement; and

controlling output of the feedback control signal to at least one tactile feedback device detectable b the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

the behavior system operative to capture a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface;

the behavior system operative to predict whether the response indicates that the particular defined adverse behavior is correctly identified;

the behavior system operative to prompt the supervising user, within the interface, to indicate whether the prediction of whether the response indicates that the particular defined adverse behavior is correctly identified is correct; and

the behavior system, responsive to the supervising user indicating the prediction is correct, operative to adjust at least one definition in the selection of adverse behavior definitions specified for the particular type of environment and the particular type of interface in the behavior database to reflect the response.

7. The system according to claim 6, further comprising:

the behavior system operative to capture the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.

8. The system according to claim 6, further comprising:

the behavior system operative to capture the three-dimensional movement using at least one stereoscopic image device and at least one sensor enabled device to detect at least one additional characteristic of a detected object in the three-dimensional movement.

9. The system according to claim 6, further comprising:

the behavior system operative to calculate a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse.

10. The system according to claim 6, further comprising:

the behavior system operative to identify from the three-dimensional object properties stream one of a suspicious activity by the monitored user, an impaired condition of the monitored user, a concealed weapon carried by the monitored user, an aggressive stance by the monitored user, an angered facial expression of the monitored user, and a nervous condition of the monitored user.

11. A computer-implemented method for warning a supervising user of behaviors of a monitored user representative of adverse behavior, comprising:

capturing a three-dimensional movement of a monitored user within a particular environment monitored by a supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

identifying a three-dimensional object properties stream using the captured movement;

accessing a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

identifying a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, notifying the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface of the particular type of interface only detectable by the supervising user;

sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user;

translating the particular defined adverse behavior into an image layer specifying a transparent overlay image and a position indicator mapping a position of the transparent overlay image to the particular defined adverse behavior within the captured three-dimensional movement for output to an image overlay controller of a display device for controlling overlay of the image layer over output of the captured three-dimensional movement in the interface to graphically indicate the warning signals for the particular defined adverse behavior to a supervising user at the interface that is undetectable by the monitored user;

translating, at the warning system, the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement; and

controlling output of the feedback control signal to at least one tactile feedback device detectable b the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

capturing a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface;

predicting whether the response indicates that the particular defined adverse behavior is correctly identified;

prompting the supervising user, within the interface, to indicate whether the prediction of whether the response indicates that the particular defined adverse behavior is correctly identified is correct; and

responsive to the supervising user indicating the prediction is correct, adjusting at least one definition in the selection of adverse behavior definitions specified for the particular type of environment and the particular type of interface in the behavior database to reflect the response.

12. The method according to claim 11, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.

13. The method according to claim 11, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using at least one stereoscopic image device and at least one sensor enabled device to detect at least one additional characteristic of a detected object in the three-dimensional movement.

14. The method according to claim 11, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse.

15. The method according to claim 11, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of adverse behavior definitions further comprises identifying from the three-dimensional object properties stream one of a suspicious activity by the monitored user, an impaired condition of the monitored user, a concealed weapon carried by the monitored user, an aggressive stance by the monitored user, an angered facial expression of the monitored user, and a nervous condition of the monitored user.