Warning a user about adverse behaviors of others within an environment based on a 3D captured image stream

• US 8,269,834 B2
• Filed: 01/12/2007
• Issued: 09/18/2012
• Est. Priority Date: 01/12/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Albertson, Jacob C., Arnold, Kenneth C., Goldman, Steven D., Paolini, Michael A., Sessa, Anthony J.
• Primary Examiner(s): Dollinger, Tonia L
• Assistant Examiner(s): Murray, Daniel C
• Application Number:
  US11/622,676
• Publication Number:
  US 20080169929A1
• Time in Patent Office: 2,076 Days
• Field of Search: 340/573.1, 340/506, 348/152, 348/143, 348/161, 382/103, 382/107, 382/190, 706/14
• US Class Current: 348/152
• CPC Class Codes: G01V 5/20 Detecting prohibited goods, e.g. weapons, explosives, hazardous substances, contraband or smuggled objects
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: Y10S 706/933 Law, law enforcement, or government

Abstract

A computer-implemented method, system, and program includes a behavior processing system for capturing a three-dimensional movement of a monitored user within a particular environment monitored by a supervising user, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the monitored user. The behavior processing system identifies a three-dimensional object properties stream using the captured movement. The behavior processing system identifies a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with multiple adverse behavior definitions. In response to identifying the particular defined adverse behavior from among the multiple adverse behavior definitions, the behavior processing system triggers a warning system to notify the supervising user about the particular defined adverse behavior of the monitored user through an output interface only detectable by the supervising user.

Claims

1. A computer-implemented method for warning a supervising user of behaviors of a monitored user representative of adverse behavior, comprising:

capturing a three-dimensional movement of a monitored user within a particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

identifying a three-dimensional object properties stream using the captured movement;

accessing a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

identifying a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, notifying the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user, wherein the warning system translates the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement, wherein the warning system controls output of the feedback control signal to at least one tactile feedback device detectable by the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

capturing a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface, wherein the response indicates whether the particular defined adverse behavior is correctly identified; and

adjusting at least one definition in the selection of adverse behavior definitions specified for the particular environment in the behavior database to reflect the response.

2. The method according to claim 1, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.

3. The method according to claim 1, wherein capturing a three-dimensional movement of the monitored user further comprises capturing the three-dimensional movement using at least one stereoscopic image device and at least one sensor enabled device to detect at least one additional characteristic of a detected object in the three-dimensional movement.

4. The method according to claim 1, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream further comprises identifying at least one of body movement of the monitored user comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the monitored user within the three-dimensional object properties stream matching at least one of the plurality of adverse behavior definitions.

5. The method according to claim 1, further comprising:

monitoring a current location of a supervising user;

identifying the particular environment from among the plurality of environments comprising the current location;

responsive to detecting the supervising user conducting law enforcement activity within the particular environment comprising a law enforcement environment in which the supervising user is conducting at least one of a traffic stop, a security checkpoint, and an interrogation, triggering the at least one image capture device within the particular environment to capture the three-dimensional movement of the monitored user within the particular environment to capture a focus area surrounding the supervising user;

identifying a particular type of the interface accessible to the supervising user from among a plurality of types of interfaces; and

accessing the selection of adverse behavior definitions further comprising at least one adverse behavior definition specified for the particular type of interface in the behavior database from among a plurality of adverse behavior definitions each specified for a separate one of the plurality of types of interfaces.

6. The method according to claim 1, wherein identifying a particular defined adverse behavior representing the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of adverse behavior definitions further comprises identifying from the three-dimensional object properties stream one of a suspicious activity by the monitored user, an impaired condition of the monitored user, a concealed weapon carried by the monitored user, an aggressive stance by the monitored user, an angered facial expression of the monitored user, and a nervous condition of the monitored user.

7. The method according to claim 1, wherein notifying a supervising user for the particular environment about the particular defined adverse behavior of the monitored user further comprises:

sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user; and

translating, at the warning system, the particular defined adverse behavior into an image layer specifying a transparent overlay image and a position indicator mapping a position of the transparent overlay image to the particular defined adverse behavior within the captured three-dimensional movement for output to an image overlay controller of a display device for controlling overlay of the image layer over output of the captured three-dimensional movement to graphically indicate the warning signals to a supervising user that are undetectable by the monitored user.

8. A system for warning a supervising user of behaviors of a monitored user representative of adverse behavior, comprising:

one or more processors;

a behavior system, for execution by at least one of said one or more processors and operative to capture a three-dimensional movement of a monitored user within a particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

the behavior system operative to identify a three-dimensional object properties stream using the captured movement;

the behavior system operative to access a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

the behavior system operative to identify a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

the behavior system operative to transmit at least one warning signal for notifying a supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user, wherein the warning system translates the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement, wherein the warning system controls output of the feedback control signal to at least one tactile feedback device detectable by the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

the behavior system operative to capture a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface, wherein the response indicates whether the particular defined adverse behavior is correctly identified; and

the behavior system operative to adjust at least one definition in the selection of adverse behavior definitions specified for the particular environment in the behavior database to reflect the response.

9. The system according to claim 8, wherein the behavior system operative to capture a three-dimensional movement of the monitored user further comprises the behavior system operative to capture the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.

10. The system according to claim 8, wherein the behavior system operative to identify a particular defined adverse behavior representing the three-dimensional object properties stream further comprises the behavior system operative to identify at least one of body movement of the monitored user comprising a fine granularity movement of at least one of a facial expression, an eye movement, a muscle contraction, and change in skin surface characteristics of the monitored user within the three-dimensional object properties stream matching at least one of the plurality of adverse behavior definitions.

11. The system according to claim 10, further comprising:

the behavior system operative to monitor a current location of a supervising user;

the behavior system operative to identify the particular environment from among the plurality of environments comprising the current location;

the behavior system, responsive to detecting the supervising user conducting law enforcement activity within the particular environment comprising a law enforcement environment in which the supervising user is conducting at least one of a traffic stop, a security checkpoint, and an interrogation, operative to trigger the at least one image capture device within the particular environment to capture the three-dimensional movement of the monitored user within the particular environment to capture a focus area surrounding the supervising user;

the behavior system operative to identify a particular type of the interface accessible to the supervising user from among a plurality of types of interfaces; and

the behavior system operative to access the selection of adverse behavior definitions further comprising at least one adverse behavior definition specified for the particular type of interface in the behavior database from among a plurality of adverse behavior definitions each specified for a separate one of the plurality of types of interfaces.

12. The system according to claim 8, wherein the behavior system operative to identify a particular defined adverse behavior representing the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of adverse behavior definitions further comprises the behavior system operative to identify from the three-dimensional object properties stream one of a suspicious activity by the monitored user, an impaired condition of the monitored user, a concealed weapon carried by the monitored user, an aggressive stance by the monitored user, an angered facial expression of the monitored user, and a nervous condition of the monitored user.

13. The system according to claim 8, further comprising the behavior system operative to translate the particular defined adverse behavior into an image layer specifying a transparent overlay image and a position indicator mapping a position of the transparent overlay image to the particular defined adverse behavior within the captured three-dimensional movement for output to an image overlay controller of a display device for controlling overlay of the image layer over output of the captured three-dimensional movement to graphically indicate the warning signals to a supervising user that are undetectable by the monitored user.

14. A computer program product for warning a supervising user of behaviors of a monitored user representative of adverse behavior, the computer program product comprising:

one or more computer-readable tangible storage devices;

program instructions, stored on at least one of the one or more storage devices, to capture a three-dimensional movement of a monitored user within a particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

program instructions, stored on at least one of the one or more storage devices, to identify a three-dimensional object properties stream using the captured movement;

program instructions, stored on at least one of the one or more storage devices, to access a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

program instructions, stored on at least one of the one or more storage devices, to identify a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with a plurality of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents a particular behavior defined in the selection of adverse behavior definitions and calculating a percentage of probability that the captured three-dimensional movement is adverse;

program instructions, stored on at least one of the one or more storage devices, to notify the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the particular defined adverse behavior to a warning system for controlling warning signals output to the supervising user, wherein the warning system translates the particular defined adverse behavior into a tactile feedback control signal with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the particular defined adverse behavior in relation to the monitored user in the captured three-dimensional movement, wherein the warning system controls output of the feedback control signal to at least one tactile feedback device detectable by the supervising user for controlling the feedback strength and feedback position of the feedback output by at least one tactile feedback device;

program instructions, stored on at least one of the one or more storage devices, to capture a response by the supervising user to the supervising user receiving the notification of the particular defined adverse behavior through the interface, wherein the response indicates whether the particular defined adverse behavior is correctly identified; and

program instructions, stored on at least one of the one or more storage devices, to adjust at least one definition in the selection of adverse behavior definitions specified for the particular environment in the behavior database to reflect the response.

15. The computer program product according to claim 14, further comprising program instructions, stored on at least one of the one or more storage devices, to capture the three-dimensional movement using a stereoscopic image device to identify and track a particular three-dimensional movement of the monitored user.