Warning a user about adverse behaviors of others within an environment based on a 3D captured image stream

• US 9,412,011 B2
• Filed: 12/08/2015
• Issued: 08/09/2016
• Est. Priority Date: 01/12/2007
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Albertson, Jacob C., Arnold, Kenneth C., Goldman, Steven D., Paolini, Michael A., Sessa, Anthony J.
• Primary Examiner(s): Murray, Daniel C
• Application Number:
  US14/962,761
• Publication Number:
  US 20160092725A1
• Time in Patent Office: 245 Days
• Field of Search: 348/152, 348/143, 348/161, 340/506, 340/573.1, 382/103, 382/107, 382/190, 706/14
• US Class Current: 1/1
• CPC Class Codes: G01V 5/20 Detecting prohibited goods, e.g. weapons, explosives, hazardous substances, contraband or smuggled objects
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data
• CPC Class Codes: G08B 6/00 Tactile signalling systems, e.g. personal calling systems
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: Y10S 706/933 Law, law enforcement, or government

Abstract

A tactile feedback output device receives one or more warning signals for controlling one or more types of tactile output by the one or more tactile feedback output devices, the one or more warning signals representing that a predicted behavior of a monitored user represents a particular behavior that is potentially adverse and a percentage probability that the predicted behavior is adverse, the monitored user detected within a particular environment monitored by a supervising user, the one or more tactile feedback output devices worn by the supervising user.

Claims

1. A system, comprising:

one or more tactile feedback output devices operative to receive one or more warning signals for controlling one or more types of tactile output by the one or more tactile feedback output devices, the one or more warning signals representing that a predicted behavior of a monitored user represents a particular behavior that is potentially adverse and a percentage probability that the predicted behavior is adverse, the monitored user detected within a particular environment monitored by a supervising user, the one or more tactile feedback output devices worn by the supervising user; and

the one or more tactile feedback output devices operative to control the one or more types of tactile outputs of the one or more tactile feedback output devices that are detectable by the supervising user wearing the one or more tactile feedback output devices according to the one or more warning signals to specifically alert the supervising user that the predicted behavior of the monitored user represents the particular behavior that is potentially adverse and the percentage probability that the predicted behavior is adverse.

2. The system according to claim 1, further comprising:

a computer system operative to capture a three-dimensional movement of the monitored user within the particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

the computer system operative to identify a three-dimensional object properties stream using the captured movement;

the computer system operative to access a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

the computer system operative to identify a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents the particular behavior defined in the selection of adverse behavior definitions and calculating the percentage of probability that the captured three-dimensional movement is adverse;

the computer system, in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, operative to notify the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the one or more warning signals to the one or more tactile feedback output devices.

3. The system according to claim 2, further comprising:

the computer system operative to capture a response by the supervising user to the supervising user receiving the notification of the particular adverse behavior through the interface of the one or more tactile feedback output devices, wherein the response indicates whether the particular adverse behavior is correctly identified; and

adjusting at least one definition in the selection of adverse behavior definitions specified for the particular environment in the behavior database to reflect the response.

4. The system according to claim 1, further comprising:

the computer system operative to translate the particular adverse behavior into the one or more warning signals comprising one or more tactile feedback control signals with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the monitored user of the particular adverse behavior within the particular environment; and

the computer system operative to control output of the one or more warning signals comprising the one or more feedback control signals to the one or more tactile feedback output devices detectable by the supervising user only for controlling a feedback strength and a feedback position of the feedback output by the one or more tactile feedback output devices.

5. The system according to claim 1, further comprising:

the one or more tactile feedback output devices each comprising a wireless transmitter for receiving the one or more warning signals via a wireless network transmission from a warning system that transmits the one or more warning signals to the tactile feedback output devices only, the warning system translating the particular adverse behavior and percentage probability into the one or more warning signals.

6. The system according to claim 1, further comprising:

the one or more tactile feedback output devices each comprising a controller for controlling the one or more types of tactile output comprising one or more detectable pressure pulses with adjustable frequency, intensity, duration, and timing; and

the controller for specifying the positions of pulses within the one or more detectable pressure pulses and for specifying the frequency, intensity, duration, and timing of the pulses based on the one or more warning signals.

7. The system according to claim 1, further comprising:

the one or more tactile feedback output devices operative to receive the one or more warning signals comprising x and y coordinates;

the one or more tactile feedback output devices operative to detect a relative position of each of the one or more tactile feedback output devices compared with a force of gravity as the supervising user wears the one or more tactile feedback output devices; and

the one or more tactile feedback output devices operative to reposition the one or more types of tactile outputs based on the x and y coordinates and the relative position of each of the one or more tactile feedback output devices compared with the force of gravity to maintain the y coordinate as up relative to the force of gravity.

8. A method, comprising:

receiving, by one or more tactile feedback output devices, one or more warning signals for controlling one or more types of tactile output by the one or more tactile feedback output devices, the one or more warning signals representing that a predicted behavior of a monitored user represents a particular behavior that is potentially adverse and a percentage probability that the predicted behavior is adverse, the monitored user detected within a particular environment monitored by a supervising user, the one or more tactile feedback output devices worn by the supervising user; and

controlling, by the one or more tactile feedback output devices, the one or more types of tactile outputs of the one or more tactile feedback output devices that are detectable by the supervising user wearing the one or more tactile feedback output devices according to the one or more warning signals to specifically alert the supervising user that the predicted behavior of the monitored user represents the particular behavior that is potentially adverse and the percentage probability that the predicted behavior is adverse.

9. The method according to claim 8, further comprising:

capturing, by a computer system, a three-dimensional movement of the monitored user within the particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

identifying, by the computer system, a three-dimensional object properties stream using the captured movement;

accessing, by the computer system, a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

identifying, by the computer system, a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents the particular behavior defined in the selection of adverse behavior definitions and calculating the percentage of probability that the captured three-dimensional movement is adverse;

in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, notify, by the computer system, the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the one or more warning signals to the one or more tactile feedback output devices.

10. The method according to claim 9, further comprising:

capturing, by the computer system, a response by the supervising user to the supervising user receiving the notification of the particular adverse behavior through the interface of the one or more tactile feedback output devices, wherein the response indicates whether the particular adverse behavior is correctly identified; and

adjusting, by the computer system, at least one definition in the selection of adverse behavior definitions specified for the particular environment in the behavior database to reflect the response.

11. The method according to claim 8, further comprising:

translating, by the computer system, the particular adverse behavior into the one or more warning signals comprising one or more tactile feedback control signals with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the monitored user of the particular adverse behavior within the particular environment; and

controlling, by the computer system, output of the one or more warning signals comprising the one or more feedback control signals to the one or more tactile feedback output devices detectable by the supervising user only for controlling a feedback strength and a feedback position of the feedback output by the one or more tactile feedback output devices.

12. The method according to claim 8, further comprising:

receiving, by a wireless transmitter of each of the one or more tactile feedback output devices, the one or more warning signals via a wireless network transmission from a warning system that transmits the one or more warning signals to the tactile feedback output devices only, the warning system translating the particular adverse behavior and percentage probability into the one or more warning signals.

13. The method according to claim 8, further comprising:

controlling, by a controller of each the one or more tactile feedback output devices, the one or more types of tactile output comprising one or more detectable pressure pulses with adjustable frequency, intensity, duration, and timing; and

specifying, by the controller, the positions of pulses within the one or more detectable pressure pulses and for specifying the frequency, intensity, duration, and timing of the pulses based on the one or more warning signals.

14. The method according to claim 8, further comprising:

receiving, by the one or more tactile feedback output devices, the one or more warning signals comprising x and y coordinates;

detecting, by the one or more tactile feedback output devices, a relative position of each of the one or more tactile feedback output devices compared with a force of gravity as the supervising user wears the one or more tactile feedback output devices; and

repositioning, by the one or more tactile feedback output devices, the one or more types of tactile outputs based on the x and y coordinates and the relative position of each of the one or more tactile feedback output devices compared with the force of gravity to maintain the y coordinate as up relative to the force of gravity.

15. A computer program product comprising one or more computer-readable storage devices and program instructions, stored on at least one of the one or more storage devices, the stored program instructions comprising:

program instructions to receive, by one or more tactile feedback output devices, one or more warning signals for controlling one or more types of tactile output by the one or more tactile feedback output devices, the one or more warning signals representing that a predicted behavior of a monitored user represents a particular behavior that is potentially adverse and a percentage probability that the predicted behavior is adverse, the monitored user detected within a particular environment monitored by a supervising user, the one or more tactile feedback output devices worn by the supervising user; and

program instructions to control, by one or more tactile feedback output devices, the one or more types of tactile outputs of the one or more tactile feedback output devices that are detectable by the supervising user wearing the one or more tactile feedback output devices according to the one or more warning signals to specifically alert the supervising user that the predicted behavior of the monitored user represents the particular behavior that is potentially adverse and the percentage probability that the predicted behavior is adverse.

16. The computer program product according to claim 15, wherein the program instructions further comprise:

program instructions to capture a three-dimensional movement of the monitored user within the particular environment monitored by the supervising user, wherein the three-dimensional movement is determined using at least one image capture device aimed at the monitored user;

program instructions to identify a three-dimensional object properties stream using the captured movement;

program instructions to access a selection of adverse behavior definitions specified for the particular environment from among a plurality of adverse behavior definitions specified for a plurality of environments from a behavior database;

program instructions to identify a particular defined adverse behavior of the monitored user represented by the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with the selection of adverse behavior definitions by calculating a percentage of probability that the captured three-dimensional movement represents the particular behavior defined in the selection of adverse behavior definitions and calculating the percentage of probability that the captured three-dimensional movement is adverse;

program instructions, in response to identifying the particular defined adverse behavior of the monitored user from among the plurality of adverse behavior definitions, to notify the supervising user for the particular environment about the particular defined adverse behavior of the monitored user through an interface only detectable by the supervising user by sending the one or more warning signals to the one or more tactile feedback output devices.

17. The computer program product according to claim 15, wherein the program instructions further comprise:

program instructions to translate the particular adverse behavior into the one or more warning signals comprising one or more tactile feedback control signals with a feedback strength specified to a percentage of total feedback strength to reflect the percentage probability that the captured three-dimensional movement is adverse and with a feedback position specified to one or multiple feedback positions to reflect a position of the monitored user of the particular adverse behavior within the particular environment; and

program instructions to control output of the one or more warning signals comprising the one or more feedback control signals to the one or more tactile feedback output devices detectable by the supervising user only for controlling a feedback strength and a feedback position of the feedback output by the one or more tactile feedback output devices.

18. The computer program product according to claim 15, wherein the program instructions further comprise:

program instructions to receive, by a wireless transmitter for each of the one or more tactile feedback output devices, the one or more warning signals via a wireless network transmission from a warning system that transmits the one or more warning signals to the tactile feedback output devices only, the warning system translating the particular adverse behavior and percentage probability into the one or more warning signals.

19. The computer program product according to claim 15, wherein the program instructions further comprise:

program instructions to control, by a controller of each of the one or more tactile feedback output devices, the one or more types of tactile output comprising one or more detectable pressure pulses with adjustable frequency, intensity, duration, and timing; and

program instructions to specify, by the controller, the positions of pulses within the one or more detectable pressure pulses and for specifying the frequency, intensity, duration, and timing of the pulses based on the one or more warning signals.

20. The computer program product according to claim 15, wherein the program instructions further comprise:

program instructions to receive, by the one or more tactile feedback output devices, the one or more warning signals comprising x and y coordinates;

program instructions to detect, by the one or more tactile feedback output devices, a relative position of each of the one or more tactile feedback output devices compared with a force of gravity as the supervising user wears the one or more tactile feedback output devices; and

program instructions to reposition, by the one or more tactile feedback output devices, the one or more types of tactile outputs based on the x and y coordinates and the relative position of each of the one or more tactile feedback output devices compared with the force of gravity to maintain the y coordinate as up relative to the force of gravity.