Anomalous object interaction detection and reporting

• US 10,410,058 B1
• Filed: 01/18/2018
• Issued: 09/10/2019
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni AI, Inc.
• Inventors: Saitwal, Kishor Adinath, Urech, Dennis G., Cobb, Wesley Kenneth
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/874,172
• Time in Patent Office: 600 Days
• Field of Search: 345474
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. The techniques include evaluating sequence pairs representing segments of object trajectories. Assuming the objects interact, each of the sequences of the sequence pair may be mapped to a sequence cluster of an adaptive resonance theory (ART) network. A rareness value for the pair of sequence clusters may be determined based on learned joint probabilities of sequence cluster pairs. A statistical anomaly model, which may be specific to an interaction type or general to a plurality of interaction types, is used to determine an anomaly temperature, and alerts are issued based at least on the anomaly temperature. In addition, the ART network and the statistical anomaly model are updated based on the current interaction.

Claims

1. A processor-implemented method, comprising:

analyzing, via a processor, a scene represented by an input stream of video frames captured by a video camera, the analysis including;

receiving a sequence pair including a first sequence and a second sequence, the first sequence associated with a segment of a trajectory of a first object through the scene, and the second sequence associated with a segment of a trajectory of a second object through the scene,

determining that the first object and the second object interact with one another based on a spatio-temporal proximity of the first and second objects; and

upon determining that the first object and the second object interact, mapping the first sequence to a first sequence cluster and mapping the second sequence to a second sequence cluster,

determining a rareness value for the first and second sequence clusters, the rareness value given by;

R_ij=1−

f_ij/f_max,

where f_ij is an observation frequency associated with the first and second sequence clusters, and f_max is a frequency of a most frequently observed sequence cluster pair;

determining an anomaly temperature based on the rareness value; and

issuing an anomalous object interaction report when the anomaly temperature has a value that exceeds a threshold value.

2. The processor-implemented method of claim 1, wherein each sequence of the sequence pair is normalized to an n-dimensional vector.

3. The processor-implemented method of claim 1, wherein determining that the first object and the second object interact includes detecting that an edge of a bounding box of the first sequence has come within a predetermined number of pixels of an edge of a bounding box of the second sequence.

4. The processor-implemented method of claim 1, wherein determining that the first object and the second object interact includes detecting that the trajectory of the first object intersects with the trajectory of the second object.

5. The processor-implemented method of claim 1, further comprising detecting, based on the first and second sequence clusters, a tailgating event, the determining the anomaly temperature further based on a tailgating anomaly model.

6. The processor-implemented method of claim 1, wherein determining the anomaly temperature is performed using a statistical anomaly model.

7. The processor-implemented method of claim 1, further comprising updating an anomaly model based on the rareness value.

8. An apparatus, comprising:

a processor; and

a memory in operable communication with the processor and storing processor-executable instructions to;

analyze, via a processor, a scene represented by an input stream of video frames captured by a video camera, by;

receiving a sequence pair including a first sequence and a second sequence, the first sequence associated with a segment of a trajectory of a first object through the scene, and the second sequence associated with a segment of a trajectory of a second object through the scene,

determining that the first object and the second object interact with one another based on a spatio-temporal proximity of the first and second objects; and

upon determining that the first object and the second object interact, mapping the first sequence to a first sequence cluster and mapping the second sequence to a second sequence cluster,

determine a rareness value for the first and second sequence clusters, the rareness value given by;

R_ij=1−

f_ij/f_max,

where f_ij is an observation frequency associated with the first and second sequence clusters, and f_max is a frequency of a most frequently observed sequence cluster pair;

determine an anomaly temperature based on the rareness value; and

issue an anomalous object interaction report if a value of the anomaly temperature exceeds a threshold value.

9. The apparatus of claim 8, wherein each sequence of the sequence pair is normalized to an n-dimensional vector.

10. The apparatus of claim 8, wherein determining that the first object and the second object interact includes detecting that an edge of a bounding box of the first sequence has come within a predetermined number of pixels of an edge of a bounding box of the second sequence.

11. The apparatus of claim 8, wherein determining that the first object and the second object interact includes detecting that the trajectory of the first object intersects with the trajectory of the second object.

12. The apparatus of claim 8, the memory further storing processor-executable instructions to detect, based on the first and second sequence clusters, a tailgating event, the determining the anomaly temperature further based on a tailgating anomaly model.

13. The apparatus of claim 8, wherein determining the anomaly temperature is performed using a statistical anomaly model.

14. The apparatus of claim 8, the memory further storing processor-executable instructions to update an anomaly model based on the rareness value.

15. A non-transitory processor-readable medium storing instructions to:

analyze, via a processor, a scene represented by an input stream of video frames captured by a video camera, by;

receiving a sequence pair including a first sequence and a second sequence, the first sequence associated with a segment of a trajectory of a first object through the scene, and the second sequence associated with a segment of a trajectory of a second object through the scene,

determining that the first object and the second object interact with one another based on a spatio-temporal proximity of the first and second objects; and

upon determining that the first object and the second object interact, mapping the first sequence to a first sequence cluster and mapping the second sequence to a second sequence cluster,

determine a rareness value for the first and second sequence clusters, the rareness value given by;

R_ij=1−

f_ij/f_max,

where f_ij is an observation frequency associated with the first and second sequence clusters, and f_max is a frequency of a most frequently observed sequence cluster pair;

determine an anomaly temperature based on the rareness value; and

issue an anomalous object interaction report if the anomaly temperature has a value that exceeds a threshold value.

16. The processor-readable medium of claim 15, wherein each sequence of the sequence pair is normalized to an n-dimensional vector.

17. The processor-readable medium of claim 15, wherein determining that the first object and the second object interact includes detecting that an edge of a bounding box of the first sequence has come within a predetermined number of pixels of an edge of a bounding box of the second sequence.

18. The processor-readable medium of claim 15, wherein determining that the first object and the second object interact includes detecting that the trajectory of the first object intersects with the trajectory of the second object.

19. The processor-readable medium of claim 15, further storing processor-executable instructions to detect, based on the first and second sequence clusters, a tailgating event, the determining the anomaly temperature further based on a tailgating anomaly model.

20. The processor-readable medium of claim 15, wherein determining the anomaly temperature is performed using a statistical anomaly model.