Loitering detection in a video surveillance system

• US 9,208,675 B2
• Filed: 03/15/2013
• Issued: 12/08/2015
• Est. Priority Date: 03/15/2012
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Xu, Gang, Cobb, Wesley Kenneth
• Primary Examiner(s): Bella, Matthew
• Assistant Examiner(s): SHIN, SOO JUNG
• Application Number:
  US13/836,372
• Publication Number:
  US 20130243252A1
• Time in Patent Office: 998 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/0409 Adaptive resonance theory [ART] networks
• CPC Class Codes: G06N 3/088 Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06V 10/7788 the supervisor being a human, e.g. interactive learning with a human teacher
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 21/182 Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: G08B 23/00 Alarms responsive to unspecified undesired or abnormal conditions
• CPC Class Codes: G08B 29/185 Signal analysis techniques for reducing or preventing false alarms or for enhancing the reliability of the system
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098

Abstract

A behavioral recognition system may include both a computer vision engine and a machine learning engine configured to observe and learn patterns of behavior in video data. Certain embodiments may be configured to learn patterns of behavior consistent with a person loitering and generate alerts for same. Upon receiving information of a foreground object remaining in a scene over a threshold period of time, a loitering detection module evaluates the whether the object trajectory corresponds to a random walk. Upon determining that the trajectory does correspond, the loitering detection module generates a loitering alert.

Claims

1. A method for detecting loitering behavior of objects depicted in a scene captured by a video camera, the method comprising:

receiving a trajectory for an object in the scene, wherein the object has been in the scene for a time period, and wherein the trajectory tracks a two-dimensional (2D) path of the object relative to a series of video frames in which the object is depicted; and

upon determining that the time period that the object has been in the scene is greater than a threshold time period and determining that the trajectory corresponds to a random walk, generating a loitering alert, wherein determining whether the trajectory corresponds to a random walk comprises;

calculating a global loitering score, wherein the global loitering score is based on a distance between a final point in the trajectory and a starting point in the trajectory,

calculating at least one local loitering score, wherein each local loitering score is based on a distance between the final point and a respective intermediate point in the trajectory,

calculating an estimated loitering score as an average of the global loitering score and the at least one local loitering score, and

comparing the estimated loitering score with a threshold loitering score, wherein the trajectory corresponds to a random walk if the estimated loitering score is greater than the threshold loitering score.

2. The method of claim 1, wherein the object trajectory corresponds to a random walk if D<

√

{square root over (2N)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory and N is a total number of steps in the trajectory.

3. The method of claim 2, wherein distance D and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

4. The method of claim 1, wherein the trajectory corresponds to a random walk if D<

√

{square root over (2N)}, and D′

<

√

{square root over (2N′

)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory, N is a total number of steps in the trajectory, D′

is a distance from an intermediate point in the trajectory to the final point in the trajectory, and N′

is the total number of steps of the trajectory from the intermediate point to the final point.

5. The method of claim 4, wherein distances D and D′

and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

6. The method of claim 1, wherein the threshold time period is a preset value.

7. The method of claim 1, wherein the threshold time period is based on a distribution of lengths of time that a plurality of objects have been in the scene.

8. The method of claim 1, wherein the distance between a final point in the trajectory and a starting point in the trajectory and the distance between the final point and each respective intermediate point is determined relative to a set of pixels corresponding to the object in the series of video frames.

9. A non-transitory computer-readable storage medium storing instructions, which, when executed on a processor, performs an operation for detecting loitering behavior, the operation comprising:

receiving a trajectory for an object in the scene, wherein the object has been in the scene for a time period, and wherein the trajectory tracks a two-dimensional (2D) path of the object relative to a series of video frames in which the object is depicted; and

upon determining that the time period that the object has been in the scene is greater than a threshold time period and determining that the trajectory corresponds to a random walk, generating a loitering alert, wherein determining whether the trajectory corresponds to a random walk comprises;

calculating a global loitering score, wherein the global loitering score is based on a distance between a final point in the trajectory and a starting point in the trajectory,

calculating at least one local loitering score, wherein each local loitering score is based on a distance between the final point and a respective intermediate point in the trajectory,

calculating an estimated loitering score as an average of the global loitering score and the at least one local loitering score, and

comparing the estimated loitering score with a threshold loitering score, wherein the trajectory corresponds to a random walk if the estimated loitering score is greater than the threshold loitering score.

10. The computer-readable storage medium of claim 9, wherein the object trajectory corresponds to a random walk if D<

√

{square root over (2N)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory and N is a total number of steps in the trajectory.

11. The computer-readable storage medium of claim 10, wherein distance D and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

12. The computer-readable storage medium of claim 9, wherein the trajectory corresponds to a random walk if D<

√

{square root over (2N)}, and D′

<

√

{square root over (2N′

)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory, N is a total number of steps in the trajectory, D′

is a distance from an intermediate point in the trajectory to the final point in the trajectory, and N′

is the total number of steps of the trajectory from the intermediate point to the final point.

13. The computer-readable storage medium of claim 12, wherein distances D and D′

and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

14. The computer-readable storage medium of claim 9, wherein the threshold time period is a preset value.

15. The computer-readable storage medium of claim 9, wherein the threshold time period is based on a distribution of lengths of time that a plurality of objects have been in the scene.

16. The computer-readable storage medium of claim 9, wherein the distance between a final point in the trajectory and a starting point in the trajectory and the distance between the final point and each respective intermediate point is determined relative to a set of pixels corresponding to the object in a series of video frames.

17. A system comprising:

a processor; and

a memory hosting an application, which, when executed on the processor, performs an operation for detecting loitering behavior, the operation comprising;

receiving a trajectory for an object in the scene, wherein the object has been in the scene for a time period, and wherein the trajectory tracks a two-dimensional (2D) path of the object relative to a series of video frames in which the object is depicted, and

upon determining that the time period that the object has been in the scene is greater than a threshold time period and determining that the trajectory corresponds to a random walk, generating a loitering alert, wherein determining whether the trajectory corresponds to a random walk comprises;

calculating a global loitering score, wherein the global loitering score is based on a distance between a final point in the trajectory and a starting point in the trajectory;

calculating at least one local loitering score, wherein each local loitering score is based on a distance between the final point and a respective intermediate point in the trajectory;

calculating an estimated loitering score as an average of the global loitering score and the at least one local loitering score; and

comparing the estimated loitering score with a threshold loitering score, wherein the trajectory corresponds to a random walk if the estimated loitering score is greater than the threshold loitering score.

18. The system of claim 17, wherein the object trajectory corresponds to a random walk if D<

√

{square root over (2N)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory and N is a total number of steps in the trajectory.

19. The system of claim 18, wherein distance D and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

20. The system of claim 17, wherein the trajectory corresponds to a random walk if D<

√

{square root over (2N)}, and D40 <

√

{square root over (2N′

)}, where D is a distance from a starting point in the trajectory to a final point in the trajectory, N is a total number of steps in the trajectory, D′

is a distance from an intermediate point in the trajectory to the final point in the trajectory, and N′

is the total number of steps of the trajectory from the intermediate point to the final point.

21. The system of claim 20, wherein distances D and D′

and steps are determined relative to distances between a set of pixels corresponding to the object in the series of video frames, and wherein the steps are determined by a centroid position of the object in each video frame.

22. The system of claim 17, wherein the threshold time period is a preset value.

23. The system of claim 17, wherein the threshold time period is based on a distribution of lengths of time that a plurality of objects have been in the scene.

24. The system of claim 17 wherein the distance between a final point in the trajectory and a starting point in the trajectory and the distance between the final point and each respective intermediate point is determined relative to a set of pixels corresponding to the object in a series of video frames.