Video surveillance system employing video primitives

• US 8,711,217 B2
• Filed: 12/15/2005
• Issued: 04/29/2014
• Est. Priority Date: 10/24/2000
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: ObjectVideo Incorporated (Alarm.com Holdings, Inc.)
• Inventors: Venetianer, Peter L., Lipton, Alan J., Hu, Yongtong, Martone, Andrew J., Yin, Weihong, Yu, Li, Zhang, Zhong
• Primary Examiner(s): Ortiz Criado, Jorge L
• Assistant Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US11/300,581
• Publication Number:
  US 20080100704A1
• Time in Patent Office: 3,057 Days
• Field of Search: 348/143, 348/148, 348/150, 348/149, 348/166, 348/169, 348/170, 382/103, 382/115, 375/240.02, 375/240.08
• US Class Current: 348/143
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06F 16/7854 using shape G06F16/7837 takes precedence
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G08B 13/196 using television cameras
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19615 wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: G08B 29/188 Data fusion; cooperative systems, e.g. voting among different detectors
• CPC Class Codes: G08B 31/00 Predictive alarm systems characterised by extrapolation or other computation using updated historic data
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

A video surveillance system is set up, calibrated, tasked, and operated. The system extracts video primitives and extracts event occurrences from the video primitives using event discriminators. The system can undertake a response, such as an alarm, based on extracted event occurrences.

Claims

1. A computer-implemented method of video surveillance comprising:

receiving an input video sequence from a video sensor or a video storage facility;

deriving, using at least one computer processor, at least one first video primitive from said received input video sequence, wherein the at least one first video primitive comprises at least one object descriptor referring to at least one observable attribute of at least one non-stationary object in the video sequence;

detecting at least one stationary object in said received input video sequence without reference to another object in said received input video sequence;

deriving, using at least one computer processor, at least one second video primitive from said received input video sequence, wherein the at least one second video primitive comprises at least one object descriptor referring to at least one observable attribute of the at least one stationary object in the video sequence;

processing at least one rule based on said at least one first video primitive and said at least one second video primitive, said at least one rule is satisfied if the at least one non-stationary object has not been in an exclusion zone around said at least one stationary object for a predetermined length of time, and said at least one rule is not satisfied if the at least one non-stationary object is at least partially within the exclusion zone within the predetermined length of time, said exclusion zone defined around and based on said at least one stationary object, said exclusion zone having a size sufficient to incorporate the at least one stationary object and a portion of the at least one non-stationary object;

generating a response if said at least one rule is satisfied.

2. The computer-implemented method of claim 1, further comprising:

permitting a user to set at least one exclusion zone parameter for said exclusion zone, said at least one exclusion zone parameter selected from the group consisting of;

the size of said exclusion zone, a shape of said exclusion zone, and said predetermined total length of time associated with said exclusion zone.

3. The computer-implemented method of claim 2, wherein said permitting includes permitting said user to enter a radius for said exclusion zone.

4. The computer-implemented method of claim 2, wherein said permitting comprises:

permitting said user to determine at least one exclusion zone parameter for a near-field exclusion zone and at least one exclusion zone parameter for a far-field exclusion zone.

5. The computer-implemented method of claim 4, further comprising at least one of the operations selected from the group consisting of:

interpolating between said exclusion zone parameter for said near-field exclusion zone and said exclusion zone parameter for said far-field exclusion zone to determine said at least one exclusion zone parameter in a region between said near-field exclusion zone and said far-field exclusion zone; and

extrapolating from said exclusion zone parameter for said near-field exclusion zone and said exclusion zone parameter for said far-field exclusion zone to determine said at least one exclusion zone parameter in a region outside of a region between said near-field exclusion zone and said far-field exclusion zone.

6. The computer-implemented method of claim 2, wherein said predetermined total length of time associated with said exclusion zone comprises a duration of time that a non-stationary object may leave said exclusion zone without generating an alert.

7. The computer-implemented method of claim 1, further comprising:

learning at least one exclusion zone parameter from said received input video sequence.

8. The computer-implemented method of claim 1, further comprising:

dynamically adjusting at least one exclusion zone parameter based on said received input video sequence.

9. The computer-implemented method of claim 1, wherein said at least one rule with said exclusion zone comprises a modified “

and”

operator to combine a stationary target activity detector and an area of interest inside activity detector with spatial and temporal modifiers.

10. The computer-implemented method of claim 1, wherein said processing at least one rule tracks a person responsible for leaving behind said at least one stationary object in said input video.

11. The computer-implemented method of claim 1, wherein said at least one video primitive comprises at least one condition based on a duration of time for which some predetermined action occurs.

12. The computer-implemented method of claim 1, further comprising:

deriving at least one non-video primitive;

wherein said at least one video primitive and said at least one non-video primitive are compared to said at least one rule with said exclusion zone.

13. The computer-implemented method of claim 12, wherein said at least one non-video primitive comprises at least one condition based on an occurrence of an event external to said at least one stationary object.

14. The computer-implemented method of claim 1, wherein said at least one rule with said exclusion zone comprises a query comprising at least two sub-queries and at least one combinator.

15. The computer-implemented method of claim 14, wherein at least one of said sub-queries includes a modifier.

16. The computer-implemented method of claim 15, wherein said modifier is selected from the group consisting of:

a modifier to address how a sub-query is stored;

a modifier to address whether a sub-query is stored;

a modifier to address how a sub-query is to be used in detecting one or more events; and

a modifier to address how many events a sub-query may be used to detect.

17. The computer-implemented method of claim 1, wherein said exclusion zone is defined after the at least one stationary object is detected.

18. The computer-implemented method of claim 1, wherein said exclusion zone is defined after the area of interest is established.

19. A non-transitory computer-readable medium containing machine-executable code that, when executed by said computer processor, causes said computer processor to execute the computer-implemented method of claim 1.

20. A computer-implemented method of video surveillance comprising:

receiving an input video sequence from a video sensor or a video storage facility;

detecting at least one stationary object in said received input video sequence without reference to another object in said received input video sequence;

determining, using at least one computer processor, an exclusion zone around said at least one stationary object, said exclusion zone defined around and based on said at least one stationary object;

monitoring said exclusion zone for no object of interest inside said exclusion zone, said exclusion zone having a size sufficient to incorporate the at least one stationary object and a portion of an object of interest;

identifying a period of time wherein no object of interest has been inside said exclusion zone;

comparing said period of time to at least one rule, said at least one rule is satisfied if said period of time wherein no object of interest has been inside said exclusion zone is greater than a predetermined length of time, and said at least one rule is not satisfied if at least one object of interest is at least partially within the exclusion zone within the predetermined length of time; and

generating a response if said period of time is greater than said predetermined length of time.

21. The computer-implemented method of claim 20, wherein said exclusion zone is defined after the at least one stationary object is detected.

22. The computer-implemented method of claim 20, wherein said exclusion zone is defined after the area of interest is established.

23. A video surveillance system comprising:

at least one video sensor or a video storage facility providing a video sequence; and

a video analysis subsystem, said video analysis subsystem to;

receive a video sequence;

detect at least one stationary object in said received video sequence without reference to another object in said received video sequence;

process at least one rule based on said at least one stationary object, said at least one rule is satisfied if a non-stationary object has not been in an exclusion zone around said at least one stationary object for a predetermined length of time, and said at least one rule is not satisfied if at least one non-stationary object is at least partially within the exclusion zone within the predetermined length of time, said exclusion zone defined around and based on said at least one stationary object, said exclusion zone having a size sufficient to incorporate the at least one stationary object and a portion of the non-stationary object; and

generate a response if said at least one rule is satisfied.

24. The system of claim 23, wherein said exclusion zone is defined after the at least one stationary object is detected.

25. The system of claim 23, wherein said exclusion zone is defined after the area of interest is established.

26. A computer-implemented method of video surveillance comprising:

receiving an input video sequence from a video sensor or a video storage facility;

deriving using at least one computer processor, at least one first video primitive from said received input video sequence, wherein the at least one first video primitive comprises at least one object descriptor referring to at least one observable attribute of at least one non-stationary object in the video sequence;

detecting at least one stationary object in the received input video sequence without reference to another object in said received input video sequence;

deriving using at least one computer processor, at least one second video primitive from said received input video sequence, wherein the at least one second video primitive comprises at least one object descriptor referring to at least one observable attribute of at least one stationary object in the video sequence;

processing at least one rule based on said at least one first video primitive and said at least one second video primitive, said at least one rule is satisfied if the at least one non-stationary object has not been in an exclusion zone around said at least one stationary object for a predetermined length of time, and said at least one rule is not satisfied if the at least one non-stationary object is at least partially within the exclusion zone within the predetermined length of time, said exclusion zone defined with respect to said at least one stationary object, said exclusion zone defined after the at least one stationary object is detected; and

generating a response if said at least one rule is satisfied.

27. The computer-implemented method of claim 26, wherein said exclusion zone is defined after the area of interest is established.

28. The computer-implemented method of claim 26, further comprising:

permitting a user to set at least one exclusion zone parameter for said exclusion zone, said at least one exclusion zone parameter selected from the group consisting of;

a size of said exclusion zone, and a shape of said exclusion zone, and said predetermined total length of time associated with said exclusion zone.

29. The computer-implemented method of claim 28, wherein said permitting includes permitting said user to enter a radius for said exclusion zone.

30. The computer-implemented method of claim 28, wherein said permitting comprises:

permitting said user to determine at least one exclusion zone parameter for a near-field exclusion zone and at least one exclusion zone parameter for a far-field exclusion zone.

31. The computer-implemented method of claim 30, further comprising at least one of the operations selected from the group consisting of:

interpolating between said exclusion zone parameter for said near-field exclusion zone and said exclusion zone parameter for said far-field exclusion zone to determine said at least one exclusion zone parameter in a region between said near-field exclusion zone and said far-field exclusion zone; and

extrapolating from said exclusion zone parameter for said near-field exclusion zone and said exclusion zone parameter for said far-field exclusion zone to determine said at least one exclusion zone parameter in a region outside of a region between said near-field exclusion zone and said far-field exclusion zone.

32. The computer-implemented method of claim 28, wherein said predetermined total length of time associated with said exclusion zone comprises a duration of time that a non-stationary object may leave said exclusion zone without generating an alert.

33. The computer-implemented method of claim 26, further comprising:

learning at least one exclusion zone parameter from said received input video sequence.

34. The computer-implemented method of claim 26, further comprising:

dynamically adjusting at least one exclusion zone parameter based on said received input video sequence.

35. The computer-implemented method of claim 26, wherein said at least one rule with said exclusion zone comprises a modified “

and”

operator to combine a stationary target activity detector and an area of interest inside activity detector with spatial and temporal modifiers.

36. The computer-implemented method of claim 26, wherein said processing at least one rule tracks a person responsible for leaving behind said at least one stationary object in said input video.

37. A non-transitory computer-readable medium containing machine-executable code that, when executed by said computer processor, causes said computer processor to execute the computer-implemented method of claim 26.

38. The computer-implemented method of claim 26, wherein said at least one video primitive comprises at least one condition based on a duration of time for which some predetermined action occurs.

39. The computer-implemented method of claim 26, further comprising:

deriving at least one non-video primitive;

wherein said at least one video primitive and said at least one non-video primitive are compared to said at least one rule with said exclusion zone.

40. The computer-implemented method of claim 39, wherein said at least one non-video primitive comprises at least one condition based on an occurrence of an event external to said at least one stationary object.

41. The computer-implemented method of claim 26, wherein said at least one rule with said exclusion zone comprises a query comprising at least two sub-queries and at least one combinator.

42. The computer-implemented method of claim 41, wherein at least one of said sub-queries includes a modifier.

43. The computer-implemented method of claim 42, wherein said modifier is selected from the group consisting of:

a modifier to address how a sub-query is stored;

a modifier to address whether a sub-query is stored;

a modifier to address how a sub-query is to be used in detecting one or more events; and

a modifier to address how many events a sub-query may be used to detect.

44. A computer-implemented method of video surveillance comprising:

receiving an input video sequence from a video sensor or a video storage facility;

detecting at least one stationary object in said received input video sequence without reference to another object in said received video input video sequence;

determining, using at least one computer processor, an exclusion zone around said at least one stationary object after the at least one stationary object is detected, said exclusion zone defined with respect to said at least one stationary object;

monitoring said exclusion zone for no object of interest inside said exclusion zone;

identifying a period of time wherein no object of interest has been inside said exclusion zone;

comparing said period of time to at least one rule, said at least one rule is satisfied to if said period of time wherein no object of interest has been inside said exclusion zone is greater than a predetermined length of time, and said at least one rule is not satisfied if an object of interest is at least partially within the exclusion zone within the predetermined length of time; and

generating a response if said period of time is greater than said predetermined length of time.

45. The computer-implemented method of claim 44, wherein said exclusion zone is determined after the area of interest is established.

46. A video surveillance system comprising:

at least one video sensor or a video storage facility providing a video sequence; and

a video analysis subsystem, said video analysis subsystem to;

receive said video sequence;

detect at least one stationary object in said received video sequence without reference to another object in said received video sequence;

process at least one rule based on said at least one stationary object, said at least one rule is satisfied if a non-stationary object has not been in an exclusion zone around said at least one stationary object for a predetermined length of time, and said at least one rule is not satisfied if at least one non-stationary object is at least partially within the exclusion zone within the predetermined length of time, said exclusion zone defined after the at least one stationary object is detected; and

generate a response if said at least one rule is satisfied.

47. The system of claim 46, wherein said exclusion zone is determined after the area of interest is established.