Object tracking and alerts

• US 9,036,028 B2
• Filed: 05/30/2006
• Issued: 05/19/2015
• Est. Priority Date: 09/02/2005
• Status: Active Grant

• Current Assignee: Johnson Controls Tyco IP Holdings LLP (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Inventors: Buehler, Christopher J.
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): FINDLEY, CHRISTOPHER G
• Application Number:
  US11/443,500
• Publication Number:
  US 20070182818A1
• Time in Patent Office: 3,276 Days
• Field of Search: None
• US Class Current: 348/159
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G07C 9/28 the pass enabling tracking or indicating presence
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/1966 Wireless systems, other than telephone systems, used to communicate with a camera
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19691 Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves
• CPC Class Codes: G08B 13/248 EAS system combined with another detection technology, e.g. dual EAS and video or other presence detection system
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

An integrated surveillance system combining video surveillance and data from other sensor-based security networks is used to identify activities that may require attention.

Claims

1. A surveillance system comprising:

a video surveillance subsystem comprising a plurality of cameras each having a field of view, each field of view segmented into a plurality of predefined static camera sub-regions;

a radio-frequency identification subsystem comprising a plurality of radio-frequency stations, each capable of detecting radio-frequency identification tags within a respective operational radius;

a tracking module for chronologically representing, based on data received from the video surveillance subsystem and the radio-frequency identification subsystem, physical locations of objects tracked by the surveillance system as the objects move among the predefined static sub-regions of the segmented fields of view of the cameras and the operational radii of the stations;

an association module in communication with the video surveillance subsystem and the radio-frequency identification subsystem for inferring, over time, associations among one or more of the predefined static camera sub-regions of the segmented fields of view and one or more of the operational radii of the radio-frequency stations from data captured by the video surveillance subsystem and data captured by the radio-frequency identification subsystem, the associations based at least in part on repeated detection of tracked objects in both the one or more predefined static camera sub-regions of the segmented fields of view and the one or more operational radii at substantially the same time;

determining a first probability that a first one of the predefined camera sub-regions is associated with a first one of the operational radii based on a detection of a particular tracked object in both the first predefined camera sub-region and the first operational radius at a first time; and

determining a second probability that the first predefined camera sub-region and the first operational radius are associated with each other based on a detection of a particular tracked object in both the first predefined camera sub-region and the first operational radius at a second time, wherein the second probability is higher than the first probability and wherein the second time is later than the first time,

wherein for a particular association between a specific one of the predefined static camera sub-regions and a specific one of the operational radii, at least one of the following is unknown;

(i) a location of the camera having the field of view comprising the specific predefined static camera sub-region and (ii) a location of the radio-frequency station associated with the specific operational radius;

a data storage module for storing rules relating to the objects, wherein at least one of the rules comprises two or more primitive events; and

an alarm module, responsive to the data storage module and the tracking module, for;

applying one or more of the rules to the chronological representations of the physical locations of one or more of the objects to determine whether a rule violation exists, and

in response to determining that one or more of the rules are violated by the chronological representations, creating one or more alerts.

2. The system of claim 1 wherein the associations are based on locational correspondences between the camera sub-regions and the operational radii.

3. The system of claim 1 wherein the data captured by the video surveillance subsystem comprises visual representations of objects moving among the plurality of cameras.

4. The system of claim 1 wherein the data captured by the video surveillance subsystem comprises visual representations of objects moving among the plurality of sub-regions of the cameras.

5. The system of claim 1 wherein the data captured by the radio-frequency identification subsystem comprises spatial information associated with objects moving among the operational radii of the stations.

6. The system of claim 1 wherein the video surveillance subsystem and the radio-frequency identification subsystem operate within a region, the region comprising at least one ineffective area outside the fields of view of the cameras and the operational radii of the stations.

7. The system of claim 6 wherein the tracking module infers the location of one or more of the objects being tracked by the surveillance system while one or more objects pass through one or more ineffective areas.

8. The system of claim 6 wherein the tracking module infers a next camera location for an object being tracked by the surveillance system, while the object passes through an ineffective area, based on a previously known location of the object.

9. The system of claim 8 wherein the previously known location is based on data received from one or more of the radio-frequency identification subsystem and the video surveillance subsystem.

10. The system of claim 1 further comprising a transmission module for transmitting the alerts to a wireless communication device.

11. The system of claim 1 wherein the cameras are responsive to the alert modules such that an alert further instructs one or more cameras to focus on an object of interest.

12. The system of claim 1 wherein the cameras are responsive to the alert modules such that an alert further instructs one or more cameras to focus on an area of interest.

13. The system of claim 1 wherein the cameras are responsive to the alert modules such that an alert further instructs one or more cameras to focus on a person of interest.

14. The system of claim 1 wherein the video surveillance subsystem and the radio-frequency identification subsystem are calibrated independently of each other.

15. The system of claim 1 wherein the association of the camera sub-regions with one or more of operational radii of the radio-frequency stations is a many-to-many association.

16. The system of claim 1 wherein the data captured by the video surveillance subsystem comprises visual representations of objects moving among the plurality of cameras and data captured by the radio-frequency identification subsystem comprises spatial information associated with objects moving among the operational radii of the stations and the associations are inferred based on object paths common to one or more of the camera sub-regions and one or more of the operational radii of the radio-frequency stations.

17. The system of claim 1 wherein two or more primitive events are combined using Boolean logic.

18. The system of claim 1 wherein the alarm module determines whether one or more of the rules are violated by determining whether a combination of primitive events has occurred.

19. The system of claim 1 wherein the alarm module determines whether one or more of the rules are violated by evaluating a temporal relationship among a plurality of primitive events.

20. The system of claim 1 wherein the alarm module determines whether one or more of the rules are violated by evaluating a sequential relationship among a plurality of primitive events.

21. The system of claim 1 wherein the alarm module determines whether one or more of the rules are violated by determining whether a first primitive event has occurred and a second primitive event has not occurred.

22. The system of claim 1, wherein the tracking module is further for creating an association between at least two of the tracked objects, the at least two tracked objects being physically separated, and wherein applying one or more of the rules comprises applying one or more of the rules to the chronological representations of the physical locations of the associated objects to determine whether a rule violation exists.

23. The system of claim 22, wherein creating the association comprises automatically linking the at least two tracked objects based on repeated detection of the at least two tracked objects in close proximity.

24. The system of claim 23, wherein determining whether a rule violation exists comprises determining whether the associated objects are separated by a minimum distance.

25. The system of claim 23, wherein determining whether a rule violation exists comprises determining whether the associated objects are separated by a minimum distance for an amount of time greater than a threshold.

26. A method of monitoring activity within an environment, the method comprising:

receiving a plurality of image signals generated by a plurality of cameras each having a field of view segmented into a plurality of predefined static camera sub-regions;

receiving data generated by a plurality of radio-frequency stations, the data being based on interaction with a plurality of radio-frequency tags associated with objects within the environment;

tracking a plurality of objects within the environment as the objects move among the predefined static sub-regions of the segmented fields of view of the cameras and the operational radii of the stations;

inferring associations among one or more of the predefined static camera sub-regions of the segmented fields of view with one or more of operational radii of the stations from the image signals and the data generated by a plurality of radio-frequency stations, the associations based at least in part on repeated detection of tracked objects in both the one or more predefined static camera sub-regions of the segmented fields of view and the one or more operational radii at substantially the same time;

determining a first probability that a first one of the predefined camera sub-regions is associated with a first one of the operational radii based on a detection of a particular tracked object in both the first predefined camera sub-region and the first operational radius at a first time; and

determining a second probability that the first predefined camera sub-region and the first operational radius are associated with each other based on a detection of a particular tracked object in both the first predefined camera sub-region and the first operational radius at a second time, wherein the second probability is higher than the first probability and wherein the second time is later than the first time,

wherein for a particular association between a specific one of the predefined static camera sub-regions and a specific one of the operational radii, at least one of the following is unknown;

(i) a location of the camera having the field of view comprising the specific predefined static camera sub-region and (ii) a location of the radio-frequency station associated with the specific operational radius;

chronologically representing, based on data received from the cameras and the radio-frequency stations, physical locations of the objects;

storing rules relating to the objects, wherein at least one of the rules comprises two or more primitive events;

applying one or more of the rules to the chronological representations of the physical locations of one or more of the objects to determine whether a rule violation exists; and

in response to determining that one or more of the rules are violated by the chronological representations, creating one or more alerts.

27. The method of claim 26 wherein the plurality of video signals generated by a plurality of cameras comprises visual representations of objects moving among the plurality of cameras.

28. The method of claim 26 wherein the plurality of video signals generated by a plurality of cameras comprises visual representations of objects moving among the plurality of sub-regions of the cameras.

29. The method of claim 26 wherein the data generated by the plurality of radio-frequency stations comprises spatial information associated with objects moving among the operational radii of the radio-frequency stations.

30. The method of claim 26 further comprising inferring the location of one or more of the objects within the environment while the objects pass through one or more ineffective areas, the ineffective areas being outside the fields of view of the cameras and the operational radii of the stations.

31. The method of claim 26 further comprising inferring a next camera location for an object within the environment, while the object passes through an ineffective area, based on a previously known location of the object.

32. The method of claim 31 wherein the previously known location is based on the plurality of video signals generated by the plurality of cameras and the data generated by the plurality of radio-frequency stations.

33. The method of claim 26 further comprising transmitting the alerts to a wireless communication device.

34. The method of claim 26 wherein the alerts further instruct one or more cameras to focus on an object of interest within the environment.

35. The method of claim 26 wherein the alerts further instruct one or more cameras to focus on an area of interest within the environment.

36. The method of claim 26 wherein the alerts further instruct one or more cameras to focus on a person of interest within the environment.

37. The method of claim 26 wherein the data captured by the video surveillance subsystem comprises visual representations of objects moving among the plurality of cameras and data captured by the radio-frequency identification subsystem comprises spatial information associated with objects moving among the operational radii of the stations and the associations are inferred based on object paths common to one or more of the camera sub-regions and one or more of the operational radii of the radio-frequency stations.

38. The method of claim 26 wherein two or more of the primitive events are combined using Boolean logic.

39. The method of claim 26 wherein determining one or more of the rules are violated comprises determining whether a combination of primitive events has occurred.

40. The method of claim 26 wherein determining one or more of the rules are violated comprises evaluating a temporal relationship among a plurality of primitive events.

41. The method of claim 26 wherein determining one or more of the rules are violated comprises evaluating a sequential relationship among a plurality of primitive events.

42. The method of claim 26 wherein determining one or more of the rules are violated comprises determining whether a first primitive event has occurred and a second primitive event has not occurred.