Complex event recognition in a sensor network

• US 10,186,123 B2
• Filed: 03/31/2015
• Issued: 01/22/2019
• Est. Priority Date: 04/01/2014
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation
• Inventors: Kanaujia, Atul, Choe, Tae Eun, Deng, Hongli
• Primary Examiner(s): Rudolph, Vincent
• Assistant Examiner(s): Patel, Pinalben
• Application Number:
  US14/674,889
• Publication Number:
  US 20150279182A1
• Time in Patent Office: 1,393 Days
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream

Abstract

Systems, methods, and manufactures for a surveillance system are provided. The surveillance system includes sensors having at least one non-overlapping field of view. The surveillance system is operable to track a target in an environment using the sensors. The surveillance system is also operable to extract information from images of the target provided by the sensors. The surveillance system is further operable to determine probabilistic confidences corresponding to the information extracted from images of the target. The confidences include at least one confidence corresponding to at least one primitive event. Additionally, the surveillance system is operable to determine grounded formulae by instantiating predefined rules using the confidences. Further, the surveillance system is operable to infer a complex event corresponding to the target using the grounded formulae. Moreover, the surveillance system is operable to provide an output describing the complex event.

Claims

1. A surveillance system comprising a computing device comprising a processor and computer-readable storage device storing program instructions that, when executed by the processor, cause the computing device to perform operations comprising:

tracking a target in an environment using sensors;

extracting information from images of the target provided by the sensors;

determining a plurality of confidences corresponding to the information extracted from images of the target;

the plurality of confidences including at least one confidence corresponding to at least one primitive event;

determining grounded formulae by instantiating predefined rules using the plurality of confidences;

inferring a complex event corresponding to the target using the grounded formulae; and

providing an output describing complex event,

wherein;

the predefined rules comprise hard rules and soft rules,

the hard rules comprise a first plurality of rules adapted to set a probability of the complex event to zero when violated,

the soft rules comprise a second plurality of rules adapted to make the complex event less probable, but not impossible, when violated, and

the soft rules are associated with weights representing uncertainty.

2. The system of claim 1, wherein extracting the information comprises:

segmenting scenes captured by the sensors;

detecting the at least one primitive event;

classifying the target; and

extracting attributes of the target.

3. The system of claim 2, wherein the at least one primitive event includes disappearing from a scene and reappearing in the scene.

4. The system of claim 1, wherein the operations further comprise constructing a Markov logic network from the grounded formulae.

5. The system of claim 1, wherein the operations further comprise controlling the computing device to fuse the trajectory of the target across more than one of the sensors using a Markov logic network.

6. The system of claim 1, wherein:

at least one of the sensors is an non-calibrated sensor; and

the sensors have at least one non-overlapping field of view.

18. The system of claim 1, wherein inferring a complex event comprises determining that a complex event likely occurred based only on other observed events and not based on a direct observation of the complex event itself.

19. The system of claim 1, wherein the hard rules and the soft rules model spatial and temporal interactions between various entities and a temporal structure of a plurality of complex events.

21. The system of claim 1, wherein the predefined rules define observable events in the environment evincing an occurrence of the complex event.

22. The system of claim 21, wherein inferring the complex event comprises determining that the complex event occurred based only on the observable events and not based on a direct observation of the complex event.

23. The system of claim 22, wherein, the complex event comprises an occurrence determined to have occurred based only on circumstantial evidence.

24. The system of claim 23, wherein the observable events comprise occurrences involving the target in relation to a predefined object in the environment.

25. The system of claim 1, wherein:

the complex event is one of a plurality of complex events predefined for a particular environment;

each of the plurality of complex events comprises a plurality of observable events relevant to a predetermined threat for which a surveillance system monitors in the environment.

26. The system of claim 1, wherein the at least one primitive event comprises time information and location information obtained from a track of the target.

27. The system of claim 1, wherein the predefined rules comprise first order predicate logic formulas of a Markov logic network.

7. A method for a surveillance system comprising:

tracking a target n an environment using sensors;

extracting information from images of the target provided by the sensors;

determining a plurality of confidences corresponding to the information extracted from images of the target, the plurality of confidences including at least one confidence corresponding to at least one primitive event;

determining grounded formulae by instantiating predefined rules using plurality of confidences;

inferring a complex event corresponding to the target using the grounded formulae; and

providing an output describing the complex event

wherein;

the predefined rules comprise hard rules and soft rules,

the hard rules comprise a first plurality of rules adapted to set a probability of the complex event to zero when violated,

the soft rules comprise a second plurality of rules adapted to make the complex event less probable, but not impossible, when violated, and

the soft rules are associated with weights representing uncertainty.

8. The method of claim 7, wherein extracting the information comprises:

segmenting scenes captured by the sensors;

detecting the at least one primitive event;

classifying the target; and

extracting attributes of the target.

9. The method of claim 8, wherein the at least one primitive event includes disappearing from a scene and reappearing in the scene.

10. The method of claim 7, further comprising constructing a Markov logic network from the grounded formulae.

11. The method of claim 7, further comprising fusing the trajectory of the target across more than one of the sensors.

12. The method of claim 11, further comprising performing the fusing using a Markov logic network.

28. The method of claim 7, wherein the predefined rules comprise first order predicate logic formulas of a Markov logic network.

13. A non-transitory computer-readable medium storing computer-executable program instructions that, when executed by a computer, cause the computer to perform operations comprising:

tracking a target in an environment using sensors;

extracting information from images of the target provided by the sensors;

determining a plurality of confidences corresponding to the information extracted from images of the target;

the plurality of confidences including at least one confidence corresponding to at least one primitive event;

determining grounded formulae by instantiating predefined rules using the plurality of confidences;

inferring a complex event corresponding to the target using the grounded formulae; and

providing an output describing the complex event

wherein;

the predefined rules comprise hard rules and soft rules,

the hard rules comprise a first plurality of rules adapted to set a probability of the complex event to zero when violated,

the soft rules comprise a second plurality of rules adapted to make the complex event less probable, but not impossible, when violated, and

the soft rules are associated with weights representing uncertainty.

14. The non-transitory computer-readable medium of claim 13, wherein extracting the information comprises:

segmenting scenes captured by the sensors;

detecting the at least one primitive event;

classifying the target; and

extracting attributes of the target.

15. The non-transitory computer-readable medium of claim 14, wherein the at least one primitive event includes disappearing from a scene and reappearing in the scene.

16. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise controlling the computing device to construct a Markov logic network from the grounded formulae.

17. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise controlling the computing device to fuse the trajectory of the target across more than one of the sensors.

29. The non-transitory computer-readable medium of claim 13, wherein the predefined rules comprise first order predicate logic formulas of a Markov logic network.

20. A surveillance system comprising a computing device comprising a processor and computer-readable storage device storing program instructions that, when executed by the processor, cause the computing device to perform operations comprising:

tracking a target in an environment using sensors;

extracting information from images of the target provided by the sensors;

determining a plurality of confidences corresponding to the information extracted from images of the target, the plurality of confidences including at least one confidence corresponding to at least one primitive event;

determining grounded formulae by instantiating predefined rules using the plurality of confidences;

inferring a complex event corresponding to the target using the grounded formulae; and

providing an output describing the complex event,

wherein;

the predefined rules comprise hard rules and soft rules,

the hard rules comprise a first plurality of rules adapted to set a probability of the complex event to zero when violated,

the soft rules comprise a second plurality of rules adapted to make the complex event less probable, but not impossible, when violated,

the hard rules and soft rules comprise first order predicate logic formulas of a Markov logic network, and

the soft rules are associated with weights representing uncertainty.

30. A surveillance system comprising a computing device comprising a processor and computer-readable storage device storing program instructions that, when executed by the processor, cause the computing device to perform operations comprising:

observing events in relation to a target moving in an environment using one or more cameras;

determining, based on the observed events, information describing the target in the environment, the information including attributes of the target and spatial-temporal interactions of the target in the environment;

determining a plurality of confidences corresponding to the information describing the target, the plurality of confidences including at least one confidence corresponding to at least one primitive event;

determining grounded formulae by instantiating a plurality of rules corresponding to the observed events using the plurality of confidences;

inferring an occurrence of a complex event in the environment corresponding to the target using the grounded formulae; and

providing an output describing the complex event,

wherein;

the predefined rules comprise hard rules and soft rules,

the hard rules comprise a first plurality of rules adapted to set a probability of the complex event to zero when violated,

the soft rules comprise a second plurality of rules adapted to make the complex event less probable, but not impossible, when violated, and

the soft rules are associated with weights representing uncertainty.