GENERATING EVENT DEFINITIONS BASED ON SPATIAL AND RELATIONAL RELATIONSHIPS

• US 20140279764A1
• Filed: 05/29/2014
• Published: 09/18/2014
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Orions Digital Systems, Inc. (Microsoft Corporation)
• Inventors: Lahr, Nils B.
• Granted Patent: US 8,930,959 B2
• Time in Patent Office: Days
• US Class Current: 706/12
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: G06F 9/5038 considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration scheduling strategies G06F9/4881 and subgroups
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 10/103 Workflow collaboration or project management
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 30/19147 Obtaining sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/1983 Syntactic or structural pattern recognition, e.g. symbolic string recognition

Abstract

Data from one or more sensors is input to a workflow and fragmented to produce HyperFragments. The HyperFragments of input data are processed by a plurality of Distributed Experts, who make decisions about what is included in the HyperFragments or add details relating to elements included therein, producing tagged HyperFragments, which are maintained as tuples in a Semantic Database. Algorithms are applied to process the HyperFragments to create an event definition corresponding to a specific activity. Based on related activity included in historical data and on ground truth data, the event definition is refined to produce a more accurate event definition. The resulting refined event definition can then be used with the current input data to more accurately detect when the specific activity is being carried out.

Claims

1. A dynamic event detection architecture configured as one or more hypernodes and comprising:

(a) a workflow engine that runs one or more workflows employed for processing input data by facilitating logic and state manipulation of the input data based on requirements of a specified task;

(b) a semantic database defined by an ontological model that is related to the specified task and which includes a meaning, rules, and data elements based on the ontological model;

(c) a hyperfragmenter system that processes the input data in real-time, in a workflow, to produce fragments of input data that are self-contained and discrete;

(d) a hyperasset file system joined with the semantic database such that both operate automatically, the hyperasset file system enabling the fragments of input data to be stored and retrieved based on specified criteria;

(e) a plurality of distributed experts that use an application program interface to facilitate review of the fragments of input data by the distributed experts at any point in the workflow and to provide additional information to the fragments of input data;

(f) a plurality of event definitions used by a first rule-based language engine to define events based on the fragments of input data in the workflow or based on relationships defined in the semantic database;

(g) a plurality of situation definitions used by a second rule-based engine to define complex situations to create situational rules; and

(h) a filtration system that applies the situational rules to the workflow to determines if an output of the workflow requires further analysis.

2. The dynamic event detection architecture of claim 1, wherein the filtration system determines if the output should be exported to an external system for further analysis to determine if the output represents a normal activity or an abnormal activity.

3. The dynamic event detection architecture of claim 1, wherein the filtration system determines whether the output represents a normal activity or an abnormal activity, the determination being based on a comparison of the output to historical training data input by the filtration system, the filtration system further refining definition criteria used to identify the abnormal activity by further analysis of related activities associated with events in the historical training data.

4. The dynamic event detection architecture of claim 1, further comprising a plurality of path algorithms that can be triggered by the workflow and can be dependent on previously processed fragments of input data becoming available, the plurality of path algorithms being responsible for running and maintaining a state engine or statistical model system.

5. The dynamic event detection architecture of claim 1, wherein the plurality of distributed experts comprises a plurality of humans.

6. The dynamic event detection architecture of claim 1, wherein the plurality of distributed experts comprises a plurality of computing devices that execute artificial intelligence algorithms.

7. The dynamic event detection architecture of claim 1, wherein the plurality of distributed experts perform at least one of the following functions in regard to a fragment of input data:

(a) determine if the fragment of input data is interesting or not interesting;

(b) determine if a fragment of input data that has been determined to be interesting includes at least one of a plurality of predefined specified elements;

or

(c) adding additional details regarding a type of the at least one of the plurality of predefined specified elements that was determined to be included within the fragment of input data.

8. The dynamic event detection architecture of claim 7, wherein a tagonomy is employed to specify tags applied to the plurality of predefined specified elements included in the fragments of input data.

9. The dynamic event detection architecture of claim 7, wherein the plurality of distributed experts include computing devices that execute artificial intelligence algorithms that perform at least one of the functions and wherein the plurality of distributed experts further include humans who review results of the functions performed by the computing devices that execute the artificial intelligence algorithms to either confirm or modify the results.

10. The dynamic event detection architecture of claim 9, wherein the plurality of event definitions and the plurality of situation definitions employ neural algorithms to refine definitions of events related to the specified task based on ground truth data, historical training data, and on input provided by the humans comprising the plurality of distributed experts who review the results of the functions performed.

11. The dynamic event detection architecture of claim 10, wherein the filtration system uses the definitions of the events related to the specified task that have been refined as the definition criteria, and the fragments of input data that have been tagged and provided with the additional details to identify any abnormal activity evident in the fragments of input data that have been tagged.

12. The dynamic event detection architecture of claim 11, wherein when identifying the abnormal activity, the filtration system applies rules included in the semantic database in analyzing the fragments of input data that have tagged.

13. The dynamic event detection architecture of claim 1, wherein the filtration system employs the situational rules created in the plurality of situation definitions in producing the output of the workflow engine.

14. A method for processing input data in a managed workflow to identify events corresponding to a specified activity, comprising:

(a) fragmenting the input data to produce fragments of input data that are self-contained and discrete;

(b) processing the fragments of input data at one or more nodes, using a plurality of distributed experts, the plurality of distributed experts working at the one or more nodes making determinations about the fragments of input data or adding details to the fragments of input data to produce tagged fragments of input data;

(c) reviewing the tagged fragments of input data to create definitions for the events evident in the tagged fragments of input data; and

(d) determining if the events evident in the tagged fragments of input data likely correspond to the specified activity.

15. The method of claim 14, further comprising providing training data and ground truth data related to events corresponding to the specified activity, for use in determining if the events evident in the tagged fragments of input data likely correspond to the specified activity.

16. The method of claim 15, further comprising refining the definitions of the events using the training data and the ground truth data to create refined definitions.

17. The method of claim 16, further comprising using the refined definitions for the tagged fragments of input data to more accurately identify events in the tagged fragments of input data that correspond to the specified activity.

18. The method of claim 15, further comprising using historical data and the refined definitions to determine if the events evident in the tagged fragments of input data likely correspond to either a normal activity or an abnormal activity.

19. The method of claim 14, wherein processing the fragments of input data using the plurality of distributed experts includes distributing the fragments of input data to a plurality of humans for review.

20. The method of claim 14, wherein processing the fragments of input data using the plurality of distributed experts includes distributing the fragments of input data to a plurality of computing devices that execute artificial intelligence algorithms for review.

21. The method of claim 20, wherein the plurality of distributed experts includes a plurality of humans that confirm or modify results of the review of the fragments of input data produced by the plurality of computing devices.

22. The method of claim 14, wherein the plurality of distributed experts perform functions that include:

(a) determining whether the fragments of input data are interesting or not interesting;

(b) if the fragments of input data are determined to be interesting, determining whether the fragments include one of a plurality of predefined specified elements; and

(c) adding additional details regarding a type of one of the plurality of predefined specified element that was determined to be included within the fragments of input data.

23. The method of claim 22, wherein determining whether the fragments of input data include one of the plurality of predefined specified elements comprises employing Tagonomy for applying tags to the fragments of input data to indicate the predefined specified elements included in the fragments of input data.

24. The method of claim 14, further comprising asserting the fragments of input data as tuples in a semantic database.

25. The method of claim 24, further comprising providing version control for each tuple.

26. The method of claim 14, further comprising executing at least one algorithm to process the fragments of input data.

27. The method of claim 14, further comprising using the managed workflow to access external data, and using the external data to modify the fragments of input data.

28. The method of claim 14, further comprising using the tagged fragments of input data, creating situation definition to generate an event corresponding to specific conditions related to the tagged fragments of input data.

29. A method for processing data used for carrying out a task, comprising:

(a) specifying one or more tagonomies for breaking down the task to facilitate processing of the data at a plurality of workflow nodes;

(b) employing at least one of the one or more tagonomies to provide a logic tree with defined paths and properties associated with the task and thereby linking the one or more tagonomies to an ontology for the task;

(c) associating a plurality of distributed experts with the plurality of workflow nodes, so that at least one of the plurality of distributed experts is assigned to process data at each of the plurality of workflow nodes in accord with the tagonomy specified for that workflow node, where the at least one tagonomy is employed for the task in the processing of data carried out at each workflow node; and

(d) collecting the results from the processing of the data by each of the plurality of distributed experts to complete the task.

30. The method of claim 29, wherein the plurality of distributed experts comprises a plurality of humans.

31. The method of claim 29, wherein the plurality of distributed experts comprises a plurality of computing devices that execute an artificial intelligence software program.