×

Data ingestion module for event detection and increased situational awareness

  • US 9,984,427 B2
  • Filed: 12/02/2014
  • Issued: 05/29/2018
  • Est. Priority Date: 12/02/2013
  • Status: Active Grant
First Claim
Patent Images

1. A computer-implemented method comprising:

  • receiving, by a computer, from a plurality of data sources, a plurality of raw data streams containing one or more events, wherein each respective data source comprises a server publishing a corresponding raw data stream;

    pre-processing, by the computer, the plurality of raw data streams for subsequent processing, wherein pre-processing the plurality of raw data streams comprises;

    responsive to identifying, by the computer, one or more ingestion modules corresponding respectively to a set of one or more raw data streams of the plurality of raw data streams;

    determining, by the computer, whether each respective raw data stream in the set satisfies a set of one or more specific criteria based on comparing each respective raw data stream in the set against one or more data files containing data of one or more types stored in a database, wherein at least one criterion of the one or more specific criteria is selected from the group consisting of;

    type of vocabulary used, data format, limitation of characters, and frequency of new data streams released, and wherein at least one data file of the one or more data files is an event template modeling an event; and

    responsive to identifying, by the computer, a second set of one or more raw data streams not corresponding to the one or more ingestion modules;

    determining, by the computer, whether each respective raw data stream in the second set satisfies a set one or more common criteria based on comparing each respective raw data stream in the second set against one or more data files containing data of one or more types stored in the database, wherein the comparison is based on at least one type of data selected from the group consisting of;

    templates, models, search paths, and definitions, and wherein at least one data file of the one or more data files is a second event template modeling a second event;

    processing, by the computer, each respective raw data stream of the second set as indicating a new event responsive to determining that the respective raw data stream satisfies the common set of criteria,determining, by the computer, whether one or more features in each respective data stream indicated as the new event satisfy one or more event models in a categorization table, based upon the computer comparing the one or more features of each respective data stream as the new event against the one or more event models; and

    responsive to the computer determining that the one or more features from each respective data stream indicated as the new event fail to satisfy at least one event model in at least one categorization table;

    comparing, by the computer, the one or more features from each respective data stream indicated as the new event against one or more uncategorized event models in an uncategorized event table, wherein the uncategorized event table store records associated with new unknown event models; and

    storing, by the computer, the one or more features from each respective data stream indicated as the new event as a new uncategorized event model in the uncategorized event table, in response to determining the one or more features from each respective data stream indicated as the new event fail to satisfy at least one uncategorized event model.

View all claims
  • 2 Assignments
Timeline View
Assignment View
    ×
    ×