Expiration of persistent data structures that satisfy search queries
First Claim
Patent Images
1. A method, comprising:
- receiving raw data from one or more sources in an information technology environment;
segmenting the raw data into a plurality of events by determining a beginning and an ending of each event in the plurality of events in the raw data, wherein each event in the plurality of events includes a portion of the raw data, wherein each event represents one or more lines of data among a plurality of lines of data in the raw data;
associating a time stamp with each event in the plurality of events;
indexing each time stamped event in the plurality of events;
creating a plurality of persistent data structures that are used to perform lookups in a search process, store the raw data of each event with its segmentation, and store metadata related to the indexed events;
wherein each persistent data structure in the plurality of persistent data structures is immutable and corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure;
wherein a process periodically wakes up and tests the plurality of persistent data structures to determine whether information associated with a persistent data structure meets expiration criteria, wherein the persistent data structure is tested upon reaching a user defined fill capacity and is not accepting further events;
in response to determining that information associated with the persistent data structure meets the expiration criteria, relocating the persistent data structure to offline storage and moving the persistent data structure out of active status.
1 Assignment
0 Petitions
Accused Products
Abstract
Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search.
116 Citations
18 Claims
-
1. A method, comprising:
-
receiving raw data from one or more sources in an information technology environment; segmenting the raw data into a plurality of events by determining a beginning and an ending of each event in the plurality of events in the raw data, wherein each event in the plurality of events includes a portion of the raw data, wherein each event represents one or more lines of data among a plurality of lines of data in the raw data; associating a time stamp with each event in the plurality of events; indexing each time stamped event in the plurality of events; creating a plurality of persistent data structures that are used to perform lookups in a search process, store the raw data of each event with its segmentation, and store metadata related to the indexed events; wherein each persistent data structure in the plurality of persistent data structures is immutable and corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure; wherein a process periodically wakes up and tests the plurality of persistent data structures to determine whether information associated with a persistent data structure meets expiration criteria, wherein the persistent data structure is tested upon reaching a user defined fill capacity and is not accepting further events; in response to determining that information associated with the persistent data structure meets the expiration criteria, relocating the persistent data structure to offline storage and moving the persistent data structure out of active status. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. An apparatus, comprising:
-
a raw data receiver, implemented at least partially in hardware, that receives raw data from one or more sources in an information technology environment; an event creator, implemented at least partially in hardware, that data segments the raw data into a plurality of events by determining a beginning and an ending of each event in the plurality of events in the raw data, wherein each event in the plurality of events includes a portion of the raw data, wherein each event represents one or more lines of data among a plurality of lines of data in the raw data; a time stamp processor, implemented at least partially in hardware, that associates a time stamp with each event in the plurality of events; an event indexer, implemented at least partially in hardware, that indexes each time stamped event in the plurality of events; a persistent data structure creation device, implemented at least partially in hardware, that creates a plurality of persistent data structures that are used to perform lookups in a search process, store the raw data of each event with its segmentation, and store metadata related to the indexed events; wherein each persistent data structure in the plurality of persistent data structures is immutable and corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fail within a particular time interval corresponding to the particular persistent data structure; a persistent data structure monitoring device, implemented at least partially in hardware, that periodically wakes up and tests the plurality of persistent data structures to determine whether information associated with a persistent data structure meets expiration criteria, wherein the persistent data structure is tested upon reaching a user defined fill capacity and is not accepting further events; wherein the persistent data structure monitoring device determines that information associated with the persistent data structure meets the expiration criteria and relocates the persistent data structure to offline storage and moving the persistent data structure out of active status. - View Dependent Claims (8, 9, 10, 11, 12)
-
-
13. One or more non-transitory computer-readable storage media,
storing one or more sequences of instructions, which when executed by one or more processors cause performance of: -
receiving raw data from one or more sources in an information technology environment; segmenting the raw data into a plurality of events by determining a beginning and an ending of each event in the plurality of events in the raw data, wherein each event in the plurality of events includes a portion of the raw data, wherein each event represents one or more lines of data among a plurality of lines of data in the raw data; associating a time stamp with each event in the plurality of events; indexing each time stamped event in the plurality of events; creating a plurality of persistent data structures that are used to perform lookups in a search process, store the raw data of each event with its segmentation, and store metadata related to the indexed events; wherein each persistent data structure in the plurality of persistent data structures is immutable and corresponds to a specific time interval, wherein events stored in a particular persistent data structure have associated time stamps that fall within a particular time interval corresponding to the particular persistent data structure; wherein a process periodically wakes up and tests the plurality of persistent data structures to determine whether information associated with a persistent data structure meets expiration criteria, wherein the persistent data structure is tested upon reaching a user defined fill capacity and is not accepting further events; in response to determining that information associated with the persistent data structure meets the expiration criteria, relocating the persistent data structure to offline storage and moving the persistent data structure out of active status. - View Dependent Claims (14, 15, 16, 17, 18)
-
Specification