ARCHIVING INDEXED DATA

• US 20160224570A1
• Filed: 01/31/2015
• Published: 08/04/2016
• Est. Priority Date: 01/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Eriksson, Petter, Szeto, Jason, Lin, Elizabeth, Haddadkaveh, Nima, Sharp, Clint, Bitincka, Ledion
• Granted Patent: US 10,152,480 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2452 Query translation

Abstract

Raw data in distributed servers is divided into groups of data called buckets containing raw data that have timestamps that fall within a specific time range. When a bucket becomes inactive a server can archive the bucket to an external storage system. The external storage system containing archived data may be specified in a search query. Archived data from the external storage system is obtained, processed, and a search performed on the processed archived data using the search query.

Claims

1. A method comprising:

receiving raw data from a data source;

parsing the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;

organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

determining that timestamped events are no longer being added to a specific group of events;

in response to determining that timestamped events are no longer being added to a specific group of events, sending data associated with the group of events to an external storage system.

2. The method of claim 1, wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system.

3. The method of claim 1, wherein the sending the data associated with the group of events sends timestamped raw data associated with the group of events to the external storage system.

4. The method of claim 1, wherein the sending the data associated with the group of events sends the timestamped events in the group of events to the external storage system.

5. The method of claim 1, wherein the sending the data associated with the group of events sends event data for each timestamped event in the group of events to the external storage system.

6. The method of claim 1, wherein the external storage system is a Hadoop distributed file system.

7. The method of claim 1, wherein the external storage system is a Hadoop compatible file system.

8. The method of claim 1, wherein the sending step further comprises:

sending the data associated with the group of events to an external storage system after a specified time span has passed after determining that timestamped events are no longer being added to the specific group of events.

9. The method of claim 1, wherein the sending step further comprises:

sending the data associated with the group of events to an external storage system after a specified time span has passed after determining that timestamped events are no longer being added to the specific group of events;

wherein the specified time span is user-defined.

10. The method of claim 1, wherein the sending step further comprises:

sending the data associated with the group of events to an external storage system after a total memory space consumed by all groups of events exceeds a limit.

11. The method of claim 1, further comprising:

receiving a search query, the search query specifying an external storage system storing the data associated with the group of events;

sending a request to the external storage system for the data associated with the group of events;

receiving the data associated with the group of events from the external storage system;

parsing the received data into a plurality of second timestamped events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

12. The method of claim 1, further comprising:

wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system;

receiving a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

sending a request to the external storage system for the raw data associated with the group of events;

receiving the raw data associated with the group of events from the external storage system;

parsing the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

13. The method of claim 1, further comprising:

receiving a path name for the external storage system via a graphical user interface.

14. The method of claim 1, further comprising:

limiting bandwidth used to send the data associated with the group of events to the external storage system by lowering a rate of data being sent to the external storage system.

15. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of:

receiving raw data from a data source;

parsing the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;

organizing the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

determining that timestamped events are no longer being added to a specific group of events;

in response to determining that timestamped events are no longer being added to a specific group of events, sending data associated with the group of events to an external storage system.

16. The non-transitory computer readable storage medium of claim 15, wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system.

17. The non-transitory computer readable storage medium of claim 15, wherein the sending the data associated with the group of events sends event data for each timestamped event in the group of events to the external storage system.

18. The non-transitory computer readable storage medium of claim 15, wherein the external storage system is a Hadoop distributed file system.

19. The non-transitory computer readable storage medium of claim 15, wherein the external storage system is a Hadoop compatible file system.

20. The non-transitory computer readable storage medium of claim 15, wherein the sending step further comprises:

sending the data associated with the group of events to an external storage system after a specified time span has passed after determining that timestamped events are no longer being added to the specific group of events.

21. The non-transitory computer readable storage medium of claim 15, further comprising:

receiving a search query, the search query specifying an external storage system storing the data associated with the group of events;

sending a request to the external storage system for the data associated with the group of events;

receiving the data associated with the group of events from the external storage system;

parsing the received data into a plurality of second timestamped events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

22. The non-transitory computer readable storage medium of claim 15, further comprising:

wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system;

receiving a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

sending a request to the external storage system for the raw data associated with the group of events;

receiving the raw data associated with the group of events from the external storage system;

parsing the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

23. An apparatus comprising:

a raw data receiving subsystem, implemented at least partially in hardware, that receives raw data from a data source;

a parsing subsystem, implemented at least partially in hardware, that parses the raw data into a plurality of timestamped events, each timestamped event in the plurality of timestamped events comprising at least a portion of the parsed raw data;

a organization subsystem, implemented at least partially in hardware, that organizes the plurality of timestamped events into groups of events, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

a determination subsystem, implemented at least partially in hardware, that determines that timestamped events are no longer being added to a specific group of events;

a data sending subsystem, implemented at least partially in hardware, that, in response to determining that timestamped events are no longer being added to a specific group of events, sends data associated with the group of events to an external storage system.

24. The apparatus of claim 23, wherein the data sending subsystem sends raw data associated with the group of events to the external storage system.

25. The apparatus of claim 23, wherein the data sending subsystem sends event data for each timestamped event in the group of events to the external storage system.

26. The apparatus of claim 23, wherein the external storage system is a Hadoop distributed file system.

27. The apparatus of claim 23, wherein the external storage system is a Hadoop compatible file system.

28. The apparatus of claim 23, wherein the data sending subsystem sends the data associated with the group of events to an external storage system after a specified time span has passed after determining that timestamped events are no longer being added to the specific group of events.

29. The apparatus of claim 23, further comprising:

a search query receiving subsystem, implemented at least partially in hardware, that receives a search query, the search query specifying an external storage system storing the data associated with the group of events;

a request subsystem, implemented at least partially in hardware, that sends a request to the external storage system for the data associated with the group of events;

a subsystem, implemented at least partially in hardware, that receives the data associated with the group of events from the external storage system;

a parser subsystem, implemented at least partially in hardware, that parses the received data into a plurality of second timestamped events;

a subsystem, implemented at least partially in hardware, that applies a late binding schema to the plurality of second timestamped events;

a search subsystem, implemented at least partially in hardware, that searches the plurality of second timestamped events using parameters from the search query;

a subsystem, implemented at least partially in hardware, that causes results from the searching the plurality of second timestamped events to be displayed.

30. The apparatus of claim 23, further comprising:

wherein the data sending subsystem sends raw data associated with the group of events to the external storage system;

a search query receiving subsystem, implemented at least partially in hardware, that receives a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

a request subsystem, implemented at least partially in hardware, that sends a request to the external storage system for the raw data associated with the group of events;

a subsystem, implemented at least partially in hardware, that receives the raw data associated with the group of events from the external storage system;

a parser subsystem, implemented at least partially in hardware, that parses the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

a subsystem, implemented at least partially in hardware, that applies a late binding schema to the plurality of second timestamped events;

a search subsystem, implemented at least partially in hardware, that searches the plurality of second timestamped events using parameters from the search query;

a subsystem, implemented at least partially in hardware, that causes results from the searching the plurality of second timestamped events to be displayed.