Archiving indexed data

• US 10,152,480 B2
• Filed: 01/31/2015
• Issued: 12/11/2018
• Est. Priority Date: 01/31/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Sharp, Clint, Eriksson, Petter, Bitincka, Ledion, Szeto, Jason, Lin, Elizabeth, Haddadkaveh, Nima
• Primary Examiner(s): Ly, Anh
• Application Number:
  US14/611,225
• Publication Number:
  US 20160224570A1
• Time in Patent Office: 1,410 Days
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2452 Query translation

Abstract

Raw data in distributed servers is divided into groups of data called buckets containing raw data that have timestamps that fall within a specific time range. When a bucket becomes inactive a server can archive the bucket to an external storage system. The external storage system containing archived data may be specified in a search query. Archived data from the external storage system is obtained, processed, and a search performed on the processed archived data using the search query.

Claims

1. A method comprising:

organizing a plurality of timestamped events into groups of events, each timestamped event in the plurality of timestamped events including a portion of raw machine data associated with a timestamp, wherein the portion of raw machine data reflects activity of a data source and is produced by the data source, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

storing the groups of events to a field-searchable data store; and

archiving a first group of events and a second group of events by sending data associated with the first group of events and the second group of events to an external storage system that is external to the data store, wherein the first group of events comprises timestamped events having associated timestamps that fall within a first time frame, and the second group of events comprises timestamped events having associated timestamps that fall within a second time frame, wherein the first group of events is archived to the external storage system based, at least in part, on a time span that the first group of events has been stored in the data store.

2. The method of claim 1, wherein the sending the data associated with the first group of events sends raw data associated with the first group of events to the external storage system.

3. The method of claim 1, wherein the sending the data associated with the first group of events sends timestamped raw data associated with the first group of events to the external storage system.

4. The method of claim 1, wherein the sending the data associated with the first group of events sends the timestamped events in the first group of events to the external storage system.

5. The method of claim 1, wherein the sending the data associated with the first group of events sends event data for each timestamped event in the first group of events to the external storage system.

6. The method of claim 1, wherein the external storage system is a Hadoop distributed file system.

7. The method of claim 1, wherein the external storage system is a Hadoop compatible file system.

8. The method of claim 1, wherein the archiving step further comprises:

sending the data associated with the first group of events to the external storage system after a total memory space consumed by all groups of events exceeds a limit.

9. The method of claim 1, further comprising:

receiving a search query, the search query specifying an external storage system storing the data associated with the group of events;

sending a request to the external storage system for the data associated with the group of events;

receiving the data associated with the group of events from the external storage system;

parsing the received data into a plurality of second timestamped events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

10. The method of claim 1, further comprising:

wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system;

receiving a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

sending a request to the external storage system for the raw data associated with the group of events;

receiving the raw data associated with the group of events from the external storage system;

parsing the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

11. The method of claim 1, further comprising:

receiving a path name for the external storage system via a graphical user interface.

12. The method of claim 1, further comprising:

limiting bandwidth used to send the data associated with the first group of events to the external storage system by lowering a rate of data being sent to the external storage system.

13. The method of claim 1, further comprising, prior to archiving the first group of events, determining that the first group of events is to be archived.

14. The method of claim 13, wherein determining that the first group of events is to be archived comprises determining that the first group of events is inactive.

15. The method of claim 13, wherein determining that the first group of events is to be archived comprises determining that the first group of events is to be archived based on a user setting.

16. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of:

organizing a plurality of timestamped events into groups of events, each timestamped event in the plurality of timestamped events including a portion of raw machine data associated with a timestamp, wherein the portion of raw machine data reflects activity of a data source and is produced by the data source, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

storing the groups of events to a field-searchable data store; and

archiving a first group of events and a second group of events by sending data associated with the first group of events and the second group of events to an external storage system that is external to the data store, wherein the first group of events comprises timestamped events having associated timestamps that fall within a first time frame, and the second group of events comprises timestamped events having associated timestamps that fall within a second time frame, wherein the first group of events is archived to the external storage system based, at least in part, on a time span that the first group of events has been stored in the data store.

17. The non-transitory computer readable storage medium of claim 16, wherein the sending the data associated with the first group of events sends raw data associated with the first group of events to the external storage system.

18. The non-transitory computer readable storage medium of claim 16, wherein the sending the data associated with the first group of events sends event data for each timestamped event in the first group of events to the external storage system.

19. The non-transitory computer readable storage medium of claim 16, wherein the external storage system is a Hadoop distributed file system.

20. The non-transitory computer readable storage medium of claim 16, wherein the external storage system is a Hadoop compatible file system.

21. The non-transitory computer readable storage medium of claim 16, further comprising:

receiving a search query, the search query specifying an external storage system storing the data associated with the group of events;

sending a request to the external storage system for the data associated with the group of events;

receiving the data associated with the group of events from the external storage system;

parsing the received data into a plurality of second timestamped events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

22. The non-transitory computer readable storage medium of claim 16, further comprising:

wherein the sending the data associated with the group of events sends raw data associated with the group of events to the external storage system;

receiving a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

sending a request to the external storage system for the raw data associated with the group of events;

receiving the raw data associated with the group of events from the external storage system;

parsing the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

applying a late binding schema to the plurality of second timestamped events;

searching the plurality of second timestamped events using parameters from the search query;

causing results from the searching the plurality of second timestamped events to be displayed.

23. An apparatus comprising:

an organization subsystem, implemented at least partially in hardware comprising a processor and memory, that organizes a plurality of timestamped events into groups of events, each timestamped event in the plurality of timestamped events including a portion of raw machine data associated with a timestamp, wherein the portion of raw machine data reflects activity of a data source and is produced by the data source, wherein timestamped events in a group of events have associated timestamps that fall within a specific time frame;

a storing subsystem, implemented at least partially in hardware, that stores the groups of events to a field-searchable data store; and

a data sending subsystem, implemented at least partially in hardware, that archives a first group of events and a second group of events by sending data associated with the first group of events and the second group of events to an external storage system that is external to the data store, wherein the first group of events comprises timestamped events having associated timestamps that fall within a first time frame, and the second group of events comprises timestamped events having associated timestamps that fall within a second time frame, wherein the first group of events is archived to the external storage system based, at least in part, on a time span that the first group of events has been stored in the data store.

24. The apparatus of claim 23, wherein the data sending subsystem sends raw data associated with the first group of events to the external storage system.

25. The apparatus of claim 23, wherein the data sending subsystem sends event data for each timestamped event in the first group of events to the external storage system.

26. The apparatus of claim 23, wherein the external storage system is a Hadoop distributed file system.

27. The apparatus of claim 23, wherein the external storage system is a Hadoop compatible file system.

28. The apparatus of claim 23, further comprising:

a search query receiving subsystem, implemented at least partially in hardware, that receives a search query, the search query specifying an external storage system storing the data associated with the group of events;

a request subsystem, implemented at least partially in hardware, that sends a request to the external storage system for the data associated with the group of events;

a subsystem, implemented at least partially in hardware, that receives the data associated with the group of events from the external storage system;

a parser subsystem, implemented at least partially in hardware, that parses the received data into a plurality of second timestamped events;

a subsystem, implemented at least partially in hardware, that applies a late binding schema to the plurality of second timestamped events;

a search subsystem, implemented at least partially in hardware, that searches the plurality of second timestamped events using parameters from the search query;

a subsystem, implemented at least partially in hardware, that causes results from the searching the plurality of second timestamped events to be displayed.

29. The apparatus of claim 23, further comprising:

wherein the data sending subsystem sends raw data associated with the group of events to the external storage system;

a search query receiving subsystem, implemented at least partially in hardware, that receives a search query, the search query specifying an external storage system storing the raw data associated with the group of events;

a request subsystem, implemented at least partially in hardware, that sends a request to the external storage system for the raw data associated with the group of events;

a subsystem, implemented at least partially in hardware, that receives the raw data associated with the group of events from the external storage system;

a parser subsystem, implemented at least partially in hardware, that parses the received raw data associated with the group of events into a plurality of second timestamped events, each timestamped event in the plurality of second timestamped events comprising at least a portion of the parsed received raw data associated with the group of events;

a subsystem, implemented at least partially in hardware, that applies a late binding schema to the plurality of second timestamped events;

a search subsystem, implemented at least partially in hardware, that searches the plurality of second timestamped events using parameters from the search query;

a subsystem, implemented at least partially in hardware, that causes results from the searching the plurality of second timestamped events to be displayed.