Adaptive monitoring and processing of new data files and changes to existing data files

• US 10,083,190 B2
• Filed: 08/29/2013
• Issued: 09/25/2018
• Est. Priority Date: 03/30/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bath, Amritpal Singh, Blank, Mitchell Neuman, Patel, Vishal, Sorkin, Stephen Phillip
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): GMAHL, NAVNEET K
• Application Number:
  US14/014,059
• Publication Number:
  US 20140136488A1
• Time in Patent Office: 1,853 Days
• Field of Search: None
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases

Abstract

Embodiments are directed towards a dynamic change evaluation mechanism, whereby items having a detected possible change are scheduled for re-evaluation for possible changes at a higher frequency than items detected to not have previously changed, while those items detected as not to have changed are dynamically scheduled for re-evaluation based on an evaluation backlog that may be in turn based, in part, on a time from when an item is assigned an expiration time to when the item is evaluated. In one embodiment, a possibly changed item may be assigned a new expiration time independent of the evaluation backlog. In another embodiment, if no change is detected, then the item may be assigned a new expiration time as a function of a previous expiration time and on the evaluation backlog.

Claims

1. A method, comprising:

observing a plurality of files in at least a portion of a filesystem for newly added files and changes to existing files;

upon detecting a new file in the at least the portion of the filesystem;

creating a record for the new file;

processing data in the new file; and

setting a time period for the new file that indicates when the observing step is to evaluate whether changes have been made to the new file;

upon detecting a change to an existing file in the at least the portion of the filesystem;

determining if a current size of the existing file is different than a previously saved size for the existing file;

based on determining that the current size of the existing file is different than the previously saved size;

identifying new data in the existing file by comparing a previous end of item identifier associated with the existing file with a current end of item identifier of the existing file, the current end of item identifier is calculated using a number of bytes from the end of the existing file; and

processing the new data in the existing file; and

setting a time period for the existing file that indicates when the observing step is to reevaluate whether changes have been made to the existing file.

2. The method as recited in claim 1, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

setting the time period based on the frequency of updates.

3. The method as recited in claim 1, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

setting the time period based on the frequency of updates, wherein the time period is set to a small value when the frequency of updates is high, and wherein the time period is set to a large value when the frequency of updates is low.

4. The method as recited in claim 1, wherein setting the time period to evaluate the new file for updates further comprises:

setting the time period to evaluate the new file for updates to a default initial value.

5. The method as recited in claim 1, wherein observing the plurality of files in at least the portion of the filesystem includes checking the existing file for changes upon the existing file having a time period to reevaluate the existing file that has lapsed.

6. The method as recited in claim 1, wherein upon detecting a change to an existing file in the at least the portion of the filesystem includes detecting a change in at least a portion of a directory listing.

7. The method as recited in claim 1, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

extending the time period based on a low frequency of updates.

8. An apparatus, comprising:

a file observation subsystem, implemented at least partially in hardware, that observes a plurality of files in at least a portion of a filesystem for newly added files and changes to existing files;

wherein the file observation subsystem, upon detecting a new file in the at least the portion of the filesystem;

creates a record for the new file;

processes data in the new file; and

sets a time period for the new file that indicates when the file observation subsystem is to evaluate whether changes have been made to the new file;

a file processor subsystem, implemented at least partially in hardware, that, upon the file observation subsystem detecting a change to an existing file in the at least the portion of the filesystem;

determines if a current size of the existing file is different than a previously saved size for the existing file;

based on a determination that the current size of the existing file is different than the previously saved size, identifies new data in the existing file by comparing a previous end of item identifier associated with the existing file with a current end of item identifier of the existing file, and processes the new data in the existing file, the current end of item identifier is calculated using a number of bytes from the end of the existing file; and

sets a time period for the existing file that indicates when the file observation subsystem is to reevaluate whether changes have been made to the existing file.

9. The apparatus as recited in claim 8, wherein sets the time period to reevaluate the existing file for new updates further comprises:

determines a frequency of updates for the existing file;

sets the time period based on the frequency of updates.

10. The apparatus as recited in claim 8, wherein sets the time period to reevaluate the existing file for new updates further comprises:

determines a frequency of updates for the existing file;

sets the time period based on the frequency of updates, wherein the time period is set to a small value when the frequency of updates is high, and wherein the time period is set to a large value when the frequency of updates is low.

11. The apparatus as recited in claim 8, wherein sets the time period to evaluate the new file for updates further comprises:

sets the time period to evaluate the new file for updates to a default initial value.

12. The apparatus as recited in claim 8, wherein the file observation subsystem checks the existing file for changes upon the existing file having a time period to reevaluate the existing file that has lapsed.

13. The apparatus as recited in claim 8, wherein the file observation subsystem detects a change in at least a portion of a directory listing.

14. The apparatus as recited in claim 8, wherein sets the time period to reevaluate the existing file for new updates further comprises:

determines a frequency of updates for the existing file;

extends the time period based on a low frequency of updates.

15. A non-transitory computer readable medium, storing software instructions, which when executed by one or more processors cause performance of:

observing a plurality of files in at least a portion of a filesystem for newly added files and changes to existing files;

upon detecting a new file in the at least the portion of the filesystem;

creating a record for the new file;

processing data in the new file; and

setting a time period for the new file that indicates when the observing step is to evaluate whether changes have been made to the new file;

upon detecting a change to an existing file in the at least the portion of the filesystem;

determining if a current size of the existing file is different than a previously saved size for the existing file;

based on determining that the current size of the existing file is different than the previously saved size;

identifying new data in the existing file by comparing a previous end of item identifier associated with the existing file with a current end of item identifier of the existing file, the current end of item identifier is calculated using a number of bytes from the end of the existing file; and

processing the new data in the existing file; and

setting a time period for the existing file that indicates when the observing step is to reevaluate whether changes have been made to the existing file.

16. The non-transitory computer readable medium as recited in claim 15, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

setting the time period based on the frequency of updates.

17. The non-transitory computer readable medium as recited in claim 15, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

setting the time period based on the frequency of updates, wherein the time period is set to a small value when the frequency of updates is high, and wherein the time period is set to a large value when the frequency of updates is low.

18. The non-transitory computer readable medium as recited in claim 15, wherein setting the time period to evaluate the new file for updates further comprises:

setting the time period to evaluate the new file for updates to a default initial value.

19. The non-transitory computer readable medium as recited in claim 15, wherein observing the plurality of files in at least the portion of the filesystem includes checking the existing file for changes upon the existing file having a time period to reevaluate the existing file that has lapsed.

20. The non-transitory computer readable medium as recited in claim 15, wherein upon detecting a change to an existing file in the at least the portion of the filesystem includes detecting a change in at least a portion of a directory listing.

21. The non-transitory computer readable medium as recited in claim 15, wherein setting the time period to reevaluate the existing file for new updates further comprises:

determining a frequency of updates for the existing file;

extending the time period based on a low frequency of updates.