File Monitoring

• US 20160117342A1
• Filed: 10/24/2014
• Published: 04/28/2016
• Est. Priority Date: 10/24/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Kylasam, Sharad, Colgate, William King
• Granted Patent: US 9,940,336 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs

Abstract

Various methods and systems for monitoring files in a computer system are provided. In this regard, aspects of the invention facilitate file monitoring without file handle use, as it pertains to file monitoring and tailing, thereby mitigating file handle locking conflicts. In various implementations, information for the monitored files is obtained from the kernel using a filter driver in the I/O path. When the filter driver detects write operations being performed on monitored files, file-write data is copied and placed in a kernel buffer, where it can be pulled by a user mode monitoring process and fed to a monitoring application. As such, there is no need for coordination between the monitoring process and the user mode processes of other third-party applications writing data to monitored files.

Claims

1. A method for monitoring one or more files, the method comprising:

detecting a file-write request for a file of a set of files;

copying one or more blocks of file-write information from the file-write request to a kernel buffer, the file write information corresponding to a file-write event; and

receiving, at a user mode process, a portion of data from the kernel buffer comprising file-write information corresponding to at least one file-write event.

2. The method of claim 1 further comprising providing the received portion of data to a monitoring application.

3. The method of claim 1, wherein the set of files comprises a plurality of files.

4. The method of claim 1 further comprising for each block of file-write information corresponding to one file-write event in the received portion of data, labeling the block of file-write information according to a source of the file-write event.

5. The method of claim 1 further comprising:

for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event; and

providing the labeled block of file-write information to a monitoring application.

6. The method of claim 1 further comprising for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event, wherein labeling the block comprises tagging the block with an identification corresponding to a monitored file targeted by the file-write event.

7. The method of claim 1, wherein each block of file-write information comprises a file name, file offset, and a write-data payload of file-write information corresponding to one file-write event, and wherein a file-write event corresponds to an I/O request to write to a particular file.

8. The method of claim 1, wherein the kernel buffer comprises a first-in-first-out queue, and the copied file-write information is placed in the tail of the queue.

9. The method of claim 1, wherein the kernel buffer comprises a first-in-first-out queue, and the copied file-write information is placed in the tail of the queue, and further comprising:

determining the length of the copied file-write information; and

storing the length of the copied file-write information as part of the file-write information in the tail of the queue.

10. The method of claim 1, wherein the kernel buffer comprises a first-in-first-out queue, and the copied file-write information is placed in the tail of the queue, and further comprising:

determining the combined size of the file name and payload of the file-write information; and

storing the combined size of the file name and payload of the file-write information as part of the file-write information in the tail of the queue.

11. The method of claim 1, wherein the kernel buffer comprises a first-in-first-out queue, and the copied file-write information is placed in the tail of the queue, and further comprising:

determining (a) the length of the copied file-write information and (b) the combined size of the file name and payload of the file-write information; and

storing (a) and (b) as part of the file-write information in the tail of the queue.

12. The method of claim 1, wherein the copied file-write information corresponding to a file-write event comprises at least a file name, file offset, and a write-data payload.

13. The method of claim 1, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, and further comprising performing deduplication on the received portion of data.

14. The method of claim 1, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, and further comprising:

performing deduplication on the received portion of data; and

providing the deduplicated portion of data to a monitoring application.

15. The method of claim 1, further comprising performing deduplication on the received portion of data, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, wherein each block of file-write information comprises a file name, file offset, and a write-data payload, and wherein the performed deduplication is based on a comparison of the file name, file offset, and a hash of the write-data payload for the at least two blocks.

16. The method of claim 1, further comprising performing deduplication on the received portion of data, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, wherein each block of file-write information comprises a file name, file offset, and a write-data payload, wherein the performed deduplication is based on a comparison of the file name, file offset, and a hash of the write-data payload for the at least two blocks, wherein the comparison is based on blocks of file-write information generated within a predetermined timeframe, and wherein a matching file name, file offset, and hash of the write-data payload indicates a duplication.

17. The method of claim 1, wherein the detecting and copying steps are facilitated by a kernel filter driver.

18. The method of claim 1, wherein the detecting and copying steps are facilitated by a kernel filter driver, and further comprising initializing the kernel filter driver based on the set of files, wherein initializing the kernel filter driver comprises providing to the kernel filter driver a listing of file names corresponding to the set of files.

19. The method of claim 1, wherein the detecting and copying steps are facilitated by a kernel filter driver, and further comprising initializing the kernel filter driver based on the set of files, wherein initializing the kernel filter driver comprises providing to the kernel filter driver a listing of file names corresponding to the set of files, and wherein the listing of file names is determined from a configuration file.

20. The method of claim 1, wherein the detecting and copying steps are facilitated by a kernel filter driver, and further comprising initializing the kernel filter driver based on the set of files, wherein initializing the kernel filter driver comprises providing to the kernel filter driver a listing of file names corresponding to the set of files, wherein the listing of file names is determined from a configuration file, and wherein the file names in the listing are translated from a file namespace format to a device namespace format.

21. The method of claim 1, wherein the portion of data from the kernel buffer received at the user mode process is pulled from the kernel buffer by the user mode process.

22. The method of claim 1, wherein the method is performed without the user mode process opening, via a file handle, the file in the set of files.

23. A system for monitoring one or more files, the system comprising:

one or more data processors; and

one or more computer-readable storage media containing instructions which, when executed on the one or more data processors, cause the one or more processors to perform operations including;

detecting a file-write request for a file of a set of files;

copying one or more blocks of one or more blocks of file-write information from the file-write request to a kernel buffer, the file-write information corresponding to a file-write event; and

receiving, at a user mode process, a portion of data from the kernel buffer comprising at file-write information corresponding to at least one file-write event.

24. The system of claim 23, wherein the set of files comprises a plurality of files, and the operations further comprising:

for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event; and

providing the labeled block of file-write information to a monitoring application.

25. The system of claim 23, wherein the set of files comprises a plurality of files, and the operations further comprising:

for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event; and

providing the labeled block of file-write information to a monitoring application, wherein each block of file-write information comprises a file name, file offset, and a write-data payload of file-write information corresponding to one file-write event, wherein a file-write event corresponds to an I/O request to write to a particular file, and wherein labeling the block comprises tagging the block with an identification corresponding to a monitored file targeted by the file-write event.

26. The system of claim 23, the operations further comprising performing deduplication on the received portion of data, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, wherein each block of file-write information comprises a file name, file offset, and a write-data payload, and wherein the performed deduplication is based on a comparison of the file name, file offset, and a hash of the write-data payload for the at least two blocks.

27. One or more computer storage media storing computer-executable instructions that, when executed by a computing device, perform a method for monitoring one or more files, the method comprising:

detecting a file-write request for a file of a set of files;

copying one or more blocks of file-write information from the file-write request to a kernel buffer, the file-write information corresponding to a file-write event; and

receiving, at a user mode process, a portion of data from the kernel buffer comprising at file-write information corresponding to at least one file-write event.

28. The media of claim 27, wherein the set of files comprises a plurality of files, the method further comprising:

for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event; and

providing the labeled block of file-write information to a monitoring application.

29. The media of claim 27, wherein the set of files comprises a plurality of files, the method further comprising:

for each block of file-write information corresponding to one file-write event, in the received portion of data, labeling the block of file-write information according to a source of the file-write event; and

providing the labeled block of file-write information to a monitoring application, wherein each block of file-write information comprises a file name, file offset, and a write-data payload of file-write information corresponding to one file-write event, wherein a file-write event corresponds to an I/O request to write to a particular file, and wherein labeling the block comprises tagging the block with an identification corresponding to a monitored file targeted by the file-write event.

30. The media of claim 27, wherein the method further comprises performing deduplication on the received portion of data, wherein the portion of data from the kernel buffer comprises at least two blocks of file-write information, each block corresponding to one file-write event, wherein each block of file-write information comprises a file name, file offset, and a write-data payload, and wherein the performed deduplication is based on a comparison of the file name, file offset, and a hash of the write-data payload for the at least two blocks.