Performing anti-virus checks for a distributed filesystem

• US 9,811,662 B2
• Filed: 09/05/2013
• Issued: 11/07/2017
• Est. Priority Date: 05/03/2010
• Status: Active Grant

• Current Assignee: Panzura, Inc.
• Original Assignee: Panzura, Inc.
• Inventors: Sharpe, Richard, Chou, Randy Yen-pang
• Primary Examiner(s): Rahman, SM
• Application Number:
  US14/019,212
• Publication Number:
  US 20140007239A1
• Time in Patent Office: 1,524 Days
• Field of Search: 709217
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 21/561 Virus type analysis

Abstract

The disclosed embodiments disclose techniques that facilitate the process of performing anti-virus checks for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a write request from a client system that seeks to store a target file in the distributed system. A scan is then performed for this target file. For instance, the scan may be an anti-virus scan that ensures that viruses are not spread to the distributed filesystem or the clients of the distributed filesystem.

Claims

1. A computer-implemented method for performing file checks for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers,

wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the cloud storage system, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the file data from the cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients;

upon receiving at a cloud controller a write request from a client system that seeks to store a target file in the distributed filesystem, sending an incremental metadata snapshot containing metadata for the target file from the cloud controller to the other cloud controllers for the distributed filesystem;

wherein a third cloud controller that is associated with scanning operations uses the incremental metadata snapshot to retrieve the target file from the cloud storage system and initiates a scan for the target file, wherein the third cloud controller is co-located with the cloud storage system and geographically distinct and separate from the cloud controller receiving the target file and the client system;

wherein the third cloud controller determines that the target file has been successfully scanned and sends a subsequent incremental metadata snapshot indicating that the target file has been scanned to the other cloud controllers of the distributed filesystem;

wherein a second cloud controller receiving a client request to access the target file uses the contents of the subsequent incremental metadata snapshot to confirm that the target file has been scanned before allowing access to the target file.

2. The computer-implemented method of claim 1, wherein the scan is an anti-virus scan that ensures that the target file is clean of viruses before allowing any clients of the distributed filesystem to access the target file.

3. The computer-implemented method of claim 2, wherein the metadata for each file in the distributed filesystem includes a clean field that is marked when the file has been scanned and determined to be clean of viruses;

wherein performing the anti-virus scan of the target file involves marking the clean field for the target file if no virus is found for the target file and sending the subsequent incremental metadata snapshot that includes the updated clean field for the target file to the other cloud controllers of the distributed filesystem; and

wherein the second cloud controller confirms that the clean field for the target file is marked before allowing access to the target file.

4. The computer-implemented method of claim 3, wherein performing the anti-virus scan of the target file further comprises:

storing one or more cloud files that contain data blocks for the target file in the cloud storage system;

receiving the incremental metadata snapshot at the third cloud controller, wherein the third cloud controller retrieves the one of more cloud files that contain the target file from the cloud storage system and performs the anti-virus scan for the target file; and

if no virus is found for the target file, sending the subsequent incremental metadata snapshot from the third cloud controller in which the clean field for the target file is marked.

5. The computer-implemented method of claim 4,

wherein the third cloud controller is co-located with the cloud storage system and an anti-virus service in a cloud computing environment;

wherein the third cloud controller downloads the target file by downloading the one or more cloud files containing the data blocks for the target file from the cloud storage system;

wherein the third cloud controller transfers the target file to the anti-virus service; and

wherein after performing the anti-virus scan for the target file the anti-virus service sends the results for the anti-virus scan to the third cloud controller.

6. The computer-implemented method of claim 5, wherein the anti-virus scan of the target file is initiated at the time that the target file is written to the distributed filesystem.

7. The computer-implemented method of claim 5,

wherein the anti-virus scan of the target file is initiated on the third cloud controller at the time that the target file is requested from the cloud storage system via the second cloud controller;

and

wherein the anti-virus scan executes in parallel with the download to the second cloud controller of the one or more cloud files that contain the data blocks for the target file from the cloud storage system, wherein co-locating the third cloud controller with the cloud storage system facilitates performing the anti-virus scan quickly and delivering the subsequent incremental metadata snapshot that includes the anti-virus scan results for the target file to the second cloud controller in the same timeframe that the requested cloud files have been decrypted and cached on the second cloud controller.

8. The computer-implemented method of claim 5,

wherein a first anti-virus scan of the target file is initiated at the time that the target file is written to the distributed filesystem;

wherein the metadata for the target file includes a version field that tracks the version of the virus definitions that was used to perform the first anti-virus scan; and

wherein upon receiving a subsequent access request for the target file, the cloud controller detects that an updated set of virus definitions are newer than the tracked version in the version field, initiates a subsequent anti-virus scan of the target file using the updated set of virus definitions, and restricts client access to the target file until at least one of a local anti-virus check or another subsequent incremental metadata update from the third cloud controller indicates that the target file is clean.

9. The computer-implemented method of claim 5,

wherein a first anti-virus scan of the target file is initiated at the time that the target file is written to the distributed filesystem;

and

wherein a subsequent anti-virus scan of the target file is initiated when the anti-virus service receives an updated set of virus definitions.

10. The computer-implemented method of claim 9, wherein subsequent anti-virus scans for the distributed filesystem are configured to selectively re-scan a subset of the distributed filesystem.

11. The computer-implemented method of claim 10, wherein the subset of the distributed filesystem that is re-scanned is selected based on the file type for a file, the owner of the file, and the location of the file in the distributed filesystem hierarchy.

12. The computer-implemented method of claim 5,

wherein the clean field is organized in a manner that allows sub-portions of the target file that have already been scanned to be marked as clean;

and

wherein the second cloud controller already serves portions of the target file that have been marked as clean to a requesting client while the anti-virus service is still scanning other portions of the target file.

13. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for performing anti-virus checks for a distributed filesystem, the method comprising:

collectively managing the data of the distributed filesystem using two or more cloud controllers, wherein collectively managing the data comprises;

storing the data for the distributed filesystem in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers,

wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the cloud storage system, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the file data from the cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients;

upon receiving at a cloud controller a write request from a client system that seeks to store a target file in the distributed filesystem, sending an incremental metadata snapshot containing metadata for the target file from the cloud controller to the other cloud controllers for the distributed filesystem;

wherein a third cloud controller that is associated with scanning operations uses the incremental metadata snapshot to retrieve the target file from the cloud storage system and initiates a scan for the target file, wherein the third cloud controller is co-located with the cloud storage system and geographically distinct and separate from the cloud controller receiving the target file and the client system;

wherein the third cloud controller determines that the target file has been successfully scanned and sends a subsequent incremental metadata snapshot indicating that the target file has been scanned to the other cloud controllers of the distributed filesystem;

wherein a second cloud controller receiving a client request to access the target file uses the contents of the subsequent incremental metadata snapshot to confirm that the target file has been scanned before allowing access to the target file.

14. The non-transitory computer-readable storage medium of claim 13,

wherein the metadata for each file in the distributed filesystem includes a clean field that is marked when the file has been scanned and determined to be clean of viruses;

wherein performing the anti-virus scan of the target file involves marking the clean field for the target file if no virus is found for the target file and sending the subsequent incremental metadata snapshot that includes the updated clean field for the target file to the other cloud controllers of the distributed filesystem; and

wherein the second cloud controller confirms that the clean field for the target file is marked before allowing access to the target file.

15. The non-transitory computer-readable storage medium of claim 14, wherein performing the anti-virus scan of the target file further comprises:

storing one or more cloud files that contain data blocks for the target file in the cloud storage system;

receiving the incremental metadata snapshot at the third cloud controller that is associated with anti-virus scanning operations; and

if no virus is found for the target file, sending a subsequent incremental metadata snapshot from the third cloud controller in which the clean field for the target file is marked.

16. The non-transitory computer-readable storage medium of claim 15,

wherein the third cloud controller is co-located with the cloud storage system and an anti-virus service in a cloud computing environment;

wherein the third cloud controller downloads the target file by downloading the one or more cloud files containing the data blocks for the target file from the cloud storage system;

wherein the third cloud controller transfers the target file to the anti-virus service; and

wherein after performing the anti-virus scan for the target file the anti-virus service sends the results for the anti-virus scan to the third cloud controller.

17. The non-transitory computer-readable storage medium of claim 16, wherein the anti-virus scan of the target file is initiated at the time that the target file is written to the distributed filesystem.

18. The non-transitory computer-readable storage medium of claim 16,

wherein the anti-virus scan of the target file is initiated at the time that the target file is accessed by the client request via the second cloud controller;

and

wherein the anti-virus scan executes in parallel with the download of the one or more cloud files that contain the data blocks for the target file from the cloud storage system.

19. The non-transitory computer-readable storage medium of claim 16,

wherein a first anti-virus scan of the target file is initiated at the time that the target file is written to the distributed filesystem;

wherein the metadata for the target file includes a version field that tracks the version of the virus definitions that was used to perform the first anti-virus scan; and

wherein a subsequent anti-virus scan of the target file is initiated if an updated set of virus definitions that are newer than the tracked version are available when the target file is accessed.

20. A cloud controller that initiates anti-virus checks for a distributed filesystem, comprising:

a processor;

a storage mechanism that stores metadata for the distributed filesystem; and

a storage management mechanism;

wherein two or more cloud controllers collectively manage the data of the distributed filesystem, wherein collectively managing the data comprises;

storing the data for the distributed filesystern in a cloud storage system, wherein the cloud controllers cache and ensure data consistency for data stored in the cloud storage system;

maintaining in each cloud controller a metadata hierarchy that reflects the current state of the distributed filesystem, wherein changes to the metadata for the distributed filesystem are synchronized across the cloud controllers, wherein the metadata hierarchy in the cloud controller tracks the location of distributed filesystem data blocks in the cloud storage system and cached distributed filesystem data blocks in the cloud controller, wherein the cloud controller uses the metadata hierarchy to locate and download requested, uncached data blocks in the distributed filesystem from the cloud storage system, wherein the file data for the distributed filesystem is stored in the cloud storage systems, wherein cloud controllers cache in their local storage devices a subset of the data from the cloud storage system that is being actively accessed by each respective cloud controller'"'"'s clients; and

wherein the cloud controller is configured to;

receive an incremental metadata snapshot containing metadata for a new the that has been written to the distributed filesystem by a distinct cloud controller for the distributed filesystem;

use the incremental metadata snapshot to download the new the from the cloud storage system, wherein the cloud controller is co-located with the cloud storage system and geographically distinct and separate from the distinct cloud controller receiving the target file;

transfer the target the to an anti-virus service to initiate an anti-virus scan for the target file; and

upon determining from the anti-virus service that no virus was found for the target file, send a subsequent incremental metadata snapshot that indicates that the target the is dean to the other cloud controllers for the distributed filesystem;

wherein a second cloud controller receiving a client request to access the target file uses the contents of the subsequent incremental metadata snapshot to confirm that the target has been successfully scanned before allowing access to the target file.