Harvesting file system metsdata

• US 8,612,404 B2
• Filed: 10/01/2009
• Issued: 12/17/2013
• Est. Priority Date: 07/30/2002
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: Storediq Incorporated (International Business Machines Corporation)
• Inventors: Bone, Jeff G., Arbilla, Laura, Zoellner, Keith, Might, Bradley, Lee, Peter A., Funderburg, Brett A., Jimenez, A. Paul
• Primary Examiner(s): DWIVEDI, MAHESH H
• Application Number:
  US12/572,116
• Publication Number:
  US 20100088317A1
• Time in Patent Office: 1,538 Days
• Field of Search: 707/694, 707/637, 707/654
• US Class Current: 707/694
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs

Abstract

A harvester is disclosed for harvesting metadata of managed objects (files and directories) across file systems which are generally not interoperable in an enterprise environment. Harvested metadata may include 1) file system attributes such as size, owner, recency; 2) content-specific attributes such as the presence or absence of various keywords (or combinations of keywords) within documents as well as concepts comprised of natural language entities; 3) synthetic attributes such as mathematical checksums or hashes of file contents; and 4) high-level semantic attributes that serve to classify and categorize files and documents. The classification itself can trigger an action in compliance with a policy rule. Harvested metadata are stored in a metadata repository to facilitate the automated or semi-automated application of policies.

Claims

1. A computer program product comprising one or more non-transitory computer readable storage media storing instructions translatable by one or more processors to perform:

accessing network file systems on one or more machines, the network file systems operate under various file system protocols at one or more physical locations in a network environment, the network file systems containing files and directories being managed by a network file system management system, wherein the network file system management system comprises a plurality of discrete components and a plurality of queues, wherein the plurality of discrete components comprises a first component, a second component, and a third component, wherein the plurality of queues comprises a first queue and a second queue, and wherein the first component, the second component, and the third component operate coordinate by way of the first queue and the second queue;

the first component collecting file system metadata of the files and directories from the network file systems;

the first component transforming the collected file system metadata in real time into metadata records having a common representation;

the first component storing the metadata records in the first queue, each metadata record of the metadata records comprises a set of attributes associated with a file or directory residing on the network file systems;

for each metadata record read from the first queue, the second component synthesizing or calculating one or more attributes from the collected file system metadata to improve the set of attributes associated with the metadata record;

the second component storing the metadata record with the improved set of attributes in the second queue;

the third component reading one or more of the metadata records from the second queue; and

the third component storing the one or more of the metadata records in a metadata repository according to a scheduling heuristic, the metadata repository being accessible by the network file system management system over a network connection.

2. The computer program product of claim 1, wherein the instructions are further translatable by the one or more processors to perform:

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

3. The computer program product of claim 1, wherein the instructions are further translatable by the one or more processors to perform:

retrieving one or more enrichment candidates from the metadata repository;

performing one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

4. The computer program product of claim 1, wherein the instructions are further translatable by the one or more processors to perform:

utilizing one or more metadata records stored in the metadata repository to generate one or more reports; and

storing the one or more reports in a report repository.

5. The computer program product of claim 4, wherein the instructions are further translatable by the one or more processors to perform:

determining, based on information from the metadata repository, the report repository, or a combination thereof, one or more actions to take on managed files and directories residing on the network file systems.

6. The computer program product of claim 5, wherein the one or more actions are triggered automatically, manually, or both.

7. The computer program product of claim 5, wherein the instructions are further translatable by the one or more processors to perform:

applying one or more filters on metadata records in the metadata repository to trigger at least one of the one or more actions on the managed files and directories residing on the network file systems.

8. A method for harvesting file system metadata, comprising:

accessing network file systems on one or more machines, the network file systems operate under various file system protocols at one or more physical locations in a network environment, the network file systems containing files and directories being managed by a network file system management system, wherein the network file system management system comprises a plurality of discrete components and a plurality of queues, wherein the plurality of discrete components comprises a first component, a second component, and a third component, wherein the plurality of queues comprises a first queue and a second queue, and wherein the first component, the second component, and the third component operate coordinate by way of the first queue and the second queue;

the first component collecting file system metadata of the files and directories from the network file systems;

the first component transforming the collected file system metadata in real time into metadata records having a common representation;

the first component storing the metadata records in the first queue, each metadata record of the metadata records comprises a set of attributes associated with a file or directory residing on the network file systems;

for each metadata record read from the first queue, the second component synthesizing or calculating one or more attributes from the collected file system metadata to improve the set of attributes associated with the metadata record;

the second component storing the metadata record with the improved set of attributes in the second queue;

the third component reading one or more of the metadata record from the second queue; and

the third component storing the one or more of the metadata record in a metadata repository according to a scheduling heuristic, the metadata repository being accessible by the network file system management system over a network connection.

9. The method of claim 8, further comprising:

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

10. The method of claim 8, further comprising:

retrieve one or more enrichment candidates from the metadata repository;

perform one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

11. The method of claim 8, further comprising:

utilizing one or more metadata records stored in the metadata repository to generate one or more reports; and

storing the one or more reports in a report repository.

12. The method of claim 11, further comprising:

determining, based on information from the metadata repository, the report repository, or a combination thereof, one or more actions to take on managed files and directories residing on the network file systems.

13. The method of claim 12, further comprising:

applying one or more filters on metadata records in the metadata repository to trigger at least one of the one or more actions on the managed files and directories residing on the network file systems.

14. A system for harvesting file system metadata, the system comprising one or more non-transitory computer readable storage media storing instructions implementing a network file system management system, the network file system management system comprising a plurality of discrete components and a plurality of queues, wherein the plurality of queues comprises a first queue and a second queue, and wherein the plurality of discrete components comprises:

a grazer module executing on the system and capable of;

accessing network file systems on one or more machines, the network file systems operate under various file system protocols at one or more physical locations in a network environment, the network file systems containing files and directories being managed by the network file system management system;

collecting file system metadata of the files and directories from the network file systems;

transforming the collected file system metadata in real time into metadata records having a common representation; and

placing the metadata records in the first queue, each metadata record of the metadata records comprises a set of attributes associated with a file or directory residing on the network file systems;

an improver module executing on the system and capable of;

for each metadata record read from the first queue synthesizing or calculating one or more attributes from the collected file system metadata to improve the set of attributes associated with the metadata record; and

placing the metadata record with the improved set of attributes in the second queue; and

a populator module executing on the system and capable of;

reading one or more of the metadata records from the second queue; and

storing the one or more of the metadata records with the improved set of attributes in a metadata repository according to a scheduling heuristic, the metadata repository being accessible by the network file system management system over a network connection.

15. The system of claim 14, the one or more non-transitory computer readable storage media further comprising instructions for:

a scrubber module executing on the system and capable of;

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

16. The system of claim 14, the one or more non-transitory computer readable storage media further comprising instructions for:

an enricher module executing on the system and capable of;

retrieving one or more enrichment candidates from the metadata repository;

perform one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

17. The system of claim 14, the one or more non-transitory computer readable storage media further comprising instructions for:

an executive module executing on the system and capable of;

determining, based on information from the metadata repository, a report repository, or a combination thereof, one or more actions to take on the managed files and directories residing on the network file systems, wherein the report repository stores one or more reports generated utilizing at least one metadata record stored in the metadata repository.

18. The system of claim 17, the one or more non-transitory computer readable storage media further comprising instructions for:

one or more filters for triggering at least one of the one or more actions on the managed files and directories residing on the network file systems.