METHOD AND APPARATUS FOR HARVESTING FILE SYSTEM METADATA

• US 20100088317A1
• Filed: 10/01/2009
• Published: 04/08/2010
• Est. Priority Date: 07/30/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Storediq Incorporated (International Business Machines Corporation)
• Inventors: Arbilla, Laura, Zoellner, Keith, Jimenez, A. Paul, Funderburg, Brett A., Bone, Jeff G., Lee, Peter A., Might, Bradley
• Granted Patent: US 8,612,404 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs

Abstract

A harvester is disclosed for harvesting metadata of managed objects (files and directories) across file systems which are generally not interoperable in an enterprise environment. Harvested metadata may include 1) file system attributes such as size, owner, recency; 2) content-specific attributes such as the presence or absence of various keywords (or combinations of keywords) within documents as well as concepts comprised of natural language entities; 3) synthetic attributes such as mathematical checksums or hashes of file contents; and 4) high-level semantic attributes that serve to classify and categorize files and documents. The classification itself can trigger an action in compliance with a policy rule. Harvested metadata are stored in a metadata repository to facilitate the automated or semi-automated application of policies.

Claims

1. A computer program product comprising one or more computer readable storage media storing instructions translatable by one or more processors to perform:

accessing network file systems at one or more physical locations;

collecting file system metadata from the network file systems, wherein the file system metadata comprises one or more pieces of metadata of interest;

applying one or more user-defined heuristics to the one or more pieces of metadata of interest to generate one or more file system statistics of interest;

associating the one or more pieces of metadata of interest and the one or more file system statistics of interest with an identifier for a harvest; and

storing the harvest in a database.

2. The computer program product of claim 1, wherein the collecting step is performed synchronously or asynchronously with a file system activity.

3. A computer program product comprising one or more computer readable storage media storing instructions translatable by one or more processors to perform:

accessing network file systems at one or more physical locations;

collecting file system metadata from the network file systems;

transforming the collected file system metadata into metadata records having a common representation;

utilizing the collected file system metadata in the metadata records to generate synthetic metadata; and

storing the metadata records with the synthetic metadata in a metadata repository.

4. The computer program product of claim 3, wherein the instructions are further translatable by the one or more processors to perform:

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

5. The computer program product of claim 3, wherein the instructions are further translatable by the one or more processors to perform:

retrieve one or more enrichment candidates from the metadata repository;

perform one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

6. The computer program product of claim 3, wherein the instructions are further translatable by the one or more processors to perform:

utilizing one or more metadata records stored in the metadata repository to generate one or more reports; and

storing the one or more reports in a report repository.

7. The computer program product of claim 6, wherein the instructions are further translatable by the one or more processors to perform:

determining, based on information from the metadata repository, the report repository, or a combination thereof, one or more actions to take on managed files and directories residing on the network file systems.

8. The computer program product of claim 7, wherein the one or more actions are triggered automatically, manually, or both.

9. The computer program product of claim 7, wherein the instructions are further translatable by the one or more processors to perform:

applying one or more filters on metadata records in the metadata repository to trigger at least one of the one or more actions on the managed files and directories residing on the network file systems.

10. A method for harvesting file system metadata, comprising:

accessing network file systems at one or more physical locations;

collecting file system metadata from the network file systems;

transforming the collected file system metadata into metadata records having a common representation;

utilizing the collected file system metadata in the metadata records to generate synthetic metadata; and

storing the metadata records with the synthetic metadata in a metadata repository.

11. The method of claim 10, further comprising:

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

12. The method of claim 10, further comprising:

retrieve one or more enrichment candidates from the metadata repository;

perform one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

13. The method of claim 10, further comprising:

utilizing one or more metadata records stored in the metadata repository to generate one or more reports; and

storing the one or more reports in a report repository.

14. The method of claim 13, further comprising:

determining, based on information from the metadata repository, the report repository, or a combination thereof, one or more actions to take on managed files and directories residing on the network file systems.

15. The method of claim 14, further comprising:

applying one or more filters on metadata records in the metadata repository to trigger at least one of the one or more actions on the managed files and directories residing on the network file systems.

16. A system for harvesting file system metadata, comprising:

a grazer module executing on the system and capable of;

accessing network file systems at one or more physical locations;

collecting file system metadata from the network file systems;

transforming the collected file system metadata into metadata records having a common representation; and

placing the metadata records in a first queue;

an improver module executing on the system and capable of;

reading the metadata records from the first queue;

generating synthetic metadata from the file system metadata in the metadata records; and

placing the metadata records with the synthetic metadata in a second queue; and

a populator module executing on the system and capable of;

reading the metadata records from the second queue; and

storing the metadata records with the synthetic metadata in a metadata repository.

17. The system of claim 16, further comprising:

a scrubber module executing on the system and capable of;

accessing the metadata repository; and

determining whether to keep, deprecate, expire, or garbage collect a metadata record.

18. The system of claim 16, further comprising:

an enricher module executing on the system and capable of;

retrieving one or more enrichment candidates from the metadata repository;

perform one or more desired enrichment operations on the one or more enrichment candidates; and

updating metadata associated with the enrichment candidates in the metadata repository.

19. The system of claim 16, further comprising:

an executive module executing on the system and capable of;

determining, based on information from the metadata repository, a report repository, or a combination thereof, one or more actions to take on managed files and directories residing on the network file systems, wherein the report repository stores one or more reports generated utilizing one or more metadata records stored in the metadata repository.

20. The system of claim 19, further comprising:

one or more filters for triggering at least one of the one or more actions on the managed files and directories residing on the network file systems.