Asynchronous methods of data classification using change journals and other data structures

• US 8,442,983 B2
• Filed: 12/23/2010
• Issued: 05/14/2013
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Pawar, Rahul S.
• Primary Examiner(s): Le, Uyen T.
• Application Number:
  US12/978,050
• Publication Number:
  US 20110161327A1
• Time in Patent Office: 873 Days
• Field of Search: None
• US Class Current: 707/741
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Aspects of the present invention are generally concerned with systems and methods for generating a database of metadata that describes system data and storage operations. The database of metadata may be referred to herein as a “metabase.” For example, to generate a metabase, a data agent may traverse a file system to obtain certain characteristics of data managed by the file system while substantially simultaneously detecting and recording change notifications. These actions may be performed even if the actions of the data agent are interrupted one or more times during the traversal of the file system. The data agent may process the characteristics and change notifications to generate and update a metabase. Once formed, the metabase may be consulted to determine changes in system data rather than determining the changes by scanning the data files themselves.

Claims

1. A system for generating a database of metadata that describes data objects, the system comprising:

a computing device including—

a data storage device that stores data, wherein the data includes data objects and metadata that describes the data objects; and

a system that indexes data objects and metadata stored on the data storage device; and

a data agent configured to—

scan a portion of the system that indexes data objects and metadata during one or more time periods, by—

requesting that the system that indexes data objects and metadata provide metadata related to a data object;

receiving the requested metadata; and

storing the received metadata in a database of metadata;

populate one or more interim change tables, by—

registering for a change notification service that provides change notifications of interactions with data objects stored on the data storage device;

receiving a change notification of an interaction with a data object stored on the data storage device; and

updating an interim change table to store information provided by the received change notification in the interim change table;

identify interactions with data objects that are not reflected by the information stored in the one or more interim change tables;

retrieve information regarding the identified interactions with data objects;

update the one or more interim change tables to store the retrieved information; and

update the database of metadata by—

analyzing the information stored in the one or more interim change tables;

generating or providing metadata based on the analyzed information; and

,

adding the generated or provided metadata to the database of metadata;

wherein the data agent is configured to populate the one or more interim change tables substantially simultaneously with scanning the portion of the system that indexes data objects and metadata.

2. The system of claim 1, wherein the data agent is further configured to perform the following for each of multiple data objects stored on the data storage device:

request that the system that indexes data objects and metadata provide metadata related to the data object;

receive the requested metadata; and

store the received metadata in the database of metadata.

3. The system of claim 1, wherein the interactions with data objects that are not reflected by the information stored in the one or more interim change tables include interactions with data objects that occur after the data agent begins scanning the portion of the system that indexes data objects and metadata.

4. The system of claim 1 wherein the system that indexes data objects and metadata is a file system of an operating system for the computing device.

5. A method for generating a database of metadata that describes data objects, the method comprising:

interfacing with a file system associated with data stored on a hardware storage device,

wherein the file system provides access to the data objects and metadata included within the data stored on the hardware storage device, and

wherein the interfacing includes performing, for each of the data objects stored on the hardware storage device—

requesting that the file system provide metadata related to the data object;

receiving the requested metadata; and

storing the received metadata within the database of metadata;

populating one or more data structures, wherein populating one or more data structures includes—

registering for a change notification service;

receiving a change notification, wherein the change notification includes information regarding an interaction with at least one of the data objects; and

updating the one or more data structures to include information provided by the received change notification; and

determining if a time required for interfacing with the file system exceeds a first time period;

if the required time exceeds the first time period, determining if change records were created outside of the required time;

if change records were created outside of the required time, retrieving information regarding the change records and storing the retrieved information in the one or more data structures; and

updating the metadata in the database of metadata to include information stored in the one or more one or more data structures.

6. The method of claim 5, wherein populating the one or more data structures occurs during substantially the same time as interfacing with the file system.

7. The method of claim 5, wherein the change records that were created outside of the required time reflect interactions with data objects that occur after interfacing with the file system.

8. The method of claim 5, wherein the determining if the change records were created outside of the required time is performed by examining the one or more data structures.

9. The method of claim 5, further comprising performing the following for each of multiple data objects stored on the hardware storage device:

requesting that the file system provide metadata related to the data object;

receiving the requested metadata; and

storing the received metadata in the database of metadata.

10. A system for generating a database of metadata, the system comprising:

a first computing device including—

a data storage device storing data; and

a file system that facilitates access to the data stored on the data storage device; and

an agent on a second computing device configured to—

scan the file system for information related to the data stored on the data storage device;

store the information in the database of metadata, wherein the information relates to interactions with data stored on the data storage device;

identify interactions with data stored on the data storage device that are not reflected by the information stored in the database of metadata;

update a data structure with information regarding the interactions with data stored on the data storage device that are not reflected by the information stored in the database of metadata;

access the information stored in the data structure; and

based on the accessed information stored in the data structure, update the database of metadata.

11. The system of claim 10, wherein the agent is further configured to store information in the data structure during substantially the same time as it scans the file system.

12. The system of claim 10, wherein the interactions with data that are not reflected by the information stored in the data structure include interactions with data that have occurred after the agent begins scanning the file system.

13. The system of claim 10, wherein the data stored on the data storage device includes multiple data objects, and wherein the agent is further configured to perform the following for each of multiple data objects:

request that the file system provide metadata related to the data object;

receive the requested metadata; and

store the received metadata in the database of metadata.

14. A non-transitory computer-readable medium having instructions for a method of generating a database of metadata which, if executed by a processor, cause the processor to perform the method, including:

interfacing with a file system associated with data stored on a hardware storage device,

wherein the file system provides access to the data objects and metadata included within the data stored on the hardware storage device, and

wherein the interfacing includes performing, for each of the data objects stored on the hardware storage device—

requesting that the file system provide metadata related to the data object;

receiving the requested metadata; and

storing the received metadata within the database of metadata;

populating one or more data structures, wherein populating one or more data structures includes—

registering for a change notification service;

receiving a change notification, wherein the change notification includes information regarding an interaction with at least one of the data objects; and

updating the one or more data structures to include information provided by the received change notification; and

determining if a time required for interfacing with the file system exceeds a first time period;

if the required time exceeds the first time period, determining if change records were created outside of the required time;

if change records were created outside of the required time, retrieving information regarding the change records and storing the retrieved information in the one or more data structures; and

updating the metadata in the database of metadata to include information stored in the one or more one or more data structures.

15. The computer-readable medium of claim 14, further comprising performing the following for each of multiple data objects stored on the hardware storage device:

requesting that the file system provide metadata related to the data object;

receiving the requested metadata; and

storing the received metadata in the database of metadata.

16. The computer-readable medium of claim 14, wherein populating the one or more data structures occurs during substantially the same time as interfacing with the file system.

17. The computer-readable medium of claim 14, wherein the change records that were created outside of the required time reflect interactions with data objects that occur after interfacing with the file system.

18. The computer-readable medium of claim 14, wherein the determining if the change records were created outside of the required time is performed by examining the one or more data structures.