Method and system for determining data profiles using block-based methodology

• US 9,690,501 B1
• Filed: 12/02/2014
• Issued: 06/27/2017
• Est. Priority Date: 12/02/2014
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Mandic, Vladimir, Chopra, Shelesh, Kumar, Mv Kiran
• Primary Examiner(s): Chappell, Daniel C
• Application Number:
  US14/558,620
• Time in Patent Office: 938 Days
• Field of Search: 711162
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

Techniques for determining characteristics of data blocks being accessed in a storage system are described herein. According to one embodiment, an input/output (IO) request is received for accessing a first data block of a first file stored in a storage system. The first file is one of the files stored in the storage system and each file contains multiple data blocks. In response to the request, a block-based monitor executed in a memory by a processor accesses a set of monitoring rules to determine whether the first file should be monitored. If so, the block-based monitor captures statistics data associated with the first data block and stores the statistics data of the first data block in a statistics database maintained in a persistent storage device. The statistics database stores statistics data of the data blocks of files monitored and captured based on the set of monitoring rules.

Claims

1. A computer-implemented method for determining characteristics of data blocks stored in a storage system, the method comprising:

receiving an input/output (IO) request for accessing a first data block of a first file stored in a storage system, the first file being one of a plurality of files stored in the storage system, each of the files containing a plurality of data blocks;

in response to the request, accessing, by a block-based monitor executed in a memory by a processor, a set of monitoring rules to determine whether the first file should be monitored, wherein the set of monitoring rules represents a set of monitoring parameters in a rule database;

in response to determining that the first file should be monitored, capturing, at a data block level by the block-based monitor, statistics data associated with the first data block, including determining at least in part a time of access of the first data block, a percentage of block change within a period of time, a level of block-based activities, and a changed block list (CBL) associated with the first data block as the statistics data, and capturing a timestamp of the first data block being accessed;

storing the statistics data of the first data block in a statistics database maintained in a persistent storage device, wherein the statistics database stores statistics data of a plurality of data blocks of a plurality of files monitored and captured based on the set of monitoring rules;

analyzing, by an analysis module executed by the processor, the statistics data stored in the statistics database, including determining accessing patterns of data blocks of the files at the data block level, to generate an analysis result; and

transmitting the analysis result to a remote analytics system over a network, wherein the remote analytics system analyzes analysis results of data blocks being accessed at a plurality of storage systems.

2. The method of claim 1, further comprising transmitting the statistics data of the statistics database to a remote analytics system, wherein the remote analytics system collects statistics data of data blocks being accessed in a plurality of storage systems, wherein the remote analytics system is to analyze the collected statistics data to determine accessing patterns of data blocks of the files at a data block level.

3. The method of claim 1, further comprising:

deduplicating the first file into a plurality of data blocks using a predetermined segmenting algorithm; and

storing the data blocks in the persistent storage device in a deduplicated manner, wherein at least one of the data blocks is referenced by a plurality of files.

4. The method of claim 1, wherein capturing statistics data associated with the first data block comprises determining a changed block list of a plurality of data blocks that have been modified from a second file, wherein the second file is a previous version of the first file.

5. The method of claim 4, wherein the second file was captured as a snapshot of files stored in the storage system at a given point of time.

6. The method of claim 4, further comprising recording the changed block list of data blocks associated with a particular file system of the storage system, which is specified as part of the set of monitoring rules.

7. The method of claim 4, further comprising recording the changed block list of data blocks associated with a particular volume of a file system of the storage system, which is specified as part of the set of monitoring rules.

8. The method of claim 4, further comprising recording the changed block list of data blocks associated with a particular directory of a file system of the storage system, which is specified as part of the set of monitoring rules.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of determining characteristics of data blocks stored in a storage system, the operations comprising:

receiving an input/output (IO) request for accessing a first data block of a first file stored in a storage system, the first file being one of a plurality of files stored in the storage system, each of the files containing a plurality of data blocks;

in response to the request, accessing, by a block-based monitor executed in a memory by a processor, a set of monitoring rules to determine whether the first file should be monitored, wherein the set of monitoring rules represents a set of monitoring parameters in a rule database;

in response to determining that the first file should be monitored, capturing, at a data block level by the block-based monitor, statistics data associated with the first data block, including determining at least in part a time of access of the first data block, a percentage of block change within a period of time, a level of block-based activities, and a changed block list (CBL) associated with the first data block as the statistics data, and capturing a timestamp of the first data block being accessed;

storing the statistics data of the first data block in a statistics database maintained in a persistent storage device, wherein the statistics database stores statistics data of a plurality of data blocks of a plurality of files monitored and captured based on the set of monitoring rules;

analyzing, by an analysis module executed by the processor, the statistics data stored in the statistics database including determining accessing patterns of data blocks of the files at the data block level, to generate an analysis result; and

transmitting the analysis result to a remote analytics system over a network, wherein the remote analytics system analyzes analysis results of data blocks being accessed at a plurality of storage systems.

10. The non-transitory machine-readable medium of claim 9, wherein the operations further comprise transmitting the statistics data of the statistics database to a remote analytics system, wherein the remote analytics system collects statistics data of data blocks being accessed in a plurality of storage systems, wherein the remote analytics system is to analyze the collected statistics data to determine accessing patterns of data blocks of the files at a data block level.

11. The non-transitory machine-readable medium of claim 9, wherein the operations further comprise:

deduplicating the first file into a plurality of data blocks using a predetermined segmenting algorithm; and

storing the data blocks in the persistent storage device in a deduplicated manner, wherein at least one of the data blocks is referenced by a plurality of files.

12. The non-transitory machine-readable medium of claim 9, wherein capturing statistics data associated with the first data block comprises determining a changed block list of a plurality of data blocks that have been modified from a second file, wherein the second file is a previous version of the first file.

13. The non-transitory machine-readable medium of claim 12, wherein the second file was captured as a snapshot of files stored in the storage system at a given point of time.

14. The non-transitory machine-readable medium of claim 12, wherein the operations further comprise recording the changed block list of data blocks associated with a particular file system of the storage system, which is specified as part of the set of monitoring rules.

15. The non-transitory machine-readable medium of claim 12, wherein the operations further comprise recording the changed block list of data blocks associated with a particular volume of a file system of the storage system, which is specified as part of the set of monitoring rules.

16. The non-transitory machine-readable medium of claim 12, wherein the operations further comprise recording the changed block list of data blocks associated with a particular directory of a file system of the storage system, which is specified as part of the set of monitoring rules.

17. A storage system, comprising:

a processor and a memory;

a storage device storing a plurality of files, each of the files containing a plurality of data blocks; and

a block-based monitoring logic coupled to the processor and memory to

receive an input/output (IO) request for accessing a first data block of a first file stored in the storage device,

in response to the request, access a set of monitoring rules maintained in the storage device to determine whether the first file should be monitored, wherein the set of monitoring rules represents a set of monitoring parameters in a rule database,

in response to determining that the first file should be monitored, capture statistics data at a data block level associated with the first data block, wherein capturing the statistics data includes determining at least in part a time of access of the first data block, a percentage of block change within a period of time, a level of block-based activities, and a changed block list (CBL) associated with the first data block as the statistics data, and capturing a timestamp of the first data block being accessed,

store the statistics data of the first data block in a statistics database maintained in the storage device, wherein the statistics database stores statistics data of a plurality of data blocks of a plurality of files monitored and captured based on the set of monitoring rules,

analyze the statistics data stored in the statistics database, wherein analyzing the statistics data stored in the statistics database includes determining accessing patterns of data blocks of the files at the data block level, to generate an analysis result, and

transmit the analysis result to a remote analytics system over a network, wherein the remote analytics system analyzes analysis results of data blocks being accessed at a plurality of storage systems.

18. The system of claim 17, wherein the statistics data of the statistics database is transmitted to a remote analytics system, wherein the remote analytics system collects statistics data of data blocks being accessed in a plurality of storage systems, wherein the remote analytics system is to analyze the collected statistics data to determine accessing patterns of data blocks of the files at a data block level.

19. The system of claim 17, further comprising a deduplication engine to

deduplicate the first file into a plurality of data blocks using a predetermined segmenting algorithm, and

store the data blocks in the persistent storage device in a deduplicated manner, wherein at least one of the data blocks is referenced by a plurality of files.

20. The system of claim 17, wherein capturing statistics data associated with the first data block comprises determining a changed block list of a plurality of data blocks that have been modified from a second file, wherein the second file is a previous version of the first file.

21. The system of claim 20, wherein the second file was captured as a snapshot of files stored in the storage system at a given point of time.

22. The system of claim 20, wherein the block-based monitoring logic is record the changed block list of data blocks associated with a particular file system of the storage system, which is specified as part of the set of monitoring rules.

23. The system of claim 20, wherein the block-based monitoring logic is record the changed block list of data blocks associated with a particular volume of a file system of the storage system, which is specified as part of the set of monitoring rules.

24. The system of claim 20, wherein the block-based monitoring logic is record the changed block list of data blocks associated with a particular directory of a file system of the storage system, which is specified as part of the set of monitoring rules.