Capacity forecasting for a deduplicating storage system

• US 8,751,463 B1
• Filed: 06/30/2011
• Issued: 06/10/2014
• Est. Priority Date: 06/30/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Chamness, Mark
• Primary Examiner(s): Jami, Hares
• Application Number:
  US13/174,504
• Time in Patent Office: 1,076 Days
• Field of Search: 707/692, 707/693, 711/161, 711/162, 711/170, 711/171
• US Class Current: 707/693
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A system for managing a storage system comprises a processor and a memory. The processor is configured to receive storage system information from a deduplicating storage system. The processor is further configured to determine a capacity forecast based at least in part on the storage system information. The processor is further configured to provide a compression forecast. The memory is coupled to the processor and configured to provide the processor with instructions.

Claims

1. A system for managing a storage system, comprising:

a processor configured to;

receive storage system information from a deduplicating storage system, wherein the deduplicating storage system;

breaks each of at least a subset of files in an input data stream into a plurality of segments;

determines, for each of at least a subset of the plurality of segments, whether the segment has been stored previously; and

stores a segment in the event that the segment is determined to have not been previously stored, wherein at least a subset of stored segments is used to reconstruct more than one of the subset of files in the input data stream;

determine a capacity forecast for the deduplicating storage system based at least in part on the storage system information, wherein the storage system information comprises a cumulative compressed size, wherein the cumulative compressed size comprises a physical used space after deduplicating; and

provide the capacity forecast for the deduplicating storage system, wherein providing the capacity forecast is based at least in part on a capacity forecast model validation criteria, wherein the capacity forecast model validation criteria comprise one or more of;

a threshold R squared value, a threshold number of standard deviations of data points from an expected value of a set of data points, and a threshold slope; and

a memory coupled to the processor and configured to provide the processor with instructions.

2. A system as in claim 1, wherein storage system information further comprises one or more of the following:

precompression size, a metadata size, index size, associated date, or time-stamp of the diagnostic data.

3. A system as in claim 1, wherein receiving storage system information comprises receiving a summary of storage system information.

4. A system as in claim 3, wherein the summary of storage information is parsed, and wherein extracted storage system information from parsing is stored.

5. A system as in claim 1, wherein the processor is further configured to determine a statistical measure, and wherein the statistical measure comprises one or more of:

a standard deviation and an average.

6. A system as in claim 1, wherein determining a capacity forecast comprises performing a regression analysis based at least in part on the storage system information, wherein the regression analysis is determined for a set of past storage system information data.

7. A system as in claim 6, wherein the regression analysis comprises one or more of:

a linear regression analysis, a logistic regression analysis, and an ensemble regression analysis.

8. A system as in claim 1, wherein the processor is further configured to provide a notification.

9. A system as in claim 8, wherein the notification comprises a ranked list of included storage systems, wherein the storage systems are included and/or excluded based at least in part on inclusion and exclusion criteria.

10. A system as in claim 9, wherein inclusion criteria comprise a number of days between a first date and a capacity threshold date.

11. A system as in claim 9, wherein exclusion criteria comprise one or more of the following:

a number of days in which total storage capacity used has not changed, a number of days in which storage system information has not been received, a percentage decrease in the percentage of total storage capacity used, and a number of standard deviations.

12. A system as in claim 9, wherein the storage system is ranked based at least in part on a capacity threshold date, wherein the capacity threshold date comprises a forecasted date on which a threshold amount of storage capacity of a storage system will be used.

13. A system as in claim 1, wherein a processor is further configured to provide reporting.

14. A system as in claim 13, wherein reporting comprises one or more of:

a report and graph.

15. A system as in claim 13, wherein reporting is provided based at least in part on a set of storage systems, wherein storage system information is aggregated information across the set of storage systems.

16. A method for managing a storage system, comprising:

receiving storage system information from a deduplicating storage system, wherein the deduplicating storage system;

breaks each of at least a subset of files in an input data stream into a plurality of segments;

determines, for each of at least a subset of the plurality of segments, whether the segment has been stored previously; and

stores a segment in the event that the segment is determined to have not been previously stored, wherein at least a subset of stored segments is used to reconstruct more than one of the subset of files in the input data stream;

determining a capacity forecast for the deduplicating storage system based at least in part on the storage system information, wherein the storage system information comprises a cumulative compressed size, wherein the cumulative compressed size comprises a physical used space after deduplicating; and

providing the capacity forecast for the deduplicating storage system, wherein providing a capacity forecast is based at least in part on a capacity forecast model validation criteria, wherein the capacity forecast model validation criteria comprise one or more of;

a threshold R squared value, a threshold number of standard deviations of data points from an expected value of a set of data points, and a threshold slope.

17. A computer program product for managing a storage system, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving storage system information from a deduplicating storage system, wherein the deduplicating storage system;

breaks each of at least a subset of files in an input data stream into a plurality of segments;

determines, for each of at least a subset of the plurality of segments, whether the segment has been stored previously; and

stores a segment in the event that the segment is determined to have not been previously stored, wherein at least a subset of stored segments is used to reconstruct more than one of the subset of files in the input data stream;

determining a capacity forecast for the deduplicating storage system based at least in part on the storage system information, wherein the storage system information comprises a cumulative compressed size, wherein the cumulative compressed size comprises a physical used space after deduplicating; and

providing the capacity forecast for the deduplicating storage system, wherein providing a capacity forecast is based at least in part on a capacity forecast model validation criteria, wherein the capacity forecast model validation criteria comprise one or more of;

a threshold R squared value, a threshold number of standard deviations of data points from an expected value of a set of data points, and a threshold slope.