Rule-based tiered storage system

• US 10,579,281 B2
• Filed: 08/18/2015
• Issued: 03/03/2020
• Est. Priority Date: 08/28/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cherubini, Giovanni, Iliadis, Ilias, Jelitto, Jens, Venkatesan, Vinodh
• Primary Examiner(s): Padmanabhan, Mano
• Assistant Examiner(s): Cheong, Andrew J
• Application Number:
  US14/829,133
• Publication Number:
  US 20160070492A1
• Time in Patent Office: 1,659 Days
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/068 Hybrid storage device
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method for storing a data segment in a storage tier of a storage unit comprising at least two storage tiers includes receiving the data segment to be stored including metadata; receiving metadata of data segments stored in the storage unit; and determining the storage tier to store the received data segment to and a protection level dependent on the metadata received and dependent on the metadata of the received data segment.

Claims

1. A device, comprising:

a processor communicatively coupled to a memory, the processor configured to;

receive a set of relevance classes for tagging data segments;

receive a first data segment including a first metadata to be stored in a storage unit, wherein the storage unit comprises at least two storage tiers;

wherein the first metadata comprises a set of variables;

wherein the set of variables includes a location of the first data segment in a directory structure, a group to which a user of the first data segment belongs, and at least one or more of;

a user of the data segment, a size of the data segment, an access control list for the data segment, a time of generation of the data segment, a time of last modification of the data segment, a time of last access of the data segment, a name of the data segment, an indication of whether structured or unstructured data are contained in the data segment, and an indication of a required data retention time;

receive other metadata for other data segments stored in the storage unit;

determine an event associated with the first data segment, the event comprising an occurrence of one or more pre-defined events;

remove at least one relevance class from the set of relevance classes to create an adjusted set of relevance classes based on the event;

determine an observable comprising two or more variables of the set of variables for the first data segment;

determine, for each value of the observable, a distribution of stored data segments across the adjusted set of relevance classes utilizing an agglomerative information bottleneck algorithm;

assign a relevance class out of the adjusted set of relevance classes to the first data segment based on one or more results of the agglomerative information bottleneck algorithm;

determine a storage tier of the at least two storage tiers to store the first data segment based on the other metadata of the other data segments stored in the storage tier, the relevance class, the first metadata of the first data segment, and a content of the first data segment;

determine a protection level for the first data segment based on one or more of the first metadata of the first data segment, the other metadata of the other data segments stored in the storage unit, and the content of the first data segment; and

storing the first data segment on the storage tier with the protection level, wherein the first data segment is encoded with the protection level by an encoder prior to being stored on the storage tier.

2. A non-transitory computer readable storage medium having computer readable instructions stored thereon that, when executed by a computer, implement a method comprising:

receiving a first data segment including metadata to be stored in a storage unit, wherein the storage unit comprises at least two storage tiers;

wherein the metadata of the first data segment comprises a set of variables;

wherein the set of variables includes a location of the first data segment in a directory structure, a group to which a user of the first data segment belongs, and at least one or more of;

a user of the data segment, a size of the data segment, an access control list for the data segment, a time of generation of the data segment, a time of last modification of the data segment, a time of last access of the data segment, a name of the data segment, an indication of whether structured or unstructured data are contained in the data segment, and an indication of a required data retention time;

receiving other metadata for other data segments stored in the storage unit;

determining an event associated with the first data segment, the event comprising an occurrence of one or more pre-defined events;

removing at least one relevance class from a set of relevance classes to create an adjusted set of relevance classes based on the event;

determining an observable comprising two or more variables of the set of variables for the first data segment;

determining, for each value of the observable, a distribution of stored data segments across the adjusted set of relevance classes utilizing an agglomerative information bottleneck algorithm;

assigning a relevance class out of the adjusted set of relevance classes to the first data segment based on one or more results of the agglomerative information bottleneck algorithm;

determining a storage tier of the at least two storage tiers to store the first data segment based on the other metadata of the other data segments stored in the storage tier, the relevance class, the first metadata of the first data segment, and a content of the first data segment;

determining a protection level to apply to the first data segment based on one or more of the first metadata of the first data segment, the other metadata of the other data segments stored in the storage unit, and the content of the received data segment; and

storing the first data segment on the storage tier with the protection level, wherein the first data segment is encoded with the protection level by an encoder prior to being stored on the storage tier.

3. The storage medium of claim 2, wherein determining the storage tier and the protection level includes identifying a storage tier and a protection level at which a majority of data segments showing same or similar metadata as the first data segment is stored.

4. The storage medium of claim 2, wherein the method further comprises:

determining information about a frequency at which data segments stored in one or more of the at least two storage tiers are accessed;

determining the storage tier for storing the first data segment to be dependent on at least access frequency information received for data segments in the same relevance class;

determining a level of protection for the classified data segment dependent on at least the assigned relevance class; and

storing the classified data segment including the assigned relevance class to the determined storage tier and according to the determined level of protection.