Scalable virtual namespace and semantic policy management for object storage

• US 10,127,266 B2
• Filed: 11/03/2015
• Issued: 11/13/2018
• Est. Priority Date: 11/03/2015
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Mandagere, Nagapramod S., Saxena, Mohit
• Primary Examiner(s): Reyes, Mariela
• Assistant Examiner(s): Harmon, Courtney
• Application Number:
  US14/931,731
• Publication Number:
  US 20170124138A1
• Time in Patent Office: 1,106 Days
• CPC Class Codes: G06F 16/2291 User-Defined Types; Storage management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

In one embodiment, a computer-implemented method for object-granular policy assignment in an object storage environment includes: automatically assigning a plurality of objects stored within the object storage environment to one or more policies; and locating one or more of the objects among a plurality of policy rings within the object storage environment based on semantics of the one or more policies. A virtual namespace maps each object to at least one of the policy rings based on the one or more policies assigned to the object; and the virtual namespace is scalable with a number of the plurality of objects stored within the object storage environment. Corresponding systems and computer program products are also disclosed.

Claims

1. A computer-implemented method for object-granular policy assignment in an object storage environment, the method comprising:

automatically assigning a plurality of objects stored within the object storage environment to one or more policies; and

locating one or more of the objects among a plurality of policy rings within the object storage environment based on semantics of the one or more policies,

wherein a virtual namespace maps each object to at least one of the policy rings based on the one or more policies assigned to the object;

wherein the virtual namespace is scalable with a number of the plurality of objects stored within the object storage environment; and

wherein the locating uses one or more probabilistic data structures; and

wherein the probabilistic data structures provide object-level granularity with respect to assigning policies to the plurality of objects stored within the storage environment.

2. The computer implemented method of claim 1, wherein the one or more probabilistic data structures are characterized by a 100% recall rate.

3. The computer implemented method of claim 1, wherein the one or more probabilistic data structures comprise one or more Bloom filters.

4. The computer implemented method of claim 1, comprising:

using a semantic policy manager to define a scan order for the one or more probabilistic data structures based on semantics of the one or more policies of the object storage environment, the semantics comprising one or more of availability characteristics and performance characteristics.

5. The computer implemented method of claim 1, comprising using a semantic policy manager to define a size of each probabilistic data structure based on semantic characteristics of a policy for which the probabilistic data structure is used within the object storage environment;

wherein the one or more probabilistic data structures comprise a plurality of probabilistic data structures;

wherein the size of each probabilistic data structure is defined so as to produce a series of probabilistic data structures each characterized by a different strength; and

wherein the respective strengths of the plurality of probabilistic data structures are configured to minimize latency, I/O and bandwidth associated with performing lookup operations within the object storage environment.

6. The computer implemented method of claim 1, comprising using a semantic policy manager to:

optimize a false positive rate for each of the probabilistic data structures, each of the probabilistic data structures being optimized to have a different false positive rate; and

define a scan order for the one or more probabilistic data structures based on the different false positive rates.

7. The computer implemented method of claim 1, comprising using a semantic policy manager to:

determine one or more of availability characteristics and performance characteristics of a plurality of physical storage devices within the object storage environment upon which the plurality of objects are stored;

determine at least one differential between the availability characteristics and/or the performance characteristics for the plurality of physical storage devices; and

define a plurality of probabilistic data structures having different false positive rates based on the at least one differential, each probabilistic data structure corresponding to at least one of the policies.

8. The computer implemented method of claim 1, comprising assigning a different of the one or more policies to one or more of the objects without modifying an externally client-visible namespace of the object.

9. The computer implemented method of claim 1, wherein the virtual namespace is communicatively coupled to each of the policy rings via a plurality of logical rings;

and

wherein each of the logical rings is communicatively coupled to one of the policy rings.

10. The computer implemented method of claim 1, wherein one or more probabilistic data structures comprise at least (n−

1) probabilistic data structures; and

wherein n is a number of the plurality of policy rings within the object storage environment.

11. The computer implemented method of claim 1, wherein locating the one or more of the objects among the plurality of policy rings within the object storage environment requires, on average, one lookup operation.

12. A computer program product for object-granular policy assignment in an object storage environment, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program instructions executable by an object storage system to cause the object storage system to perform a method comprising:

automatically assigning a plurality of objects stored within the object storage environment to one or more policies; and

locating one or more of the objects among a plurality of policy rings within the object storage environment based on semantics of the one or more policies;

wherein a virtual namespace maps each object to at least one of the policy rings based on the one or more policies assigned to the object; and

wherein the virtual namespace is scalable with a number of the plurality of objects stored within the object storage environment;

wherein the locating uses one or more probabilistic data structures; and

wherein the probabilistic data structures provide object-level granularity with respect to assigning policies to the plurality of objects stored within the storage environment.

13. The computer program product of claim 12, wherein the one or more probabilistic data structures comprise one or more Bloom filters characterized by a 100% recall rate.

14. The computer program product of claim 12, comprising program instructions executable by an object storage system to cause a semantic policy manager of the object storage system to define:

a scan order for the one or more probabilistic data structures based on semantics of the one or more policies of the object storage environment, the semantics comprising one or more of availability characteristics and performance characteristics; and

a size of each probabilistic data structure based on a policy for which the probabilistic data structure is used within the object storage environment.

15. The computer program product of claim 12, comprising program instructions executable by an object storage system to cause a semantic policy manager of the object storage system to:

optimize a false positive rate for each of the probabilistic data structures using the semantic policy manager, each of the probabilistic data structures being optimized to have a different false positive rate; and

define a scan order for the one or more probabilistic data structures based on the different false positive rates.

16. The computer program product of claim 12, comprising program instructions executable by the object storage environment to cause a semantic policy manager of the object storage environment to:

assign a different of the one or more policies to one or more of the objects without modifying a physical storage location of the object.

17. The computer program product of claim 12, comprising program instructions executable by the object storage environment to cause a semantic policy manager of the object storage environment to:

determine one or more of availability characteristics and performance characteristics of a plurality of physical storage devices within the object storage environment upon which the plurality of objects are stored;

determine at least one differential between the availability characteristics and/or the performance characteristics for the plurality of physical storage devices; and

define a plurality of probabilistic data structures having different false positive rates based on the at least one differential, each probabilistic data structure corresponding to at least one of the policies.

18. A system for object-granular policy assignment in an object storage environment, comprising:

a processor and logic integrated with and/or executable by the processor, the logic being configured to;

automatically assign a plurality of objects stored within the object storage environment to one or more policies utilizing a semantic policy manager;

locate one or more of the objects among a plurality of policy rings within the object storage environment based on semantics of the one or more policies; and

wherein a virtual namespace maps each object to at least one of the policy rings based on the one or more policies assigned to the object;

wherein the virtual namespace is scalable with a number of the plurality of objects stored within the object storage environment;

wherein the locating uses one or more probabilistic data structures; and

wherein the probabilistic data structures provide object-level granularity with respect to assigning policies to the plurality of objects stored within the storage environment.

19. The system of claim 18, comprising logic configured to cause a semantic policy manager to:

use a semantic policy manager to prioritize one or more probabilistic data structures based on semantics of the one or more policies of the object storage environment, and define a scan order for locating the one or more of the objects using the one or more probabilistic data structures, the scan order being based on a priority of each probabilistic data structure;

determine availability characteristics and performance characteristics of a plurality of physical storage devices within the object storage environment upon which the plurality of objects are stored;

determine a plurality of differentials between the availability characteristics and the performance characteristics for each pair of the plurality of physical storage devices;

define a plurality of Bloom filters having different false positive rates based on the plurality of differentials, each Bloom filter corresponding to at least one of the policies;

define a size of each of the Bloom filters based on a policy for which the Bloom Filter is used within the object storage environment;

optimize a false positive rate for each of the Bloom filters using the semantic policy manager, each of the Bloom filters being optimized to have a different false positive rate; and

wherein the scan order is further defined based on the different false positive rates.