Directed placement of data in a redundant data storage system

• US 8,504,571 B2
• Filed: 01/24/2012
• Issued: 08/06/2013
• Est. Priority Date: 04/09/2008
• Status: Active Grant

• Current Assignee: Harmonic Incorporated
• Original Assignee: Harmonic Incorporated
• Inventors: Howe, John, Mitaru, Alexander, Morris, Charles, Wanigasekara-Mohotti, Don Harschadath
• Primary Examiner(s): Breene, John E
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US13/357,524
• Publication Number:
  US 20120191710A1
• Time in Patent Office: 560 Days
• Field of Search: 707/737, 707/626, 707/633, 707/635, 707/637, 707/638, 707/639, 707/E17.089, 707/812, 707/E17.032, 707/999.01, 709/206, 709/226, 709/227, 709/201
• US Class Current: 707/737
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/184 implemented as replicated file system

Abstract

A data processing apparatus, comprising a metadata store storing information about files that are stored in a distributed data storage system, and comprising a class database; one or more processing units; logic configured for receiving and storing in the class database a definition of a class of data storage servers comprising one or more subclasses each comprising one or more server selection criteria; associating the class with one or more directories of the data storage system; in response to a data client storing a data file in a directory, binding the class to the data file, determining and storing a set of identifiers of one or more data storage servers in the system that match the server selection criteria, and providing the set of identifiers to the data client.

Claims

1. A data processing apparatus, comprising:

one or more processors;

a memory;

a metadata store storing information about files that are stored in a distributed data storage system, and comprising a class database;

logic coupled to the processor(s) and to the metadata store and comprising stored sequences of instructions which, when executed by the processor(s), cause;

receiving and storing in the class database a definition of a class of data storage servers comprising one or more server selection criteria and one or more representation counts;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory of the one or more directories, determining a set of data storage servers that match the server selection criteria and storing a plurality of replicas in the data storage servers according to the one or more representation counts;

wherein the one or more representation counts control how many replicas, of data files that map to the class, are stored across the set of data storage servers that match the server selection criteria.

2. The apparatus of claim 1, wherein the class of data storage servers comprises a plurality of subclasses;

wherein each subclass of the plurality of subclasses is associated with at least one server selection criteria and a representation count.

3. The apparatus of claim 2, wherein a first representation count for a first subclass of the plurality of subclasses.

4. The apparatus of claim 2, further comprising sequences of instructions which, when executed by the processor(s), causes determining a replication factor for the data file based on a sum of the representation counts, and storing a number of replicas of the data file in the data storage servers determined by the replication factor.

5. The apparatus of claim 4, further comprising sequences of instructions which, when executed by the processor(s), cause:

receiving an updated definition, determining a second replication factor for the data file based on a sum of the representation counts in the updated definition; and

storage the data file only on one or more second data storage servers that are identified according to the updated server selection criteria, in a plurality of replicas according to the second replication factor.

6. The apparatus of the claim 2, further comprising sequences of instructions which, when executed by the processor(s), causes determining that the plurality of subclasses have a zero reorientation count, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a number of replicas of the data file in the data storage servers determined by the replication factor.

7. A non-transitory computer-readable data storage medium comprising stored sequences of instructions which, when executed by one or more processing unit(s), cause:

receiving and storing, in a class database, a definition of a class of data storage servers comprising one or more server selection criteria and one or more representation counts;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory of the one or more directories, determining a set of data storage servers that match the server selection criteria and storing a plurality of replicas in the data storage servers according to the representation counts;

wherein the one or more representation counts control how many replicas, of data files that map to the class, are stored across the set of data storage servers that match the server selection criteria.

8. The computer-readable data storage medium of claim 7, wherein the class of data storage servers comprises a plurality of subclasses;

wherein each subclass of the plurality of subclasses is associated with at least one server selection criteria and a representation count.

9. The computer-readable data storage medium of claim 8, wherein a first representation count for a first subclass of the plurality of subclasses is a different value than a second representation count for a second subclass of the plurality of subclasses.

10. The computer-readable data storage medium of claim 8, further comprising sequences of instructions which, when executed by the processing unit(s), causes determining a replication factor for the data file based on a sum of the representation counts, and storing a number of replicas of the data file in the data storage server determined by the replication factor.

11. The computer-readable data storage medium of claim 10, further comprising sequences of instructions which, when executed by the processing unit(s), cause:

receiving an updated definition of the plurality of subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining a second replication factor for the data file based on a sum of the representation counts in the updated definition;

storing the data file only on one or more second data storage servers that are identified according to the updated server selection criteria, in a plurality of replicas according to the second replication factor.

12. The computer-readable data storage medium of claim 8, further comprising sequences of instructions which, when executed by processing unit(s), causes determining that the plurality of subclasses have a zero reorientation count, in response to determining that the plurality of subclasses have a zero representation count, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a number of the data file in the data storage servers determined by the replication factor.

13. A computer-implemented method for use in a computer system comprising one or more processors, comprising:

receiving and storing, in a class database, a definition of a class of data storage servers comprising one or more server selection criteria and one or more representation counts;

associating the class with one or more directories of the data storage system;

in response to a data client storing a data file in a directory of the one or more directories,

determining a set of data storage servers that match the server selection criteria and storing a plurality of replicas in the data storage servers according to the one or more representation counts;

wherein the one or more representation counts control how many replicas, of data files that map to the class, are stored across the set of data storage servers that match the serve selection criteria.

14. The method of claim 13, wherein the class of data storage servers comprises a plurality of subclasses;

wherein each subclass of the plurality of subclasses is associated with at least one server selection criteria and a representation count.

15. The method of claim 14, wherein a first representation count for a first subclass of the plurality of subclasses is a different value than a second representation for a second subclass of the plurality of subclasses.

16. The method of claim 14, further comprising determining a replication factor for the data file based on a sum of the representation counts, and storing a number of replicas of the data file in the data storage servers determined by the replication factor.

17. The method of claim 16, further comprising:

receiving an updated definition of the plurality of subclasses comprising one or more updated server selection criteria;

based on the updated definition, determining a second replication factor for the data file based on a sum of the representation counts in the updated definition; and

storing the data file only on one or more second data storage servers that are identified according to the updated server selection criteria, in a plurality of replicas according to the second replication factor.

18. The method of claim 14, further comprising determining that the plurality of subclasses have a zero reorientation count, in response to determining that the plurality of subclasses have a zero representation count, determining a replication factor for the data file based on a replication factor property of the data file that is inherited from a directory in which the data file was created, and storing a number of the data file storage servers determined by the replication factor.