Distributed file system for intelligently managing the storing and retrieval of data

• US 8,706,755 B2
• Filed: 03/19/2010
• Issued: 04/22/2014
• Est. Priority Date: 08/03/2001
• Status: Expired due to Term

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Patel, Sujal M., Mikesell, Paul A., Schack, Darren P.
• Primary Examiner(s): Chen, Susan
• Application Number:
  US12/728,081
• Publication Number:
  US 20100257219A1
• Time in Patent Office: 1,495 Days
• Field of Search: 711/164, 711/113, 714/4, 709/223, 707/770, 707/771, 707/775
• US Class Current: 707/770
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2211/104 Metadata, i.e. metadata associated with RAID systems with parity

Abstract

A distributed file system enables the intelligent storing and retrieval of files among a plurality of storage units. The distributed file system utilizes a metadata data structure to track and manage detailed information about each file, including, for example, the device and block locations of the file'"'"'s data blocks, to permit different levels of replication and/or redundancy. The distributed file system includes a switch to receive a read request for a file and to send the read request to one of the storage units. Each of the plurality of storage units is operable to monitor access to the file, such that a block cache module is operable to perform a scalable number of packet read aheads, wherein the scalable number is based on at least a current read rate and a current latency of a link to access the at least one file.

Claims

1. A distributed file storage system comprising:

a plurality of storage units that communicate over a network, wherein each storage unit includes at least a processor and a storage device;

a file stored on one of the plurality of the storage units;

a switch that is configured to receive a read request for the file and to send the read request to one of the plurality of storage units that stores at least the file or a replicated copy of the file, wherein each of the plurality of storage units is operable to monitor access to the file, such that a block cache module is operable to perform a scaled amount of packet read aheads, and wherein the scaled amount is based on at least a current read rate and a current latency of a link to access the file;

wherein the distributed file storage system is arranged for determining at least one copy of the file to be replicated and determining at least a portion of the plurality of storage units to separately store each replicated copy of the file, wherein monitoring is in real time for a pattern of access and based on the monitored pattern of access indicating a frequent access pattern, modifying storage of data blocks for the file by replicating the data blocks on multiple storage units that creates multiple points of access for at least one of the data blocks for the file; and

wherein the distributed file storage system includes different levels of replication including different types and times of replications for different files.

2. The distributed file storage system of claim 1, wherein each of the plurality of storage units is further operable to monitor a latency to access each copy of the file, and content included in the file.

3. The distributed file storage system of claim 1, further comprises metadata stored on each storage unit is employable to process the read request for the file.

4. The distributed file storage system of claim 1, further comprises a plurality of replicated copies of metadata that are mirrored on at least a portion of the plurality of storage units, wherein the mirrored replicated copies of metadata are employable to process the read request for the file.

5. The distributed file storage system of claim 1, wherein at least one of the plurality of storage units is operable to store at least a data block of the file based on at least one of a size of the file, an importance of the file, an anticipated access rate of the file, available storage capacity at the at least one storage unit, the processor utilization at the at least one storage unit, and utilization of the network by each storage unit.

6. A storage unit for a distributed file storage system, comprising:

a storage device;

an interface for communicating with a plurality of storage units and a switch over a network;

a file that is stored on the storage unit and at least one replicated copy of the file is stored on at least one of the other storage units; and

a processor that is operable to execute instructions that enable actions, including;

enabling the distributed file storage system to include different levels of replication including different types and times of replications for different files;

enabling a switch to receive a read request for the file and to send the read request to at least one of the plurality of storage units that stores at least one of the file or a replicated copy of the file, wherein each of the plurality of storage units is operable to monitor access to the file, such that a block cache module is operable to perform a scaled amount of packet read aheads, wherein the scaling of the amount is based on at least a current read rate and a current latency of a link to access the file;

enabling the distributed file storage system to determine at least one copy of the file to be replicated and to determine at least a portion of the plurality of storage units to store each replicated copy of the file; and

wherein the monitoring of access to the file further comprises monitoring in real time a pattern of access; and

based on the monitored pattern of access indicating a frequent access pattern, modifying storage of data blocks for the file by replicating the data blocks on multiple storage units that creates multiple points of access for at least one of the data blocks for the file.

7. The storage unit of claim 6, wherein each of the plurality of storage units is further operable to monitor a latency to access each copy of the file, and content included in the file.

8. The storage unit of claim 6, further comprises metadata stored on each storage unit is employable to process the read request for the file.

9. The storage unit of claim 6, further comprises a plurality of replicated copies of metadata that are mirrored on at least a portion of the plurality of storage units, wherein the mirrored replicated copies of metadata are employable to process the read request for the file.

10. The storage unit of claim 6, wherein at least one of the plurality of storage units is operable to store at least a data block of the file based on at least one of a size of the file, an importance of the file, an anticipated access rate of the file, available storage capacity at the one storage unit, the processor utilization at the at least one storage unit, and utilization of the network by the plurality of storage units.

11. A processor readable storage media that includes processor readable instructions for enabling actions for a distributed file system to operate over a network, comprising:

enabling communication between a switch and a plurality of storage units;

enabling the distributed file system to include different levels of replication including different types and times of replications for different files;

storing a file on at least one of the plurality of storage units and a replicated copy of the file on at least one of the other storage units;

enabling a read request for the at least one file to be provided by the switch to one of the storage units, wherein each of the plurality of storage units is operable to monitor access to the file, such that a block cache module is operable to perform a scaled amount of packet read aheads, and wherein the scaled amount is based on at least a current read rate and a current latency of a link to access the file;

determining a number of copies of the file to be replicated and determining at least a portion of the plurality of storage units to store each replicated copy of the file; and

wherein the monitoring of access to the file further comprises monitoring in real time a pattern of access; and

based on the monitored pattern of access indicating a frequent access pattern, modifying storage of data blocks for the file by replicating the data blocks on multiple storage units that creates multiple points of access for at least one of the data blocks for the file.

12. The processor readable storage media of claim 11, wherein each of the plurality of storage units is further operable to monitor a latency to access each copy of the file, and content included in the file.

13. The processor readable storage media of claim 11, further comprises metadata stored on each storage unit that is employable to process the read request for the file.

14. The processor readable storage media of claim 11, further comprises a plurality of replicated copies of metadata that are mirrored on at least a portion of the plurality of storage units, wherein the mirrored replicated copies of metadata are employable to process the read request for the file.

15. The processor readable storage media of claim 11, wherein at least one of the plurality of storage units is operable to store at least a data block of the file based on at least one of a size of the file, an importance of the file, an anticipated access rate of the file, available storage capacity at the at least one storage unit, the processor utilization at the at least one storage unit, and utilization of the network by the plurality of storage units.

16. A method for enabling actions for a distributed file system to operate over a network, comprising:

enabling communication between a switch and a plurality of storage units;

enabling the distributed file system to include different levels of replication including different types of replications for different files;

storing a file on at least one of the plurality of storage units and a replicated copy of the file on at least one of the other storage units;

enabling a read request for the at least one file to be provided by the switch to one of the storage units, wherein each of the plurality of storage units is operable to monitor access to the file, such that a block cache module is operable to perform a scaled amount of packet read aheads, and wherein the scaled amount is based on at least a current read rate and a current latency of a link to access the file;

determining a number of copies of the file to be replicated and determining at least a portion of the plurality of storage units to store each replicated copy of the file; and

wherein the monitoring of access to the file further comprises monitoring in real time a pattern of access; and

based on the monitored pattern of access indicating a frequent access pattern, modifying storage of data blocks for the file by replicating the data blocks on multiple storage units that creates multiple points of access for at least one of the data blocks for the file.

17. The method of claim 16, wherein at least one of the plurality of storage units is operable to store at least a data block of the file based on at least one of a size of the file, an importance of the file, an anticipated access rate of the file, available storage capacity at the at least one storage unit, the processor utilization at the at least one storage unit, and utilization of the network by the plurality of storage units.

18. The method of claim 16, wherein each of the plurality of storage units is further operable to monitor a latency to access each copy of the file, and content included in the file.

19. The method of claim 16, further comprises metadata stored on each storage unit that is employable to process the read request for the file.

20. The method of claim 16, further comprises a plurality of replicated copies of metadata that are mirrored on at least a portion of the plurality of storage units, wherein the mirrored replicated copies of metadata are employable to process the read request for the file.