Method and apparatus for implementing high-performance, scaleable data processing and storage systems

• US 8,291,094 B2
• Filed: 03/20/2006
• Issued: 10/16/2012
• Est. Priority Date: 06/30/2000
• Status: Active Grant

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Bopardikar, Raju C., Bast, Jacob Y., Cardone, Gary A., Kaufman, David E., MacEachern, Stuart P., McLeod, Bruce D., Radouch, Zdenek, Stiffler, Jack J., Nolan, James M., Wentworth, James A.
• Primary Examiner(s): Mirza, Adnan
• Application Number:
  US11/384,779
• Publication Number:
  US 20060161678A1
• Time in Patent Office: 2,402 Days
• Field of Search: 709/229, 709/231, 709/238
• US Class Current: 709/229
• CPC Class Codes: G06F 2206/1012 Load balancing
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/5055 considering software capabilities, i.e. software resources associated or available to the machine
• CPC Class Codes: H04L 12/54 Store-and-forward switching systems packet switching systems H04L45/00, H04L47/00
• CPC Class Codes: H04L 41/0695 the faulty arrangement being the maintenance, administration or management system

Abstract

A data system architecture is described that allows multiple processing and storage resources to be connected to multiple clients so as 1) to distribute the clients'"'"' workload efficiently across the available resources; and 2) to enable scaleable expansion, both in terms of the number of clients and in the number of resources. The major features of the architecture are separate, modular, client and resource elements that can be added independently, a high-performance cross-bar data switch interconnecting these various elements, separate serial communication paths for controlling the cross-bar switch settings, separate communication paths for passing control information among the various elements and a resource utilization methodology that enables clients to distribute processing or storage tasks across all available resources, thereby eliminating “hot spots” resulting from uneven utilization of those resources.

Claims

1. A disk-based storage system for providing high-performance, scaleable storage services to a client from a plurality of disks, comprising

a disk interface module connected to the plurality of disks for controlling data stored on the plurality of disks;

a host interface module which receives requests for storage service from the client, each storage service request including a data object to be stored and a data object identifier that identifies that data object and, in response to each storage service request and based on a workload and on relative demands placed on subsets of the plurality of disks instead of a physical location in the plurality of disks, selects a subset of the plurality of disks to provide the requested storage so that data corresponding to the same data object identifier can be transferred to different physical disk locations from request to request in order to distribute the workload across the plurality of disks; and

a switch fabric for temporarily connecting the host interface module to the resource module for providing the storage service to the client.

2. The storage system of claim 1 wherein the switch fabric comprises a control switch fabric optimized for transferring control information and a separate data switch fabric optimized for transferring data.

3. The storage system of claim 2 wherein the request for storage service includes control information and data and wherein the host interface module separates the control information and the data and transfers the data to the selected subset of the plurality of disks over the data switch fabric.

4. The storage system of claim 2 wherein the data switch fabric comprises a non-blocking crossbar switch for data transfer and the control switch fabric comprises an Ethernet switch for control information transfer.

5. The storage system of claim 1 wherein the disk interface module generates preallocation information that preallocates physical storage in the plurality of disks in order to evenly distribute data across the plurality of disks.

6. The storage system of claim 5 wherein the physical storage in the plurality of disks is divided into zones and the disk interface module preallocates selected zones to frequently-accessed data, wherein the selected zones are selected in order to decrease disk access time.

7. The storage system of claim 5 wherein the host interface module logically maps data items to be stored into allocation units preallocated to the host interface module by the disk interface modules.

8. The storage system of claim 1 wherein the host interface module comprises a first data memory and the resource module comprises a second data memory and wherein the first and second data memories temporarily store information transferred between the host interface module and the disk interface module.

9. The storage system of claim 1 further comprising a plurality of host interface modules, each host interface module receiving storage service requests from a plurality of clients.

10. The storage system of claim 1 further comprising a plurality of disk interface modules connected to the plurality of disks.

11. The storage system of claim 10 wherein the host interface module directs writes to disk interface modules of data based on the relative demand placed on those modules without regard to either the contents of associated disks or the location of other copies of the data.

12. The storage system of claim 11 further comprising a metadata module that is connected to the switch fabric and assigns each data object an identifying handle.

13. The storage system of claim 12 wherein the host interface module and the metadata module store metadata information associated with each data object in the plurality of disks separately from, and independently of, data associated with each data object.

14. The storage system of claim 13 wherein the handle contains the location of the metadata in the plurality of disks.

15. The storage system of claim 11 wherein the metadata module is dedicated exclusively to fetching, caching and manipulating file and directory attributes of each data object thereby allowing data paths to be optimized to maximize data flow.

16. The storage system of claim 1 wherein the disk interface module preallocates storage in the plurality of disks in allocation units.

17. The storage system of claim 16 wherein the host interface module selects a subset of the plurality of disks from the preallocated storage.

18. The storage system of claim 16 wherein the host interface module reads a file from the plurality of disks and writes the file to disk in a single allocation unit.

19. The storage system of claim 16 wherein the disk interface module comprises a data memory for temporarily storing a data file that is to be written to disk and a parity generator for generating parity information on the data file stored in the data memory prior to the transfer of the data file to the disk.

20. The storage system of claim 19 wherein data and its associated parity are stored on disk in their entirety in order to avoid reading previously stored data to determine data modifications and re-calculating parity.

21. The storage system of claim 1 wherein the host interface module stores data to disk without regard to where any earlier versions of that data were previously stored and without regard to the file and file system to which the data belongs.

22. The storage system of claim 1 wherein the data to be stored on the plurality of disks is arranged in a plurality of data pages and wherein the storage system comprises a reference counter for maintaining reference counts on each data page so that unused disk space can be readily identified.

23. The storage system of claim 1 in which the modules are interconnected using a high-speed, non-blocking, crossbar switch for transferring data.

24. The storage system of claim 23 in which separate, serial Interprocessor Communication (IPC) channels are used to transfer metadata between pairs of modules, thereby enabling the crossbar switch to be used at maximum efficiency for transferring data and allowing memory elements in the external interface to be partitioned into dedicated incoming and outgoing data buffers.

25. A method for providing high-performance, scaleable storage services to a client from a plurality of disks, comprising

(a) providing a disk interface module connected to the plurality of disks for controlling data stored on the plurality of disks;

(b) providing a host interface module which receives requests for storage service from the client, each storage service request including a data object to be stored and a data object identifier that identifies that data object and, in response to each storage service request and based on a workload and on relative demands placed on subsets of the plurality of disks instead of a physical location in the plurality of disks, selects a subset of the plurality of disks to provide the requested storage so that data corresponding to the same data object identifier can be transferred to different physical disk locations from request to request in order to distribute the workload across the plurality of disks; and

(c) using a switch fabric to temporarily connect the host interface module to the resource module for providing the storage service to the client.

26. The method of claim 25 wherein step (c) comprises:

(c1) using a control switch fabric optimized for transferring control information; and

(c2) using a separate data switch fabric optimized for transferring data.

27. The method of claim 26 wherein the request for storage service includes control information and data and wherein step (b) comprises separating the control information and the data and step (c) comprises transferring the data to the selected subset of the plurality of disks over the data switch fabric.

28. The method of claim 26 wherein step (c1) comprises using a non-blocking crossbar switch for data transfer and step (c2) comprises using an Ethernet switch for control information transfer.

29. The method of claim 25 wherein step (a) comprises generating preallocation information that preallocates physical storage in the plurality of disks in order to evenly distribute data across the plurality of disks.

30. The method of claim 29 wherein the physical storage in the plurality of disks is divided into zones and step (a) further comprises preallocating selected zones to frequently-accessed data, wherein the selected zones are selected in order to decrease disk access time.

31. The method of claim 29 wherein step (b) comprises logically mapping data items to be stored into allocation units preallocated to the host interface module by the disk interface modules.

32. The method of claim 29 wherein the host interface module comprises a first data memory and the disk interface module comprises a second data memory and wherein step (a) comprises using the first data memory to temporarily store information transferred from the host interface module to the disk interface module and step (b) comprises using the second data memory to temporarily store information received by the disk interface module from the host interface module.

33. The method of claim 25 wherein step (b) further comprises providing a plurality of host interface modules, each host interface module receiving storage service requests from a plurality of clients.

34. The method of claim 25 wherein step (a) comprises providing a plurality of disk interface modules connected to the plurality of disks.

35. The method of claim 34 wherein the host interface module directs writes to disk interface modules of data based on the relative demand placed on those modules without regard to either the contents of associated disks or the location of other copies of the data.

36. The method of claim 35 further comprising

(d) providing a metadata module that is connected to the switch fabric and assigns each data object an identifying handle.

37. The method of claim 36 further comprising:

(f) using the host interface module and the metadata module to store metadata information associated with each data object in the plurality of disks separately from, and independently of, data associated with each data object.

38. The method of claim 36 wherein the handle contains the location of the metadata in the plurality of disks.

39. The method of claim 36 wherein the metadata module is dedicated exclusively to fetching, caching and manipulating file and directory attributes of each data object thereby allowing data paths to be optimized to maximize data flow.

40. The method of claim 25 wherein step (a) comprises using the disk interface module to preallocate storage in the plurality of disks in allocation units.

41. The method of claim 40 wherein step (a) comprises using the host interface module to select a subset of the plurality of disks from the preallocated storage.

42. The method of claim 40 wherein step (a) comprises using the host interface module to read a file from the plurality of disks and write the file to disk in a single allocation unit.

43. The method of claim 40 wherein step (a) comprises using a data memory for temporarily storing a data file that is to be written to disk and using a parity generator to generate parity information on the data file stored in the data memory prior to the transfer of the data file to the disk.

44. The method of claim 43 wherein data and its associated parity are stored on disk in their entirety in order to avoid reading previously stored data to determine data modifications and re-calculating parity.

45. The method of claim 25 wherein step (b) comprises using the host interface module to store data to disk without regard to where any earlier versions of that data were previously stored and without regard to the file and file system to which the data belongs.

46. The method of claim 25 wherein the data to be stored on the plurality of disks is arranged in a plurality of data pages and wherein the method further comprises:

(g) maintaining reference counts on each data page so that unused disk space can be readily identified.

47. The method of claim 25 wherein step (c) comprises interconnecting the modules using a high-speed, non-blocking, crossbar switch for transferring data.

48. The method of claim 47 wherein step (c) further comprises using separate, serial Interprocessor Communication (IPC) channels to transfer metadata between pairs of modules, thereby enabling the crossbar switch to be used at maximum efficiency for transferring data and allowing memory elements in the external interface to be partitioned into dedicated incoming and outgoing data buffers.