Loosely coupled mass storage computer cluster

• US 5,862,312 A
• Filed: 10/24/1995
• Issued: 01/19/1999
• Est. Priority Date: 10/24/1995
• Status: Expired due to Term

• Current Assignee: SeaChange International Incorporated
• Original Assignee: SeaChange Technology, Inc.
• Inventors: Mann, Bruce E., Ywoskus, John A., McGrath, Edward J., Carlozzi, Michael D., Trasatti, Philip J.
• Primary Examiner(s): DeCady, Albert
• Application Number:
  US08/547,565
• Time in Patent Office: 1,183 Days
• Field of Search: 395/200.01, 395/200.02, 395/200.03, 395/200.08, 395/200.09, 395/441, 395/181, 395/182.03, 395/182.04, 395/182.08, 395/182.09 , 395/200.43, 395/200.46, 395/200.47, 395/200.48, 395/200.69, 395/200.3, 370/58.1, 370/58.3, 370/60, 370/60.1, 370/62, 370/16, 370/16.1, 370/352, 711/114, 711/162
• US Class Current: 714/6.22
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 2211/1045 Nested RAID, i.e. implementing a RAID scheme in another RAID scheme
• CPC Class Codes: H04N 7/17336 Handling of requests in head-ends

Abstract

A method and apparatus redundantly store data, in particular video data objects, in a distributed computer system having at least three processor systems, each processor system being connected in point to point two way channel interconnection with each other processor system. The data is stored in a redundant fashion both at the computer system level as well as the processor system level. Accordingly, the failure of a single processor does not adversely affect the integrity of the data. The computer system can also overlay a switching system connected in a ring fashion for providing a fault tolerance to the failure of a single connected processor system at the switch level. Accordingly, there results a fault tolerant data distribution system.

Claims

1. A method for redundantly storing data in a distributed computer system having at least three processor systems, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising the steps of:

interconnecting each one of said processor systems in a point-to-point two way channel interconnection with each other one of said processor systems; and

storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processor systems.

2. The method of claim 1 wherein said storing step comprises the step of storing data across said processor systems according to a RAID-5 process.

3. The method of claim 2 further comprising the step of storing data at each processor system according to a RAID-5 process.

4. The method of claim 1 further comprising the step of

reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over respective ones of said data channel interconnections to establish a load balance across said processor systems.

5. The method of claim 4 further comprising the steps of

reading data from said computer system, in the presence of a failure of one of said processor systems, said reading step, in the presence of failure, comprising the steps of

reading data from each non-failed processor system storing said data, and

reading redundant data from said non-failed processor systems in place of said data stored at said failed processor system, and

reproducing said data stored at said failed processor system using said redundant data and data read from said non-failed processor systems.

6. The method of claim 5 wherein said data input storing step comprises the step of storing said input data according to a RAID-5 process.

7. The method of claim 6 wherein each said processor system stores data on its associated mass storage sub-system according to a RAID-5 process.

8. The method of claim 5 further comprising the step of

preventing, during the presence of the failure of any of said processing systems, the writing of any data in the mass storage sub-system of any of said processor systems having the failure.

9. The method of claim 1 wherein said storing step comprises the step of

designating one processor system to effect all write functions for said computer system.

10. The method of claim 1 wherein said storing step comprises the steps of

one processor system allocating files for each data input, and

enabling all processor systems to write input data to its associated allocated files.

11. The method of claim 1 wherein said storing step comprises the step of

arbitrating write operations among said processor systems using a distributed lock manager.

12. The method of claim 1 wherein said data input storing step comprises the step of

storing said input data as named fragment files in each said processor system.

13. The method of claim 12 wherein said data input storing step further comprises the step of

storing said input data according to a RAID-5 process.

14. The method of claim 13 further comprising the steps of

adding a processor system to said distributed computer system, and

reconstructing said named fragment files to extend over all the processor systems while said distributed computer system continues to actively deliver stored data as an output stream.

15. The method of claim 14 further comprising the step of

reconstructing a failed processor system by reconstructing only data objects which were written while the processor system was in a failed state.

16. A distributed computer system having at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising

interconnecting channels providing a point-to-point two way channel interconnection from each one of said processor systems to each other one of said processor systems;

and

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored in the at least one mass storage sub-system at each of said processor systems and a portion of a redundant representation of the data stored in the at least one mass storage sub-system at each of said processor systems.

17. The apparatus of claim 16 wherein said storage controllers store data across said processor systems according to a RAID-5 process.

18. The apparatus of claim 17 wherein each said storage controller stores data at each processor system according to a RAID-5 process.

19. The apparatus of claim 18 wherein said data storage controllers store said input data among the processors according to a RAID-5 process.

20. The apparatus of claim 19 wherein each said processor system controller stores data on its associated mass storage according to a RAID-5 process.

21. The apparatus of claim 16 wherein said controllers read data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over respective ones of said interconnecting channels, in a manner that enables said controllers to establish a load balance across said processor systems.

22. The apparatus of claim 21 wherein

said controllers read data from said computer system, in the presence of a failure of one of said processor systems, by reading data from each non-failed processor system storing said data, and by reading redundant data from said non-failed processor systems in place of said data stored at said failed processor system, and

the requesting processor system reproducing said data stored at said failed processor system using said read data and said redundant data.

23. The apparatus of claim 22 wherein said storage controllers prevent, during the presence of a failure of any of said processing systems, writing of any data to the mass storage sub-system of any of said processor systems experiencing a failure.

24. The apparatus of claim 16 wherein

one processor system is designated to effect all write functions for said computer system.

25. The apparatus of claim 16 wherein said one processor system allocates files for each data input, and each processor system is enabled to write input data to its associated allocated files.

26. The apparatus of claim 16 wherein said controllers arbitrate write operations among said processor systems using a distributed lock.

27. The apparatus of claim 16 wherein said storage controllers store said input data as named fragment files in said distributed computer system.

28. The apparatus of claim 27 wherein said storage controllers store said data objects as named fragment files across said processor systems according to a RAID-5 process.

29. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

interconnecting data channels providing a point-to-point two way channel interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors,

a switching circuit having n interruptible input ports, at least n+1 interrupting input ports, and at least n+1 output ports, said n interruptible input ports being connected to a respective primary processor system, each said primary processor system having an output connected to a respective interrupting input port, and

each said processor system connected to two of said switching circuits and able to selectively interrupt one of the n interruptible input ports with the signal information available from a respective one of said processor systems, and

said processor systems, in the event of a failure at one processor system, using a previously unused one of said processor systems, and causing said switching circuit to connect the interruptible input port of the failed processor system to a different processor system, and to replace the output of said failed processor system with the output of said different processor system.

30. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

interconnecting data channels providing a point-to-point two way channel interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors,

(n+1) switched systems, each switched system having at least two control inputs, a first input, a second input, a third input, and a fourth input, and a first output and a second output, each switched system being connected at its second output to a processor system, an interrupting output of said associated processor system being connected to the second input of said connected switched system,

said switched systems being interconnected in a ring structure so that each switched system further has an interruptible input signal connected to the first input, the second input of a first neighbor switched system connected to the third input, and the interruptible input from the other neighbor switched system on the ring connected to the fourth input,

each said switched system having switching circuitry responsive to switch control signals at said control inputs for switching any of its inputs to at least its first output, and for connecting either of its first and fourth inputs to its second output, and

said processor systems for providing said switch control input signals to said switched systems.

31. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

interconnecting data channels providing a point-to-point two way channel interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data is stored at each of said processors,

(n+1) switched systems, each switched system having at least two control inputs, four signal inputs, and two signal outputs, each switched system being connected at one of its outputs to an associated processor system, and an interrupting output of said associated processor system being connected to an input of said connected switched system,

said switched systems being interconnected in a ring structure so that each switched system is connected to a first and a second neighbor switched system,

each said switched system having switching circuitry responsive to switch control signals at said control inputs for selectively switching its inputs to its outputs, and

said processors providing said switch control input signals to said switched systems to enable said switched systems to effectively rotate signal switching functions one position around the ring so that a failed signal processor system can be bypassed and the signals on said n first outputs continue unimpaired.

32. The distributed data delivery system of claim 31 further wherein said processors effectively rotate signal switching functions in either direction around the ring.

33. A method for redundantly storing data in a distributed computer system having at least two processor systems, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising the steps of:

interconnecting each one of said processor systems in a point-to-point two way channel interconnection with each other one of said processor systems;

storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors; and

reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over respective ones of said data channel interconnections, in a manner that establishes a load balance across said processor systems.

34. The method of claim 33 further comprising the step of storing data at each processor system according to a RAID-5 process.

35. A distributed computer system having at least two processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising

interconnecting channels providing a point-to-point two way channel interconnection from each one of said processor systems to each other one of said processor systems;

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process whereby data is stored at each of said processor systems and some of a redundant representation of the data is stored at each of said processors; and

said controllers reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over respective ones of said interconnecting channels, whereby said controllers establish a load balance across said processor systems.

36. The apparatus of claim 35 wherein each said storage controller stores data at each processor system according to a RAID-5 process.

37. A method for redundantly storing data in a distributed computer system having at least three processor systems, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising the steps of:

interconnecting each one of said processor systems through a network for data communications with each other one of said processor systems; and

storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data is stored at each of said processors.

38. The method of claim 37 wherein said storing step comprises the step of storing data across said processor systems according to a RAID-5 process.

39. The method of claim 38 further comprising the step of storing data at each processor system according to a RAID-5 process.

40. The method of claim 37 further comprising the step of

reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over said data communications network to establish a load balance across said processor systems.

41. The method of claim 40 further comprising the steps of

reading data from said computer system, in the presence of a failure of one of said processor systems, said reading step, in the presence of failure, comprising the steps of

reading data from each non-failed processor system storing said data, and

reading redundant data from said non-failed processor systems in place of said data stored at said failed processor system, and

reproducing said data stored at said failed processor system using said redundant data and data read from said non-failed processor systems.

42. The method of claim 41 wherein said data input storing step comprises the step of storing said input data according to a RAID-5 process.

43. The method of claim 42 wherein each said processor system stores data on its associated mass storage sub-system according to a RAID-5 process.

44. The method of claim 41 further comprising the step of

preventing, during the presence of the failure of any of said processing systems, the writing of any data in the mass storage sub-system of any of said processor systems having the failure.

45. The method of claim 37 wherein said storing step comprises the step of

designating one processor system to effect all write functions for said computer system.

46. The method of claim 37 wherein said storing step comprises the steps of

one processor system allocating files for each data input, and

enabling all processor systems to write input data to its associated allocated files.

47. The method of claim 37 wherein said storing step comprises the step of

arbitrating write operations among said processor systems using a distributed lock manager.

48. The method of claim 37 wherein said data input storing step comprises the step of

storing said input data as named fragment files in each said processor system.

49. The method of claim 48 wherein said data input storing step further comprises the step of

storing said input data according to a RAID-5 process.

50. The method of claim 49 further comprising the steps of

adding a processor system to said distributed computer system, and

reconstructing said named fragment files to extend over all the processor systems while said distributed computer system continues to actively deliver stored data as an output stream.

51. The method of claim 50 further comprising the step of

reconstructing a failed processor system by reconstructing only data objects which were written while the processor system was in a failed state.

52. A distributed computer system having at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising

an interconnecting data communications network providing communications from each one of said processor systems to each other one of said processor systems;

and

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored in the at least one mass storage sub-system at each of said processor systems and a portion of a redundant representation of the data stored in the at least one mass storage sub-system at each of said processors.

53. The apparatus of claim 52 wherein said storage controllers store data across said processor systems according to a RAID-5 process.

54. The apparatus of claim 53 wherein each said storage controller stores data at each processor system according to a RAID-5 process.

55. The apparatus of claim 54 wherein said data storage controllers store said input data among the processors according to a RAID-5 process.

56. The apparatus of claim 55 wherein each said processor system controller stores data on its associated mass storage according to a RAID-5 process.

57. The apparatus of claim 52 wherein said controllers read data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over said interconnecting data communications network, in a manner that enables said controllers to establish a load balance across said processor systems.

58. The apparatus of claim 57 wherein said controllers read data from said computer system, in the presence of a failure of one of said processor systems, by reading data from each non-failed processor system storing said data, and by reading redundant data from said non-failed processor systems in place of said data stored at said failed processor system, and

the requesting processor system reproducing said data stored at said failed processor system using said read data and said redundant data.

59. The apparatus of claim 58 wherein said storage controllers prevent, during the presence of a failure of any of said processing systems, writing of any data to the mass storage sub-system of any of said processor systems experiencing a failure.

60. The apparatus of claim 52 wherein one processor system is designated to effect all write functions for said computer system.

61. The apparatus of claim 52 wherein

one processor system allocates files for each data input, and

each processor system is enabled to write input data to its associated allocated files.

62. The apparatus of claim 52 wherein said controllers arbitrate write operations among said processor systems using a distributed lock.

63. The apparatus of claim 52 wherein said storage controllers store said input data as named fragment files in said distributed computer system.

64. The apparatus of claim 63 wherein said storage controllers store said data objects as named fragment files across said processor systems according to a RAID-5 process.

65. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

an interconnecting data communications network providing interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors,

a switching circuit having n interruptible input ports, at least n+1 interrupting input ports, and at least n+1 output ports, said n interruptible input ports being connected to a respective primary processor system, each said primary processor system having an output connected to a respective interrupting input port, and

each said processor system connected to two of said switching circuits and able to selectively interrupt one of the n interruptible input ports with the signal information available from a respective one of said processor systems, and

said processor systems, in the event of a failure at one processor system, using a previously unused one of said processor systems, and causing said switching circuit to connect the interruptible input port of the failed processor system to a different processor system, and to replace the output of said failed processor system with the output of said different processor system.

66. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

an interconnecting data communications network providing interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data is stored at each of said processors,

(n+1) switched systems, each switched system having at least two control inputs, a first input, a second input, a third input, and a fourth input, and a first output and a second output, each switched system being connected at its second output to a processor system, an interrupting output of said associated processor system being connected to the second input of said connected switched system,

said switched systems being interconnected in a ring structure so that each switched system further has an interruptible input signal connected to the first input, the second input of a first neighbor switched system connected to the third input, and the interruptible input from the other neighbor switched system on the ring connected to the fourth input,

each said switched system having switching circuitry responsive to switch control signals at said control inputs for switching any of its inputs to at least its first output, and for connecting either of its first and fourth inputs to its second output, and

said processor systems for providing said switch control input signals to said switched systems.

67. A distributed data delivery system comprising

at least three processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system,

an interconnecting data communications network providing interconnection from each one of said processor systems to each other one of said processor systems,

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors,

(n+1) switched systems, each switched system having at least two control inputs, four signal inputs, and two signal outputs, each switched system being connected at one of its outputs to an associated processor system, and an interrupting output of said associated processor system being connected to an input of said connected switched system,

said switched systems being interconnected in a ring structure so that each switched system is connected to a first and a second neighbor switched system,

each said switched system having switching circuitry responsive to switch control signals at said control inputs for selectively switching its inputs to its outputs, and

said processors providing said switch control input signals to said switched systems to enable said switched systems to effectively rotate signal switching functions one position around the ring so that a failed signal processor system can be bypassed and the signals on said n first outputs continue unimpaired.

68. The distributed data delivery system of claim 67 further wherein said processors effectively rotate signal switching functions in either direction around the ring.

69. A method for redundantly storing data in a distributed computer system having at least two processor systems, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising the steps of:

interconnecting each one of said processor systems through a data communications network for communicating with each other one of said processor systems;

storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors; and

reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over said data communications channel in a manner that establishes a load balance across said processor systems.

70. The method of claim 69 further comprising the step of storing data at each processor system according to a RAID-5 process.

71. A distributed computer system having at least two processor systems for redundantly storing data, each processor system comprising at least one central processing unit and at least one mass storage sub-system, comprising

an interconnecting data channel communications network providing communications from each one of said processor systems to each other one of said processor systems;

a data storage controller at each processor system, said controller storing data input at any one of said processor systems according to a distributed, redundant storage process with data stored at each of said processor systems and a portion of a redundant representation of the data stored at each of said processors with said controllers reading data from said computer system, in the absence of a failure of any of said processor systems, from each of said processor systems over said data communications channel in a manner that establishes a load balance across said processor systems.

72. The apparatus of claim 71 wherein each said storage controller stores data at each processor system according to a RAID-5 process.