Method and apparatus for reliable and scalable distribution of data files in distributed networks

• US 6,718,361 B1
• Filed: 04/07/2000
• Issued: 04/06/2004
• Est. Priority Date: 04/07/2000
• Status: Expired due to Term

• Current Assignee: NetApp, Inc.
• Original Assignee: Network Appliance Incorporated (NetApp, Inc.)
• Inventors: Karge, Leroy R., Revsin, Vitaly S., Basani, Vijay R., Crovella, Mark E., LaRosa, Domenic J., Bestavros, Azer, Murach, Lynne M., Mangiapudi, Krishna
• Primary Examiner(s): Barot, Bharat
• Application Number:
  US09/545,067
• Time in Patent Office: 1,460 Days
• Field of Search: 709/200-205, 709/213, 709/223-224, 707/8-10, 707/200-203, 713/155, 713/163
• US Class Current: 709/201
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

The present invention provides a system and apparatus for efficient and reliable, control and distribution of data files or portions of files, applications, or other data objects in large-scale distributed networks. A unique content-management front-end provides efficient controls for triggering distribution of digitized data content to selected groups of a large number of remote computer servers. Transport-layer protocols interact with distribution controllers to automatically determine an optimized tree-like distribution sequence to group leaders selected by network devices at each remote site. Reliable store-and-forward transfer to clusters is accomplished using a unicast protocol in the ordered tree sequence. Once command messages and content arrive at all participating group leaders, local hybrid multicast protocols efficiently and reliably distribute them to the back-end nodes for interpretation and execution. Positive acknowledgement is then sent back to the content manager from each group leader, and the-updated content in each remote device autonomously goes “live” when the content change is locally completed.

Claims

1. A system for distributing information to a plurality of group members connected via a communication network, comprising:

a content control manager (CCM) for processing said information into at least one assignment for a distribution job, and for managing said distribution of said at least one assignment to said plurality of group members;

a set of group leaders, each having a corresponding set of group members and each communicating with said CCM, for forwarding said assignments to other group leaders in said set of group leaders according to commands from said CCM, and for forwarding said assignments to said corresponding set of group members belonging to a set of destinations;

each of said group members being associated with a command interpreter for carrying out said assignments on each group member,

whereby said information is communicated to each of said group members within said set of destinations.

2. The system of claim 1 further comprising a database including:

a copy of said information to be distributed;

a set of content groups each comprising a set of destinations;

a list of group members currently included in at least one of said set of destinations; and

a set of user-specified policies relating to distribution of said information, including a set of scheduling parameters.

3. The system of claim 2 in which said database is further accessible from a workstation on said network, said workstation having a graphical user interface adapted for interactive operation of said database by an operator.

4. The system of claim 1 in which said distribution server further comprises

a scheduler for scheduling a distribution job according to events scheduled in a database,

a communication interface to said network for communicating with at least one of said set of group leaders;

and

a graphical user interface to a database.

5. The system of claim 1 in which said distribution server further comprises

a first version of a source file;

a second version of a source file; and

a file delta generator for comparing said first version with said second version to generate a set of changes between said first and second version;

whereby a set of information for distribution comprises said set of changes.

6. The system of claim 1 in which said at least one assignment is further comprised of:

a set of commands for at least one of said command interpreters; and

if said assignment is for distribution of information, further including a set of said information for distribution.

7. The system of claim 6 in which said assignment further comprises

a header having:

an assignment identifier,

an assignment type selected from a set of assignment types, and

an error detection hash for said set of information for distribution.

8. The system of claim 1 in which said content control manager further comprises

a distribution manager for managing distribution of said at least one assignment including

a list of group leaders corresponding to said set of destinations;

a transmission process for sending a copy of said at least one assignment to each of said group leaders on said list as a set of data packets using a reliable transport protocol;

a completion process for determining when a copy of said assignment has been successfully sent to each of said group leaders on said list; and

a verification process for collecting reports from each group leader and for determining which destination of said set of destinations has received said assignment and successfully carried out said assignment.

9. The system of claim 1 in which each of said group leaders further comprises

a first reliable transport protocol for communicating with said CCM and for communicating with other group leaders;

a second reliable transport protocol for communicating with each of said group members corresponding to said group leader;

a store-and-forward process for receiving an assignment from said CCM, for receiving an address of another group leader of said set of group leaders from said CCM, for sending a copy of said assignment to said other group leader if so commanded by said CCM, and for requesting additional group leader addresses from said CCM until said CCM signals that each of said set of group leaders has received a copy of said assignment;

a distribution process for forwarding a copy of said assignment to each of said corresponding group members and for verifying that each of said group members has successfully handled any commands and information of said assignment, and for generating and sending a report to said CCM according to said handling by each of said group members.

10. The system of claim 9 in which

each of said corresponding group members is connected to said group leader by way of a network supporting a multicast protocol;

and

said second reliable transport protocol is a multicast protocol, whereby each group member in said set of destinations receives a copy of said assignment at substantially the same time as each other group member in said set of destinations.

11. The system of claim 1 in which each of said set of group leaders communicates with its said corresponding set of group members using a reliable multicast protocol, whereby each group member in said set of destinations receives a copy of said assignment at substantially the same time as each other group member in said set of destinations.

12. The system of claim 1 in which each of said group members further comprises

a reliable multicast transport protocol for receiving a copy of said assignment from said corresponding group leader;

a reliable transport protocol for communicating with said corresponding group leader; and

a reporting mechanism for generating and sending a status report to said group leader after receiving and processing said assignment.

13. The system of claim 1 in which at least one of said set of group leaders communicates with said CCM and with other group leaders of said set via TCP/IP over the Internet and at least one of said group members receives said assignment from said corresponding group leader using a reliable multicast protocol over a local area network.

14. A method of distributing information via a communication network, comprising the steps of:

determining a content change in a source file;

determining a set of destination servers for receiving an update;

generating a sequence of update commands for said destination servers;

obtaining a list of clusters for delivery of said sequence to said set of destination servers;

communicating said sequence to each of said clusters on said list;

forwarding said sequence to each destination server within said cluster; and

executing said sequence on at least some of said destination servers.

15. The method of claim 14 further comprising the steps of:

monitoring within each said cluster a status indication of which destination servers properly execute said sequence of commands; and

reporting said monitored status to a designated monitoring center.

16. The method of claim 14 in which said step of determining a content change further comprises the steps of:

generating a list of filenames that occur in both said current version and said present version;

comparing each file, corresponding to each filename in said list, between said prior version and said current version to build a table of file changes according to each filename;

generating a set of directory names and filenames that occur in either said prior version or said current version, but not both; and

reporting said set and said table as a content update.

17. The method of claim 14 in which said step of generating a sequence of update commands further comprises the steps of:

selecting a command set according to a type of destination server for each destination;

identifying any data change in said determination of content change that requires transmission of file data; and

making a list of commands corresponding to each said determined content change, along with associated references to file data for each identified data change;

whereby said list of commands and said associated references are sequenced in a data structure for transmission to each destination server.

18. The method of claim 17 in which said step of making a list of commands further comprises the steps of:

evaluating configuration input to determine which file data must be encrypted or compressed, or both, prior to transmission;

carrying out such encryption or compression, or both, as indicated for said associated file references; and

inserting corresponding decryption or decompression commands, or both, into said list of commands.

19. The method of claim 14 further comprising the steps of:

administratively dividing said set of destination servers into a set of content groups;

determining a cluster identifier corresponding to each destination server; and

nominating a group leader as a communication channel for each said cluster identifier;

whereby a list of clusters is created for each content group and a nominated group leader represents each cluster.

20. The method of claim 14 in which said step of communicating said sequence further comprises the steps of:

(A) nominating a group leader for each said cluster;

(B) ordering said list of group leaders for carrying out a store-and-forward distribution;

(C) sending a notification to a target group leader to obtain a copy of said sequence from a named source;

(D) receiving said notification at said target group leader and requesting said sequence from said named source;

(E) receiving said sequence from said named source and reporting completion of said receiving;

(F) responding to said report by notifying a next target leader determined from said ordered list to obtain a copy of said sequence from a named source; and

(G) repeating steps (C) through (G) for each group leader until said list of group leaders is exhausted.

21. The method of claim 14 in which said step of determining a set of destination servers for receiving an update further comprises the step of identifying destination servers having stale content, wherein any of said set of destination servers were omitted from one or more earlier updates.

22. The method of claim 14 in which said step of determining a set of destination servers for receiving an update further comprises the steps of:

identifying a set of network resources for which load must be reduced; and

notifying a resource manager of said set of network resources to be taken out of service during an update.

23. The method of claim 14 in which said step of determining a set of destination servers for receiving an update further comprises the step of:

managing a load factor on a set of network resources by communicating with a resource manager to determine which of said network resources should be added or removed from service during a particular update;

whereby selected network resources can be taken out of service to receive an update, thereby reducing the load on said network resources.

24. The method of claim 23 in which said network resources are selected from the set of group leader, destination server, and group of destination servers.