Methods and apparatus for declarative log collection

• US 7,743,030 B1
• Filed: 09/29/2006
• Issued: 06/22/2010
• Est. Priority Date: 09/29/2006
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Ninan, Anoop George, Farizon, Boris, Apte, Charuta Vijaykumar, Korolev, Eugenio
• Primary Examiner(s): Trujillo; James
• Assistant Examiner(s): Mina; Fatima P
• Application Number:
  US11/540,184
• Time in Patent Office: 1,362 Days
• Field of Search: 707/102, 707/103, 707/204, 707/205
• US Class Current: 707/648
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04L 67/34 involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

A log collection utility (LCU) operable in a storage area network (SAN) to identify the various types of log files generated in the SAN, compute the location of each of the log files, and collect the log files for consolidation and analysis in a common repository. A designator file enumerates log files for collection. The LCU generates a discovery script for each host, and the hosts generate an identity of each of the log files via execution of the discovery script. The LCU generates a collection script operable to gather the each of the log files. The LCU thus gathers a plurality of log files generated by dissimilar applications and nodes in the SAN via a declarative mechanism that specifies the types of log files via a log file designator, and generates executable scripts that collects the log files in a manner that is agnostic to operating systems, platforms, and directory structures.

Claims

1. A method of retrieving log files comprising:

computing an identity of a set of log files in a network environment including a plurality of nodes, the network environment including a storage area network (SAN) having SAN nodes;

generating, from the identity of the log files, a collection script operable to retrieve each of the identified log files, the collection script generated independently of an operating system of a SAN node;

sending the collection script to the SAN node;

receiving, from the SAN node, each of the log files collected by the collection script;

determining, based on system load, whether to compress collected log files at a host or at a server;

selectively compressing the collected log files at the host or at the server based on the determining;

aggregating, on each host, the collected log files obtained from the collection script;

transmitting, by each host, the collected log files to the server;

identifying categories of log files, categories including at least one of infrastructure, agent or 3rd party applications; and

aggregating transmitted log files from each host at the server in a common repository, the log files arranged by categories.

2. The method of claim 1 wherein computing the identity of the set of log files further comprises:

determining a set of log files types accessible via the network environment

generating a discovery script having commands for discovering each of the log files identified in the set of log files;

sending the discovery script to a SAN node having at least one of the identified log files, the SAN node operable to execute the discovery script; and

receiving, from the SAN node, an identity of each of the log files identified via the executed discovery script.

3. The method of claim 2 further comprising a plurality of SAN nodes, each of the SAN nodes having at least one agent, the agent responsive to the discovery script, and:

sending the discovery script to each of the plurality of SAN nodes;

receiving the identity of each of the log files;

generating a task definition indicative of the collective log files on each of the SAN nodes; and

generating the collection script from the task definition.

4. The method of claim 3 wherein each SAN node further comprises a host having a master agent, the master agent operable to execute the collection script, further comprising:

generating, for each host, a collection script operable to collect the discovered log files corresponding to that host;

executing the generated collection script on the host; and

receiving the collected log files at the server.

5. The method of claim 2 further comprising:

receiving the collection script at each of a plurality of SAN nodes;

executing the script in each of the SAN nodes;

identifying, from the executed script, a set of log files;

zipping the identified log files into an aggregated log file; and

transmitting the aggregated log file back to the initiating host.

6. The method of claim 5 wherein the discovery script is executable on the SAN node, the SAN node operable to generate the identity of the log files from execution of the discovery script, the discovery script independent of the operating system and file system of the SAN node, further comprising:

generating the collection script from the identified set of available log files and the discovery script and the received identity of the log files.

7. The method of claim 6 wherein sending the discovery script further comprises a plurality of discovery scripts, each discovery script corresponding to a particular SAN node, further comprising:

sending the generated discovery scripts to a plurality of SAN nodes, each SAN node having a master agent responsive to the script and operable to retrieve at least one log file.

8. The method of claim 6 wherein generating the discovery script further comprises traversing a list of log types, the list of log types indicative of the location of each of the log files, wherein generating the discovery script includes using a log file designator that describes a manner of locating log files by referencing multiple log files via wildcards and directory specifications, wherein generating the discovery script occurs before generating the collection script.

9. The method of claim 6 wherein identifying the log files comprises enumerating the log files in a list, the list including definitions indicative of the location of each of the log files, wherein the log file list is a task definition file indicative of a location of at least one log file on each of a set of hosts.

14. The method of claim 4, wherein each of the SAN nodes, within the plurality of SAN nodes, contains one or more log files, and wherein at least two of the SAN nodes have different operating systems that store log files in different locations within the at least two of the SAN nodes, wherein the log files are initially unknown to an entity that that retrieves the log files.

15. The method of claim 4, further comprising:

generating the collection script from the identified set of available log files, from the discovery script, and from the received identity of the log files;

receiving the collection script at each of a plurality of SAN nodes;

executing the script in each of the SAN nodes;

identifying, from the executed script, a set of log files;

zipping the identified log files into an aggregated log file;

transmitting the aggregated log file back to the initiating host;

wherein the discovery script is executable on the SAN node, the SAN node operable to generate the identity of the log files from execution of the discovery script, the discovery script independent of the operating system and file system of the SAN node; and

wherein generating the discovery script includes using a log file designator that describes a manner of locating log files by referencing multiple log files via wildcards and directory specifications, wherein generating the discovery script occurs before generating the collection script.

10. A storage area network (SAN) infrastructure server for retrieving log files comprising:

a log collection utility operable to compute an identity of a set of log files on a SAN node in a storage area network (SAN);

a task definition operable to store each of a plurality of the computed identities, the log collection utility responsive to the task definition for generating, from the identity of the log files, a collection script operable to retrieve each of the identified log files, the collection script generated independently of an operating system of the SAN node;

an interface to a SAN infrastructure operable to send the collection script to the SAN node, the log collection utility further operable to receive, from the interface, each of the log files collected by the collection script;

the log collection utility further operable to;

determine, based on system load, whether to compress collected log files at a host or at a server;

selectively compress the collected log files at the host or at the server based on the determine;

aggregate, on each host, the collected log files obtained from the collection script;

transmit, by each host, the collected log files to the server;

identify categories of log files, categories including at least one of infrastructure, agent or 3rd party applications; and

aggregate transmitted log files from each host at the server in a common repository, the log files arranged by categories.

11. The server of claim 10 further comprising at least one log file designator, the log collection utility further operable to:

determine, using the log file designator, a set of log files types accessible via a SAN;

generate a discovery script having commands for discovering each of the log files identified in the set of log files;

send the discovery script to a SAN node having at least one of the identified log files, the SAN node operable to execute the discovery script; and

receive, from the SAN node, an identity of each of the log files identified via the executed discovery script;

further comprising a plurality of SAN nodes, each of the SAN nodes having at least one agent and a corresponding discovery script, the agent responsive to the discovery script, wherein the log collection utility is further operable to;

send the discovery script to each of the plurality of SAN nodes;

receive the identity of each of the log files;

generate a task definition indicative of the collective log files on each of the SAN nodes; and

generate the collection script form the task definition;

wherein each SAN node further comprises a host having a master agent, the master agent operable to execute the collection script, further comprising;

generate, for each host, a collection script operable to collect the discovered log files corresponding to that host;

execute the generated collection script on the host; and

receive the collected log files at the server.

12. The server of claim 11 wherein the host is further operable to:

receive the collection script at each of a plurality of SAN nodes;

execute the script on each of the SAN nodes;

identify, from the executed script, a set of log files;

compress the identified log files into an aggregated log file; and

transmit the aggregated log file back to the initiating host;

wherein the discovery script is executable on the SAN node, the SAN node operable to generate the identity of the log files from execution of the discovery script, the discovery script independent of the operating system and file system of the SAN node, the log collection utility further operable to;

enumerate the log files in a list, the list including definitions indicative of the location of each of the log files, wherein the log file list is a task definition file indicative of a location of at least one log file on each of a set of hosts; and

generate the collection script from the identified set of available log files and the received identity of the log files, the collection script having commands for accessing each of the log files identified in the set of log files.

13. A computer program product having a computer readable storage medium operable to store computer program logic embodied in computer program code encoded thereon as an encoded set of processor based instructions for performing a method of retrieving log files in a heterogeneous storage area network (SAN), comprising:

computer program code for computing an identity of a set of log files in a network environment including a plurality of nodes, the network environment including a storage area network (SAN) having SAN nodes;

computer program code for generating, from the identity of the the log files, a collection script operable to retrieve each of the identified log files, the collection script generated independently of an operating system of a SAN node;

computer program code for sending the collection script to the SAN node;

computer program code for receiving, from the SAN node, each of the log files collected by the collection script;

computer program code for determining, based on system load, whether to compress collected log files at a host or at a server;

computer program code for selectively compressing the collected log files at the host or at the server based on the determining;

computer program code for aggregating, on each host, the collected log files obtained from the collection script;

computer program code for transmitting, by each host, the collected log files to the server;

computer program code for identifying categories of log files, categories including at least one of infrastructure, agent or 3rd party applications; and

computer program code for aggregating transmitted log files from each

host at the server in a common repository, the log files arranged by categories.

16. The computer program product of claim 13, further comprising:

computer program code for identifying a set of log files accessible via a SAN;

computer program code for generating a discovery script having commands for discovering each of the log files identified in the set of log files;

computer program code for sending the discovery script to a SAN node having at least one of the identified log files, the SAN node operable to execute the discovery script; and

computer program code for receiving, from the SAN node, an identity of each of identified via the executed discovery script.