Centralized configuration of a distributed computing cluster

• US 9,716,624 B2
• Filed: 10/08/2014
• Issued: 07/25/2017
• Est. Priority Date: 02/07/2012
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Zeyliger, Philip, Langdale, Philip Lee, Hunt, Patrick David
• Primary Examiner(s): KIM, EDWARD J
• Application Number:
  US14/509,300
• Publication Number:
  US 20150039735A1
• Time in Patent Office: 1,021 Days
• Field of Search: 709220, 709221, 709222, 709223, 709224
• CPC Class Codes: G06F 11/3051 Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs verification or detection of system hardware configuration G06F11/2247
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/328 Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 9/44505 Configuring for program initiating, e.g. using registry, configuration files
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/5041 characterised by the time relationship between creation and deployment of a service
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

Systems and methods for centralized configuration of a distributed computing cluster are disclosed. One embodiment of the disclosed technology provides a user environment that facilitates a selection of a service to be run on hosts in the distributed computing cluster and configuration of the service or hosts in the distributed computer cluster. The disclosed technology can further configure each of the hosts in the distributed computing cluster to run the service based on a set of configuration settings.

Claims

1. A method of centralized configuration of a distributed computing cluster including a catalog of hosts, the method comprising:

a plurality of agents deployed to the catalog of hosts,

wherein the agents are configured to start an in-memory process on each of the catalog of hosts to aggregate statistics associated with each of the catalog of hosts,

wherein, to aggregate the statistics, the agents are configured to perform a plurality of tests suitable for one or more of;

(1) a distributed file storage system jointly operated among the catalog of hosts, (2) a distributed data processing system jointly operated among the catalog of hosts, or (3) a distributed database management system jointly operated among the catalog of hosts,

wherein the plurality of tests are configured with one or more configurable thresholds, and

wherein the agents are further configured to transmit the aggregated statistics and a plurality of heartbeat signals to a server; and

the server, having a memory and a processor, coupled over a network to the computing cluster, wherein the server, when in operation,

provides a user environment enabling a selection of a service to be run on the catalog of hosts in the distributed computing cluster;

wherein the user environment further enables configuration of the service on the catalog of hosts in the distributed computer cluster; and

configures each of the catalog of hosts in the distributed computing cluster to run the service based on a set of configuration settings.

2. The method of claim 1, further comprising, deploying agents to the hosts in the distributed computing cluster to configure each of the hosts.

3. The method of claim 1, further comprising, generating recommended configuration settings of the service or the hosts in the distributed computing cluster to run the service;

wherein, the recommended configuration settings include suggested ranges for parameters and invalid values for the parameters.

4. The method of claim 1, wherein, the user environment further enables, actions to be performed on the set of configuration settings, the actions including, one or more of, reading, validating, updating, and deleting.

5. The method of claim 1, wherein, the user environment further enables addition of services and reconfiguration of the service and user configuration of the hosts with functions to perform under the service.

6. The method of claim 1, wherein, the user environment further enables assignment and re-assignment of roles to each of the hosts to indicate a function under the service that each of the hosts is to perform.

7. The method of claim 1, wherein, the user environment further enables the selection of the service during an installation phase.

8. The method of claim 1, wherein, the user environment further displays current or historical disk usage, CPU, virtual memory consumption, or RAM usage of the hosts in the distributed computing cluster.

9. The method of claim 1, wherein, the user environment further indicates current or historical performance metrics or operational status of the hosts in the distributed computing cluster and one or more of, current or historical performance metrics of the service, a history of actions performed on the service, or a log of configuration changes of the service.

10. The method of claim 1, wherein, the user environment further displays performance metrics of a job or comparison or performance of similar jobs.

11. The method of claim 1, wherein, the user environment further indicates current user activities or historical user activities on the distributed computing cluster.

12. The method of claim 1, wherein, the user environment further displays history of activity monitoring data and configuration changes of the hosts in the distributed computing cluster.

13. The method of claim 1, wherein, the user environment further provides access to log entries associated with the service or events in the distributed computing cluster.

14. The method of claim 13, wherein, the events include, one or more of, a record of change of state of health of the server, a message has been logged, a service has been added or reconfigured, a new job has been setup, an error, a change in operational or on/off state of a given host.

15. The method of claim 13, wherein, the events include, one or more of, a health check event, a log message event, an audit event, and an activity event, and wherein the events are searchable via the user environment.

16. The method of claim 14, wherein, the user environment further enables search or filtering of the log entries by one or more of, time range, service, host, keyword, and user and depicts alerts triggered by certain events or actions in the distributed computing cluster.

17. The method of claim 1, wherein, the user environment further displays operational reports that include, one or more of, disk use by user, user group, or directory, duster job activity by user, group or job ID.

18. The method of claim 1, wherein, the service includes one or more Hadoop services that include one or more of, MapReduce, HDFS, Hue, ZooKeeper or Oozie.

19. The method of claim 1, wherein, the user environment is accessed via a web browser.

20. A system for centralized configuration of a distributed computing duster including a catalog of hosts, the system comprising:

a plurality of agents deployed to the catalog of hosts,

wherein the agents are configured to start an in-memory process on each of the catalog of hosts to aggregate statistics associated with each catalog of hosts,

wherein, to aggregate the statistics, the agents are configured to perform a plurality of tests suitable for one or more of;

(1) a distributed file storage system jointly operated among the catalog of hosts, (2) a distributed data processing system jointly operated among the catalog of hosts, or (3) a distributed database management system jointly operated among the catalog of hosts,

wherein the plurality of tests are configured with one or more configurable thresholds, and

wherein the agents are further configured to transmit the aggregated statistics and a plurality of heartbeat signals to a server; and

the server, wherein the server is coupled to hosts in the distributed computing cluster, and is configured to;

provide a user environment enabling a selection of a service to be run on the catalog of hosts in the distributed computing cluster;

wherein the user environment further enables configuration of the service on the catalog of hosts in the distributed computer cluster; and

configure each of the catalog of hosts in the distributed computing cluster to run the service based on a set of configuration settings.