CLUSTER MANAGEMENT

• US 20150229715A1
• Filed: 12/31/2014
• Published: 08/13/2015
• Est. Priority Date: 02/13/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: LinkedIn Corporation (Microsoft Corporation)
• Inventors: Sankar, Sriram, Buthay, Diego
• Application Number:
  US14/587,771
• Publication Number:
  US 20150229715A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 2209/505 Clust
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06Q 10/047 Optimisation of routes or paths, e.g. travelling salesman problem
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/083 Shipping
• CPC Class Codes: G06Q 50/30 Transportation; Communications
• CPC Class Codes: G06Q 50/40 Business processes related to the transportation industry shipping G06Q10/083
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

Methods and systems cluster management are disclosed. In some example embodiments, a cluster manager determines a configuration of roles for a plurality of distinct server machines and for a plurality of builder machines, with each one of the server machines storing a corresponding shard of data, and each one of the plurality of builder machines comprising a corresponding one of the corresponding shards of data of the server machines. The cluster manager applies the configuration of roles to the plurality of server machines, the plurality of builder machines, and an aggregator, with the configuration of the builder machines being characterized by an absence of communication with the aggregator. The configuration is used to determine which machines to be communicated with by the aggregator for a client request and which machines to be communicated with by an update service for an update of data.

Claims

1. A method comprising:

determining, by a cluster manager implemented by at least one processor, a configuration of roles for a plurality of distinct server machines and for a plurality of builder machines, each one of the server machines storing a corresponding shard of data, and each one of the plurality of builder machines comprising a corresponding one of the corresponding shards of data of the server machines;

applying, by the cluster manager, the configuration of roles to the plurality of server machines, the plurality of builder machines, and an aggregator, the configuration of the builder machines being characterized by an absence of communication with the aggregator;

receiving, by the aggregator, a client request to perform an online service;

transmitting, by the aggregator, a service request to each one of the plurality of server machines based on the client request;

receiving, by each one of the server machines, the service request, each one of the server machines storing a corresponding shard of data;

accessing, by each one of the server machines, the corresponding shard of data;

transmitting, by each one of the server machines, a corresponding response to the aggregator based on the accessing the corresponding shard of data;

receiving, by an update service, update data;

updating, by the update service, the corresponding shard of data of at least one of the server machines based on the update data and the configuration of roles; and

updating, by the update service, the corresponding shard of data of at least one of the builder machines based on the update data and the configuration of roles.

2. The method of claim 1, wherein the online service comprises a search function.

3. The method of claim 1, wherein further comprising the cluster manager managing a plurality of replica groups, each replica group comprising a corresponding one of the server machines and at least one replica machine, the at least one replica machine comprising the corresponding shard of data of the corresponding server machine of the corresponding replica group, wherein the managing comprises, in response to an update of the corresponding shard of one of the server machines, causing the update service to perform a corresponding update to the at least one replica machine in the corresponding replica group of the one of the server machines.

4. The method of claim 1, further comprising:

detecting, by the cluster manager, one of the server machines that is unable to satisfy a predetermined threshold condition of a function;

selecting, by the cluster manager, a replacement server from amongst a plurality of replacement servers based on a determination that the selected replacement server satisfies at least one predetermined constraint; and

replacing, by the cluster manager, the detected server machine with the selected replacement machine.

5. The method of claim 1, wherein the shards of data of the server machines comprise ranking model files of a search index.

6. The method of claim 1, wherein the shards of data of the server machines comprise language model files for a query rewriter.

7. The method of claim 1, wherein the server machines are incorporated into an online social networking service.

8. A system comprising:

a memory; and

at least one processor configured to perform operations comprising;

determining, by a cluster manager, a configuration of roles for a plurality of distinct server machines and for a plurality of builder machines, each one of the server machines storing a corresponding shard of data, and each one of the plurality of builder machines comprising a corresponding one of the corresponding shards of data of the server machines;

applying, by the cluster manager, the configuration of roles to the plurality of server machines, the plurality of builder machines, and an aggregator, the configuration of the builder machines being characterized by an absence of communication with the aggregator;

receiving, by the aggregator, a client request to perform an online service;

transmitting, by the aggregator, a service request to each one of the plurality of server machines based on the client request;

receiving, by each one of the server machines, the service request, each one of the server machines storing a corresponding shard of data;

accessing, by each one of the server machines, the corresponding shard of data;

transmitting, by each one of the server machines, a corresponding response to the aggregator based on the accessing the corresponding shard of data;

receiving, by an update service, update data;

updating, by the update service, the corresponding shard of data of at least one of the server machines based on the update data and the configuration of roles; and

updating, by the update service, the corresponding shard of data of at least one of the builder machines based on the update data and the configuration of roles.

9. The system of claim 8, wherein the online service comprises a search function.

10. The system of claim 8, wherein the operations further comprise the cluster manager managing a plurality of replica groups, each replica group comprising a corresponding one of the server machines and at least one replica machine, the at least one replica machine comprising the corresponding shard of data of the corresponding server machine of the corresponding replica group, wherein the managing comprises, in response to an update of the corresponding shard of one of the server machines, causing the update service to perform a corresponding update to the at least one replica machine in the corresponding replica group of the one of the server machines.

11. The system of claim 8, wherein the operations further comprise:

detecting, by the cluster manager, one of the server machines that is unable to satisfy a predetermined threshold condition of a function;

selecting, by the cluster manager, a replacement server from amongst a plurality of replacement servers based on a determination that the selected replacement server satisfies at least one predetermined constraint; and

replacing, by the cluster manager, the detected server machine with the selected replacement machine.

12. The system of claim 8, wherein the shards of data of the server machines comprise ranking model files of a search index.

13. The system of claim 8, wherein the shards of data of the server machines comprise language model files for a query rewriter.

14. The system of claim 8, wherein the server machines are incorporated into an online social networking service.

15. A non-transitory machine-readable storage medium embodying a set of instructions that, when executed by a processor, cause the processor to perform operations comprising:

determining, by a cluster manager, a configuration of roles for a plurality of distinct server machines and for a plurality of builder machines, each one of the server machines storing a corresponding shard of data, and each one of the plurality of builder machines comprising a corresponding one of the corresponding shards of data of the server machines;

applying, by the cluster manager, the configuration of roles to the plurality of server machines, the plurality of builder machines, and an aggregator, the configuration of the builder machines being characterized by an absence of communication with the aggregator;

receiving, by the aggregator, a client request to perform an online service;

transmitting, by the aggregator, a service request to each one of the plurality of server machines based on the client request;

receiving, by each one of the server machines, the service request, each one of the server machines storing a corresponding shard of data;

accessing, by each one of the server machines, the corresponding shard of data;

transmitting, by each one of the server machines, a corresponding response to the aggregator based on the accessing the corresponding shard of data;

receiving, by an update service, update data;

updating, by the update service, the corresponding shard of data of at least one of the server machines based on the update data and the configuration of roles; and

updating, by the update service, the corresponding shard of data of at least one of the builder machines based on the update data and the configuration of roles.

16. The storage medium of claim 15, wherein the online service comprises a search function.

17. The storage medium of claim 15, wherein the operations further comprise the cluster manager managing a plurality of replica groups, each replica group comprising a corresponding one of the server machines and at least one replica machine, the at least one replica machine comprising the corresponding shard of data of the corresponding server machine of the corresponding replica group, wherein the managing comprises, in response to an update of the corresponding shard of one of the server machines, causing the update service to perform a corresponding update to the at least one replica machine in the corresponding replica group of the one of the server machines.

18. The storage medium of claim 15, wherein the operations further comprise:

detecting, by the cluster manager, one of the server machines that is unable to satisfy a predetermined threshold condition of a function;

selecting, by the cluster manager, a replacement server from amongst a plurality of replacement servers based on a determination that the selected replacement server satisfies at least one predetermined constraint; and

replacing, by the cluster manager, the detected server machine with the selected replacement machine.

19. The storage medium of claim 15, wherein the shards of data of the server machines comprise ranking model files of a search index.

20. The storage medium of claim 15, wherein the shards of data of the server machines comprise language model files for a query rewriter.