Dynamically instantiating and terminating data queues

• US 10,419,528 B2
• Filed: 10/31/2017
• Issued: 09/17/2019
• Est. Priority Date: 03/17/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Vlachogiannis, Ioannis, Papadomitsos, Panagiotis
• Primary Examiner(s): Hussain, Imad
• Application Number:
  US15/799,682
• Publication Number:
  US 20180054474A1
• Time in Patent Office: 686 Days
• Field of Search: 709213
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3442 for planning or managing the needed capacity
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/87 Monitoring of transactions
• CPC Class Codes: G06F 9/46 Multiprogramming arrangements
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A computer-implemented method, system, and computer-readable media are disclosed herein. In embodiments, the computer-implemented method may entail receiving, by a data service, live data associated with an entity. The entity may be, for example, a customer of the data service. The method may further include determining that a dual-queue node assigned to the entity is uninstantiated on the data service. As a result, a dual-queue node associated with the entity may be instantiated on the data service. The dual-queue node may be instantiated by initializing a live data queue, of the dual-queue node, in which to place the live data for processing and a stale data queue, of the dual-queue node, in which to store a persistent backup of the live data. The method may then route the live data to the dual-queue node. The dual-queue node may then process the live data. Additional embodiments are described and/or claimed.

Claims

1. A computer-implemented method comprising:

receiving live data at a data queue system (DQS), wherein the live data is associated with an entity;

dynamically instantiating a dual-queue node associated with the entity on the DQS, wherein dynamically instantiating the dual-queue node includes;

initializing a live data queue included in the dual-queue node, wherein the initialized live data queue is enabled to receive and process the live data and wherein the live data queue is assigned to the entity; and

determining whether a persistent data store of the DQS includes a partition assigned to the entity, wherein the partition assigned to the entity is enabled to serve as a stale data queue for the dual-queue node, and wherein the stale data queue stores a persistent copy of the live data;

routing the live data to the dual-queue node; and

employing the dual-queue node to process the live data.

2. The computer-implemented method of claim 1, further comprising:

determining whether the DQS currently includes the dual-queue node associated with the entity; and

instantiating the dual queue node associated with the entity based on determining whether the DQS currently includes the dual-queue node associated with the entity.

3. The computer-implemented method of claim 1, further comprising:

receiving additional live data at the DQS, wherein the additional live data is associated with the entity;

in response to determining that the DQS includes the dual-queue node associated with the entity, routing the additional live data to the dual-queue node; and

employing the dual-queue data node to process the additional live data.

4. The computer-implemented method of claim 1, further comprising:

determining whether the dual-queue node is active based on a period of inactivity; and

in response to determining that the dual-queue node is active, employing the active dual-queue node to process the additional live data.

5. The computer-implemented method of claim 1, further comprising:

determining whether the dual-queue node is inactive based on a period of inactivity; and

in response to determining that the dual-queue node is inactive, terminating the dual-queue node.

6. The computer-implemented method of claim 1, further comprising:

in response to determining that the dual-queue node is inactive, deallocating memory associated with the live data queue.

7. The computer-implemented method of claim 1, further comprising:

in response to determining that the dual-queue node is inactive, preserving stale data or state data stored in the stale data queue for the dual-queue node.

8. The computer-implemented method of claim 1, further comprising:

in response to determining that the dual-queue node is inactive, terminating the dual-queue node; and

in response to receiving additional live data associated with the entity, dynamically re-instantiating the terminated dual-queue node based on preserved state data stored in the stale data queue for the terminated dual-queue node.

9. The computer-implemented method of claim 1, further comprising:

determining whether to terminate the dual-queue node based on a period of inactivity, wherein the period of inactivity is dynamically adjusted based on an amount of data stored in the stale data queue of the dual-queue node, an activity level of the DQS, or an amount of available resources of the DQS.

10. The computer-implemented method of claim 1, further comprising:

determining whether to terminate the dual-queue node based on a period of inactivity, wherein the period of inactivity is dynamically adjusted based on a determined re-instantiation period associated with the entity.

11. The computer-implemented method of claim 1, further comprising:

in response to determining that the dual-queue node is inactive, updating queue-entity mapping data to remove the terminated dual-queue data node.

12. The computer-implemented method of claim 1, further comprising:

determining whether the dual-queue node is inactive;

in response to determining that the dual-queue node is inactive, adding the dual-queue node to a prioritized queue of inactive dual-queue nodes; and

in response to determining that a utilization of resources of the DQS exceeds a predefined threshold, selecting at least one dual-queue node included in the prioritized queues for termination based on a prioritization associated with each of the inactive dual-queue nodes.

13. The computer-implemented method of claim 1, further comprising:

initializing a dual-queue manager for the dual-queue node;

determining an amount of memory for the live data queue based on an activity level of the entity or an amount of available resources of the DQS; and

employing the dual-queue manager to allocate the determined amount of memory for the live data queue.

14. The computer-implemented method of claim 1, wherein dynamically instantiating the dual-queue node associated with the entity further includes:

identifying the partition included in the persistent data store that is associated with the entity; and

associating the partition with the dual-queue node, such that the partition serves as the stale data queue of the dual-queue node.

15. The computer-implemented method of claim 1, wherein dynamically instantiating the dual-queue node associated with the entity further includes:

in response to determining that the persistent data store does not include the partition associated with the entity, initializing the partition such that the persistent data store includes the partition; and

associating the partition with the dual-queue node, such that the partition serves as the stale data queue of the dual-queue node.

16. The computer-implemented method of claim 1, wherein dynamically instantiating the dual-queue node associated with the entity further includes:

updating queue-entity mapping data to include the dual-queue node and the dual-queue node'"'"'s association with the entity.

17. The computer-implemented method of claim 1, wherein the DQS is a multi-tenant DQS and the method further comprises:

receiving a request to migrate the received live data to a single-tenant DQS associated with the entity;

dynamically instantiating another dual-queue node associated with the entity on the single-tenant DQS;

forwarding the live data to the other dual-queue node included in the single-tenant DQS; and

terminating the dual-queue node associated with the entity included in the multi-tenant DQS.

18. The computer-implemented method of claim 1, wherein the method further comprises:

receiving a request to migrate the received live data to a single-tenant DQS associated with the entity;

dynamically instantiating another dual-queue node associated with the entity on the single-tenant DQS; and

forwarding a portion of the stale data stored in the stale data queue for the dual-queue node included in the multi-tenant DQS to the other dual-queue node included in the single-tenant DQS.

19. The computer-implemented method of claim 1, wherein the method further comprises:

receiving a request for the live data associated with the entity;

determining that a copy of the live data is stored as stale data in the stale data queue for the dual-queue node;

retrieving the copy of the live data stored as stale data; and

employing the copy of the live data to service the request for the live data.

20. The computer-implemented method of claim 1, wherein the method further comprises:

loading the live data into the live data queue of the dual-queue node; and

storing a persistent copy of the live data in the stale data queue of the dual-queue node.

21. The computer-implemented method of claim 1, wherein the method further comprises:

establishing a connection with a remote data store;

employing the connection with the remote data store to transmit a persistent copy of the live data stored in the stale data queue of the dual-queue node to the remote data store; and

deleting the portion of the persistent copy of the live data from the stale data queue of the dual-queue node.

22. The computer-implemented method of claim 1, wherein the method further comprises:

monitoring the stale data queue of the dual-queue node; and

in response to detecting a deletion event when monitoring the stale data queue of the dual-queue node, deleting a portion of stale data stored in the stale data queue for the dual-queue node.

23. The computer-implemented method of claim 1, wherein processing the live data includes generating summary statistics associated with the entity and the method further comprises:

in response to generating the summary statistics, providing the summary statistics to the entity.

24. The computer-implemented method of claim 1, wherein processing the live data includes detecting an alert event associated with the entity and the method further comprises:

in response to detecting the alert event, providing an alert to the entity.

25. A system comprising:

one or more processors; and

one or more computer-readable storage media containing instructions which, in response to execution by the one or more processors, cause the one or more processors to;

receive live data at a data queue system (DQS), wherein the live data is associated with an entity;

dynamically instantiate a dual-queue node associated with the entity on the DQS, wherein dynamically instantiating the dual-queue node includes;

initializing a live data queue included in the dual-queue node, wherein the initialized live data queue is enabled to receive and process the live data and wherein the live data queue is assigned to the entity; and

determining whether a persistent data store of the DQS includes a partition assigned to the entity, wherein the partition assigned to the entity is enabled to serve as a stale data queue for the dual-queue node, and wherein the stale data queue stores a persistent copy of the live data;

routing the live data to the dual-queue node; and

employing the dual-queue node to process the live data.

26. The system of claim 25, wherein in response to execution of the instruction by the one or more processors, the one or more processors further:

determine whether the DQS currently includes the dual-queue node associated with the entity; and

instantiate the dual queue node associated with the entity based on determining whether the DQS currently includes the dual-queue node associated with the entity.

27. The system of claim 25, wherein in response to execution of the instruction by the one or more processors, the one or more processors further:

receive additional live data at the DQS, wherein the additional live data is associated with the entity;

in response to determining that the DQS includes the dual-queue node associated with the entity, route the additional live data to the dual-queue node; and

employ the dual-queue data node to process the additional live data.

28. One or more non-transitory computer-storage media having executable instructions, which, when executed by a computing device, cause the computing device to:

receive live data at a data queue system (DQS), wherein the live data is associated with an entity;

dynamically instantiate a dual-queue node associated with the entity on the DQS, wherein dynamically instantiating the dual-queue node includes;

initializing a live data queue included in the dual-queue node, wherein the initialized live data queue is enabled to receive and process the live data and wherein the live data queue is assigned to the entity; and

determining whether a persistent data store of the DQS includes a partition assigned to the entity, wherein the partition assigned to the entity is enabled to serve as a stale data queue for the dual-queue node, and wherein the stale data queue stores a persistent copy of the live data;

route the live data to the dual-queue node; and

employ the dual-queue node to process the live data.

29. The one or more non-transitory computer-storage media of claim 28, wherein when the instructions are executed by the computing device, cause the computing device to further:

determine whether the dual-queue node is active based on a period of inactivity; and

in response to determining that the dual-queue node is active, employing the active dual-queue node to process the additional live data.

30. The one or more non-transitory computer-storage media of claim 28, wherein when the instructions are executed by the computing device, cause the computing device to further:

determining whether the dual-queue node is inactive based on a period of inactivity; and

in response to determining that the dual-queue node is inactive, terminating the dual-queue node.