Ephemeral remote data store for dual-queue systems

• US 9,836,358 B2
• Filed: 04/29/2015
• Issued: 12/05/2017
• Est. Priority Date: 03/17/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Vlachogiannis, Ioannis, Papadomitsos, Panagiotis
• Primary Examiner(s): Hussain, Imad
• Application Number:
  US14/699,992
• Publication Number:
  US 20150261622A1
• Time in Patent Office: 951 Days
• Field of Search: 707644
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11

Abstract

A computer-implemented method, system, and computer-readable media are disclosed herein. In embodiments, the computer-implemented method may entail receiving, by a data service, live data associated with an entity. The entity may be, for example, a customer of the data service. The method may then route the live data to a dual-queue system. The live data may then be loaded into a live data queue for processing of the live data. In addition, the live data may be stored as a persistent backup of the live data in a stale data queue. A remote data store may periodically establish a connection with the dual-queue system, after which, at least a portion of the stale data may be transmitted to the remote data store. Additional embodiments are described and/or claimed.

Claims

1. A computer-implemented method comprising:

receiving, by a data service, live data associated with an entity;

routing the live data to a dual-queue system assigned to the entity;

loading the live data into a live data queue of the dual-queue system for processing of the live data;

storing a persistent backup of the live data in a stale data queue of the dual-queue system as stale data;

in response to establishing a connection with a remote data store, instantiating a dual-queue node for the remote data store, wherein the instantiated dual-queue node for the remote data store includes the stale data queue and an initialized second live data queue;

transmitting, via the dual-queue node for the remote data store, at least a portion of the stale data stored in the stale data queue of the dual-queue node for the remote data store to the remote data store.

2. The computer-implemented method of claim 1, further comprising:

deleting the stale data transmitted to the remote data store from the stale data queue.

3. The computer-implemented method of claim 1, wherein the remote data store is associated with the entity.

4. The computer-implemented method of claim 1, wherein the stale data transmitted to the remote data store is transmitted from the stale data queue.

5. The computer-implemented method of claim 1, wherein routing the live data to a dual-queue system assigned to the entity further comprises:

identifying the dual-queue system assigned to the entity from a plurality of dual-queue systems operated by the data service, wherein each of the plurality of dual-queue systems is assigned to a respective entity of a plurality of entities.

6. The computer-implemented method of claim 1, wherein the entity is a customer, of the data service, to which the live data belongs, and

wherein the live data is associated with the entity via an identifier of the customer contained within the live data.

7. The computer-implemented method of claim 1,

wherein the live data is received from an application associated with a customer of the data service, and

wherein the entity is the customer.

8. The computer-implemented method of claim 1,

wherein the live data is received from an application associated with a customer of the data service,

wherein the customer is the entity,

wherein the live data is associated with the entity via an identifier of the customer or the application contained within the live data, and

wherein the live data is related to usage or performance of the application.

9. The computer-implemented method of claim 1, wherein the entity is a customer, and wherein the remote data store is not part of the data service.

10. The computer-implemented method of claim 1, wherein the entity is a customer, and wherein the remote data store is a portable computing device that is not part of the data service.

11. The computer-implemented method of claim 1, wherein the entity is an internet of things device.

12. The computer-implemented method of claim 1, further comprising:

receiving an acknowledgement from the remote data store confirming receipt of the stale data transmitted to the remote data store; and

deleting the stale data transmitted to the remote data store from the stale data queue in response to receiving the acknowledgment.

13. The computer-implemented method of claim 1, further comprising:

monitoring the stale data queue to determine whether a deletion event has occurred; and

deleting at least a subset of the stale data from the stale data queue in response to determining that the deletion event has occurred.

14. The computer-implemented method of claim 1, further comprising:

determining that a subset of the stale data exceeds a threshold of age; and

deleting the subset of the stale data from the stale data queue in response to the determining.

15. The computer-implemented method of claim 1, further comprising:

determining that a cumulative size of the stale data exceeds a predefined threshold; and

deleting a subset of the stale data from the stale data queue in response to the determining.

16. The computer-implemented method of claim 1, further comprising:

determining that an amount of available space in the stale data queue has reached a predefined threshold; and

deleting a subset of the stale data from the stale data queue in response to the determining.

17. The computer-implemented method of claim 1, further comprising:

determining that a cumulative size of the stale data exceeds a predefined size threshold or that an amount of available space in the stale data queue has reached a predefined space threshold;

reserving additional space in a persistent data store for an additional stale data queue when additional space in the persistent data store is available;

storing a persistent backup of additional live data received by the data service associated with the entity to the additional stale data queue as additional stale data;

transmitting the additional stale data to a customer of the data service in response to receiving payment for the stale data from the customer; and

terminating the additional stale data queue.

18. The computer-implemented method of claim 1, wherein the stale data is first stale data and the stale data queue is a first stale data queue, the method further comprising:

determining that a cumulative size of the first stale data exceeds a predefined size threshold or that an amount of available space in the first stale data queue has reached a predefined space threshold;

reserving additional space in a persistent data store for a second stale data queue;

storing a persistent backup of additional live data, associated with the entity and received by the data service, to the second stale data queue as second stale data;

transmitting the second stale data to a customer of the data service in response to determining that the first stale data has been downloaded from the first stale data queue; and

terminating the second stale data queue.

19. The computer-implemented method of claim 1, further comprising:

monitoring the stale data queue to determine whether a deletion event has occurred, wherein the deletion event is a selected one of;

an amount of available space in the stale data queue reaching a predefined threshold, or

a cumulative size of the stale data exceeding a predefined threshold;

in response to determining that the deletion event has occurred, identifying a subset of the stale data that is of lower priority when compared to a remainder of the stale data; and

deleting the subset of the stale data from the stale data queue.

20. The computer-implemented method of claim 1, further comprising receiving a pull request from the remote data store and wherein transmitting the at least portion of the stale data to the remote data store from the stale data queue is in response to receiving the pull request.

21. The computer-implemented method of claim 1, wherein transmitting the at least portion of the stale data to the remote data store from the stale data queue further comprises automatically pushing the portion of the stale data to the remote data store in response to establishing the connection.

22. The computer-implemented method of claim 1, wherein the dual-queue system is a cloud based dual-queue system.

23. The computer-implemented method of claim 1, further comprising:

processing the live data from the live data queue to generate a summary and statistics of the live data; and

transmitting the summary of the live data concurrently with transmitting the at least portion of the stale data to the remote data store to enable a user of the remote data store to view the summary and statistics while receiving the at least portion of the stale data.

24. The computer-implemented method of claim 1, further comprising

processing the live data from the live data queue to generate summary and statistics of the live data;

monitoring customer access to the data service; and

transmitting the summary and statistics of the live data in response to determining that a predefined length of time has passed since the customer last accessed the data service.

25. A system comprising:

one or more processors; and

one or more computer-readable storage media containing instructions which, in response to execution by the one or more processors, cause the one or more processors to;

receive live data associated with an entity;

route the live data to a dual-queue system assigned to the entity;

load the live data into a live data queue of the dual-queue system for processing of the live data;

store a persistent backup of the live data in a stale data queue of the dual-queue system as stale data;

in response to establishing a connection with a remote data store, instantiate a dual-queue node for the remote data store, wherein the instantiated dual-queue node for the remote data store includes the stale data queue and an initialized second live data queue;

transmit, via the dual-queue node for the remote data store, at least a portion of the stale data to the remote data store; and

delete the stale data transmitted to the remote data store from the stale data queue included in the dual-queue node for the remote data store.

26. The system of claim 25, wherein to route the live data to a dual-queue system assigned to the entity the instructions further causes the one or more processors to:

identify the dual-queue system assigned to the entity from a plurality of dual-queue systems operated by the data service, wherein each of the plurality of dual-queue systems is assigned to a respective entity of a plurality of entities.

27. The system of claim 25, wherein the instructions further cause the one or more processors to:

monitor the stale data queue to determine whether a deletion event has occurred; and

delete at least a subset of the stale data from the stale data queue in response to determining that the deletion event has occurred.

28. One or more non-transitory computer-storage media having executable instructions stored thereon, which, when executed by a computing device, cause the computing device to:

receive live data associated with an entity;

route the live data to a dual-queue system assigned to the entity;

load the live data into a live data queue of the dual-queue system for processing of the live data;

store a persistent backup of the live data in a stale data queue of the dual-queue system as stale data;

in response to establishing a connection with a remote data store, instantiate a dual-queue node for the remote data store, wherein the instantiated dual-queue node for the remote data store includes the stale data queue and an initialized second live data queue;

transmit, via the dual-queue node for the remote data store, at least a portion of the stale data stored in the stale data queue of the dual-queue node for the remote data store to the remote data store; and

delete the stale data transmitted to the remote data store from the stale data queue of the dual-queue node for the remote data store.

29. The one or more computer-storage media of claim 28, wherein to route the live data to a dual-queue system assigned to the entity the instructions further cause the computing device to:

identify the dual-queue system assigned to the entity from a plurality of dual-queue systems operated by the data service, wherein each of the plurality of dual-queue systems is assigned to a respective entity of a plurality of entities.

30. The one or more computer-storage media of claim 28, wherein the instructions further cause the one or more computer-storage media to:

monitor the stale data queue to determine whether a deletion event has occurred; and

delete at least a subset of the stale data from the stale data queue in response to determining that the deletion event has occurred.