Data forwarding using multiple data pipelines

• US 9,753,818 B2
• Filed: 10/31/2014
• Issued: 09/05/2017
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Vlachogiannis, Ioannis, Papadomitsos, Panagiotis
• Primary Examiner(s): PATEL, KAMINI B
• Application Number:
  US14/530,445
• Publication Number:
  US 20160085637A1
• Time in Patent Office: 1,040 Days
• Field of Search: 714 18
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: H04L 1/16 in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 47/2416 Real-time traffic

Abstract

In accordance with implementations of the present disclosure, a computer-implemented method for forwarding data includes generating a backup of received live data. The backup of the received live data may be generated on non-volatile storage. The method further includes providing the received live data to a first data buffer for forwarding of the received live data, where a portion of the received live data is dropped prior to being forwarded. Based on the portion of the received live data being dropped, the portion of the received live data is recovered from the backup and the recovered portion of the received live data is provided to a second data buffer for forwarding of the recovered portion of the received live data.

Claims

1. A computer-implemented method comprising:

receiving live data at an input interface of a data forwarder, the data forwarder comprising a real-time data pipeline and a stale data pipeline;

generating a backup of the received live data at the data forwarder while providing the received live data to a first data buffer of the real-time data pipeline for forwarding of the received live data from an output interface of the data forwarder to a destination node, wherein a portion of the received live data is dropped prior to the forwarding; and

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer of the stale data pipeline, the second data buffer for forwarding of the recovered portion of the received live data from the output interface to the destination node.

2. The computer-implemented method of claim 1, wherein the backup of the received live data is generated on non-volatile storage.

3. The computer-implemented method of claim 1, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

4. The computer-implemented method of claim 1, wherein the portion of the received live data is dropped from the first data buffer.

5. The computer-implemented method of claim 1, wherein the first data buffer comprises a data queue.

6. The computer-implemented method of claim 1, wherein the forwarding of the received live data is in response to the data forwarder receiving a pull request.

7. The computer-implemented method of claim 1, further comprising logging one or more acknowledgments of the received live data, the one or more acknowledgements being received by the data forwarder in response to the forwarding of the received live data.

8. The computer-implemented method of claim 1, wherein the recovering comprises identifying the portion of the received live data based on a comparison between the backup of the received live data and one or more recorded acknowledgments of at least some of the received live data.

9. The computer-implemented method of claim 1, further comprising:

performing the forwarding of the recovered portion of the received live data; and

based on the recovered portion of the received live data being unacknowledged, subsequently recovering the portion of the received live data from the backup and providing the subsequently recovered portion of the received live data to the second data buffer for forwarding of the subsequently recovered portion of the received live data from the output interface.

10. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data includes another portion of the received live data from the first data buffer in a response to a pull request.

11. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data is in a response to a pull request received by the data forwarder, the recovered portion being included in the response along with live data from the first data buffer based on emptying of the first data buffer.

12. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data is based on the first data buffer having less than an amount of data requested by a pull request received by the data forwarder.

13. The computer-implemented method of claim 1, further comprising removing the portion of the received live data from the backup of the received live data based on receiving an acknowledgement in response to the forwarding of the recovered portion of the received live data.

14. The computer-implemented method of claim 1, wherein the recovering of the portion of the received live data from the backup comprises:

closing a transaction log in the backup of the received live data;

opening a new transaction log for a backup of subsequently received live data;

retrieving the portion of the received live data from the closed transaction log; and

generating the backup of the subsequently received live data in the new transaction log.

15. The computer-implemented method of claim 1, wherein the backup of the received live data is generated on a hard disk.

16. The computer-implemented method of claim 1, wherein the first data buffer comprises a live data queue and the second data buffer comprises a stale data queue.

17. The computer-implemented method of claim 1, wherein the backup comprises transactions and transaction identifiers corresponding to the received live data.

18. The computer-implemented method of claim 1, further comprising:

receiving an acknowledgement in response to the forwarding of the recovered portion of the received live data; and

recording the received acknowledgement in an acknowledgement log.

19. The computer-implemented method of claim 1, wherein the recovering of the portion of the received live data from the backup comprises:

closing an acknowledgement log comprising recorded acknowledgments of forwarded data;

opening a new acknowledgement log for recording subsequent acknowledgments of forwarded data;

identifying the portion of the received live data in the backup based on a comparison between the closed acknowledgment log and the backup of received live data; and

recording the subsequent acknowledgments of forwarded data in the new acknowledgment log.

20. The computer-implemented method of claim 1, further comprising sending the recovered portion of the received live data to the output interface of the data forwarder for the forwarding.

21. The computer-implemented method of claim 19, wherein the recovering of the portion of the received live data from the backup comprises:

closing a transaction log in the backup of the received live data;

opening a new transaction log for a backup of subsequently received live data;

retrieving the identified portion of the received live data from the closed transaction log based on the identifying; and

generating the backup of the subsequently received live data in the new transaction log.

22. The computer-implemented method of claim 1, further comprising sending at least some of the received live data to the output interface for the forwarding with live data from the first data buffer in response to receiving a pull request from the destination node.

23. A system for forwarding data, the system comprising:

one or more data processors; and

one or more computer-readable storage media containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations including;

receiving live data at an input interface of a data forwarder, the data forwarder comprising a real-time data pipeline and a stale data pipeline;

generating a backup of the received live data at the data forwarder while providing the received live data to a first data buffer of the real-time data pipeline for forwarding of the received live data from an output interface of the data forwarder to a destination node, wherein a portion of the received live data is dropped prior to the forwarding; and

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer of the stale data pipeline, the second data buffer for forwarding of the recovered portion of the received live data from the output interface to the destination node.

24. The system of claim 23, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

25. The system of claim 23, wherein the portion of the received live data is dropped from the first data buffer.

26. The system of claim 23, wherein the forwarding of the received live data is in response to the data forwarder receiving a pull request.

27. One or more non-transitory computer-storage media storing computer-useable instructions that, when executed by a computing device, perform a method for forwarding data, the method comprising:

receiving live data at an input interface of a data forwarder, the data forwarder comprising a real-time data pipeline and a stale data pipeline;

generating a backup of the received live data at the data forwarder while providing the received live data to a first data buffer of the real-time data pipeline for forwarding of the received live data from an output interface of the data forwarder to a destination node, wherein a portion of the received live data is dropped prior to the forwarding; and

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer of the stale data pipeline, the second data buffer for forwarding of the recovered portion of the received live data from the output interface to the destination node.

28. The one or more computer-storage media of claim 27, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

29. The one or more computer-storage media of claim 27, wherein the portion of the received live data is dropped from the first data buffer.

30. The one or more computer-storage media of claim 27, wherein the forwarding of the received live data is in response to the data forwarder receiving a pull request.