DATA FORWARDING USING MULTIPLE DATA PIPELINES

• US 20160085637A1
• Filed: 10/31/2014
• Published: 03/24/2016
• Est. Priority Date: 09/19/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: VLACHOGIANNIS, IOANNIS, PAPADOMITSOS, PANAGIOTIS
• Granted Patent: US 9,753,818 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: H04L 1/16 in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 47/2416 Real-time traffic

Abstract

In accordance with implementations of the present disclosure, a computer-implemented method for forwarding data includes generating a backup of received live data. The backup of the received live data may be generated on non-volatile storage. The method further includes providing the received live data to a first data buffer for forwarding of the received live data, where a portion of the received live data is dropped prior to being forwarded. Based on the portion of the received live data being dropped, the portion of the received live data is recovered from the backup and the recovered portion of the received live data is provided to a second data buffer for forwarding of the recovered portion of the received live data.

Claims

1. A computer-implemented method for forwarding data, the method comprising:

generating a backup of received live data;

providing the received live data to a first data buffer for forwarding of the received live data, wherein a portion of the received live data is dropped prior to being forwarded;

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer for forwarding of the recovered portion of the received live data.

2. The computer-implemented method of claim 1, wherein the backup of the received live data is generated on non-volatile storage.

3. The computer-implemented method of claim 1, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

4. The computer-implemented method of claim 1, wherein the portion of the received live data is dropped from the first data buffer.

5. The computer-implemented method of claim 1, wherein the first data buffer comprises a data queue.

6. The computer-implemented method of claim 1, wherein the forwarding of the received live data is in response to a pull request.

7. The computer-implemented method of claim 1, further comprising logging one or more acknowledgments of the received live data, the one or more acknowledgements being received in response to the forwarding of the received live data.

8. The computer-implemented method of claim 1, wherein the recovering comprises identifying the portion of the received live data based on a comparison between the backup of the received live data and one or more recorded acknowledgments of at least some of the received live data.

9. The computer-implemented method of claim 1, further comprising:

performing the forwarding of the portion of the received live data; and

based on the forwarded portion of the received live data being unacknowledged, subsequently recovering the portion of the received live data from the backup and providing the subsequently recovered portion of the received live data to the second data buffer for forwarding of the subsequently recovered portion of the received live data.

10. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data includes another portion of the received live data from the first data buffer in a response to a pull request.

11. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data is based on a response to a pull request emptying the first data buffer.

12. The computer-implemented method of claim 1, wherein the forwarding of the recovered portion of the received live data is based on the first data buffer having less than an amount of data requested by a pull request.

13. The computer-implemented method of claim 1, further comprising removing the portion of the received live data from the backup of the received live data based on receiving an acknowledgement in response to the forwarding of the recovered portion of the received live data.

14. The computer-implemented method of claim 1, wherein the recovering of the portion of the received live data from the backup comprises:

closing a transaction log in the backup of the received live data;

opening a new transaction log for a backup of subsequently received live data;

retrieving the portion of the received live data from the closed transaction log; and

generating the backup of the subsequently received live data in the new transaction log.

15. The computer-implemented method of claim 1, wherein the backup of the received live data is generated on a hard disk.

16. The computer-implemented method of claim 1, wherein the live data buffer comprises a live data queue and the stale data buffer comprises a stale data queue.

17. The computer-implemented method of claim 1, wherein the backup comprises transactions and transaction identifiers corresponding to the received live data.

18. The computer-implemented method of claim 1, further comprising:

receiving an acknowledgement in response to the forwarding of the recovered portion of the received live data; and

recording the received acknowledgement in an acknowledgement log.

19. The computer-implemented method of claim 1, wherein the recovering of the portion of the received live data from the backup comprises:

closing an acknowledgement log comprising recorded acknowledgments of forwarded data;

opening a new acknowledgement log for recording subsequent acknowledgments of forwarded data;

identifying the portion of the received live data in the backup based on a comparison between the closed acknowledgment log and the backup of received live data; and

recording the subsequent acknowledgments of forwarded data in the new acknowledgment log.

20. The computer-implemented method of claim 1, further comprising sending the recovered portion of the received live data to an output interface of a data forwarder for forwarding.

21. The computer-implemented method of claim 19, wherein the recovering of the portion of the received live data from the backup comprises:

closing a transaction log in the backup of the received live data;

opening a new transaction log for a backup of subsequently received live data;

retrieving the identified portion of the received live data from the closed transaction log based on the identifying; and

generating the backup of the subsequently received live data in the new transaction log.

22. The computer-implemented method of claim 1, further comprising sending at least some of the received live data to an output interface of a data forwarder for forwarding.

23. A system for forwarding data, the system comprising:

one or more data processors; and

one or more computer-readable storage media containing instructions which when executed on the one or more data processors, cause the one or more processors to perform operations including;

generating a backup of received live data;

providing the received live data to a first data buffer for forwarding of the received live data, wherein a portion of the received live data is dropped prior to being forwarded;

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer for forwarding of the recovered portion of the received live data.

24. The system of claim 23, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

25. The system of claim 23, wherein the portion of the received live data is dropped from the first data buffer.

26. The system of claim 23, wherein the forwarding of the received live data is in response to a pull request.

27. One or more computer-storage media storing computer-useable instructions that, when executed by a computing device, perform a method for forwarding data, the method comprising:

generating a backup of received live data;

providing the received live data to a first data buffer for forwarding of the received live data, wherein a portion of the received live data is dropped prior to being forwarded;

based on the portion of the received live data being dropped, recovering the portion of the received live data from the backup and providing the recovered portion of the received live data to a second data buffer for forwarding of the recovered portion of the received live data.

28. The one or more computer-storage media of claim 27, wherein the portion of the received live data is dropped based on the first data buffer being at a capacity.

29. The one or more computer-storage media of claim 27, wherein the portion of the received live data is dropped from the first data buffer.

30. The one or more computer-storage media of claim 27, wherein the forwarding of the received live data is in response to a pull request.