Speculative data processing of streaming data

• US 9,438,676 B2
• Filed: 07/25/2014
• Issued: 09/06/2016
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Krishnamurthy, Saileshwar, Kumar, Madhu, Bhat, Amit
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US14/341,436
• Publication Number:
  US 20160026530A1
• Time in Patent Office: 774 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0796 Safety measures, i.e. ensuring safe condition in the event of error, e.g. for controlling element
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

In an embodiment, a method comprises receiving a plurality of source data records from one or more source computers, wherein one or more first source data records are associated with a first source transaction and one or more second source data records are associated with a second source transaction; generating a first derived transaction comprising one or more first derived records based on the plurality of source data records; generating a first transaction mapping between the first derived transaction and the first source transaction; generating a second transaction mapping between the first derived transaction and the second source transaction; determining that the first derived transaction has ended, and in response, committing first derived transaction including the one or more first derived records to a persistent storage; receiving a first ending punctuation associated with the first source transaction, and in response, committing the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage; after committing the first derived transaction, determining that the second source transaction failed, and in response: determining that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerating the first derived transaction.

Claims

1. A computer system comprising:

a memory;

a processor coupled to the memory;

processing logic coupled to the memory and the processor and configured to;

receive a plurality of source data records from one or more source computers, wherein one or more first source data records are associated with a first source transaction and one or more second source data records are associated with a second source transaction;

generate a first derived transaction including one or more first derived records based on the plurality of source data records;

generate a first transaction mapping between the first derived transaction and the first source transaction;

generate a second transaction mapping between the first derived transaction and the second source transaction;

determine that the first derived transaction has ended, and in response, commit first derived transaction including the one or more first derived records to a persistent storage;

receive a first ending punctuation associated with the first source transaction, and in response, commit the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage;

after committing the first derived transaction, determine that the second source transaction failed, and in response;

determine that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerate the first derived transaction by;

generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records;

committing the one or more second derived records to the persistent storage.

2. The computer system of claim 1, wherein the processing logic is configured to send, in response to determine that the first derived transaction has ended, the one or more first derived records to a downstream device without determining that the second source transaction has failed.

3. The computer system of claim 1, wherein the processing logic is configured to:

receive a first beginning punctuation associated with the first source transaction from a first source computer, wherein the one or more first source data records are received from the first source computer;

in response to receiving the first ending punctuation, update the first transaction mapping to indicate that the first source transaction is complete.

4. The computer system of claim 1, wherein after committing the one or more first derived records, the processing logic is configured to:

receive a late source record that is associated with the first source transaction;

determine that the late source record could have been processed with the first derived transaction based on the first transaction mapping between the first source transaction and the first derived transaction, and in response, flag the first derived transaction to be regenerated.

5. The computer system of claim 1, wherein the processing logic is configured to:

associate each record of the one or more first derived records with the first derived transaction;

in response to determining that the second source transaction failed, delete each record associated with the first derived transaction from the persistent storage.

6. The computer system of claim 1, wherein the processing logic is configured to send, after regenerating the first derived transaction, the one or more second derived records to a downstream device.

7. The computer system of claim 1, wherein the processing logic is configured to:

associate the one or more first derived records with an identifier that identifies the first derived transaction;

in response to determining that the first derived transaction has ended, send the one or more first derived records to a downstream device with the identifier;

in response to determining that the second source transaction failed and generating the one or more second derived records;

associate the one or more second derived records with the identifier;

notify the downstream device that the first derived transaction, which is associated with the identifier, was regenerated.

8. The computer system of claim 1, wherein the processing logic is configured to receive a request for the first derived transaction, and in response, determine whether the first derived transaction should be regenerated.

9. The computer system of claim 1, wherein the processing logic is configured to:

associate an identifier with the first derived transaction;

in response to determining that the first derived transaction has ended, send the one or more first derived records to a downstream device with the identifier;

associate the one or more second derived records with the identifier;

subsequent to generating the one or more second derived records;

receive a request with the identifier from the downstream device;

in response to receiving the request, send the one or more second derived records to the downstream device, but not the one or more first derived records.

10. A method comprising:

receiving a plurality of source data records from one or more source computers, wherein one or more first source data records are associated with a first source transaction and one or more second source data records are associated with a second source transaction;

generating a first derived transaction comprising one or more first derived records based on the plurality of source data records;

generating a first transaction mapping between the first derived transaction and the first source transaction;

generating a second transaction mapping between the first derived transaction and the second source transaction;

determining that the first derived transaction has ended, and in response, committing first derived transaction including the one or more first derived records to a persistent storage;

receiving a first ending punctuation associated with the first source transaction, and in response, committing the first source transaction including the one or more first source data records associated with the first source transaction to the persistent storage;

after committing the first derived transaction, determining that the second source transaction failed, and in response;

determining that the first derived transaction should be regenerated based on the second transaction mapping, and in response, regenerating the first derived transaction by;

generating one or more second derived records that are based on the one or more first source data records but not the one or more second source data records;

committing the one or more second derived records to the persistent storage;

wherein the method is performed by one or more computing devices.

11. The method claim 10 comprising, in response to determining that the first derived transaction has ended, sending the one or more first derived records to a downstream device without determining that the second source transaction has failed.

12. The method of claim 10 comprising:

receiving a first beginning punctuation associated with the first source transaction from a first source computer, wherein the one or more first source data records are received from the first source computer;

in response to receiving the first ending punctuation, updating the first transaction mapping to indicate that the first source transaction is complete.

13. The method of claim 10 comprising:

after committing the one or more first derived records, receiving a late source record that is associated with the first source transaction;

determining that the late source record should have been processed with the first derived transaction based on the first transaction mapping between the first source transaction and the first derived transaction, and in response, flagging the first derived transaction to be regenerated.

14. The method of claim 10 comprising:

associating each record of the one or more first derived records with the first derived transaction;

in response to determining that the second source transaction failed, deleting each record associated with the first derived transaction from the persistent storage.

15. The method claim 10 comprising, after regenerating the first derived transaction, sending the one or more second derived records to a downstream device.

16. The method of claim 10 comprising:

associating the one or more first derived records with an identifier that identifies the first derived transaction;

in response to determining that the first derived transaction has ended, sending the one or more first derived records to a downstream device with the identifier;

in response to determining that the second source transaction failed and generating the one or more second derived records;

associating the one or more second derived records with the identifier;

notifying the downstream device that the first derived transaction, which is associated with the identifier, was regenerated.

17. The method of claim 10 comprising receiving a request for the first derived transaction, and in response, determining whether the first derived transaction should be regenerated.

18. The method of claim 10 comprising:

associating an identifier with the first derived transaction;

in response to determining that the first derived transaction has ended, sending the one or more first derived records to a downstream device with the identifier;

associating the one or more second derived records with the identifier;

subsequent to generating the one or more second derived records;

receiving a request with the identifier from the downstream device;

in response to receiving the request, sending the one or more second derived records to the downstream device, but not the one or more first derived records.