Aggregating data in a mediation system

• US 9,185,235 B2
• Filed: 08/02/2012
• Issued: 11/10/2015
• Est. Priority Date: 08/02/2012
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Rossi, Larry Paul
• Primary Examiner(s): Btit, Jacob F
• Assistant Examiner(s): Lin, Allen
• Application Number:
  US13/565,321
• Publication Number:
  US 20140040213A1
• Time in Patent Office: 1,195 Days
• Field of Search: 707/737
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: H04M 15/00 Arrangements for metering, time-control or time indication ; Metering, charging or billing arrangements for voice wireline or wireless communications, e.g. VoIP
• CPC Class Codes: H04M 15/41 Billing record details, i.e. parameters, identifiers, structure of call data record [CDR]
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 15/44 Augmented, consolidated or itemized billing statement or bill presentation
• CPC Class Codes: H04M 15/53 using mediation
• CPC Class Codes: H04M 15/62 based on trigger specification

Abstract

Records received from one or more sources in a network are processed. For each of multiple intervals of time, a matching procedure is attempted on sets of one or more records, including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets. The processing also includes, for at least some of the intervals of time, processing at least one complete set, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time, and for at least some of the intervals of time, processing at least one incomplete set, consisting of one or more records stored in the data store before the interval of time.

Claims

1. A method for processing data received at a node in a network, the method including:

receiving records from one or more sources in the network, at least some of the records each being associated with an identifier;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time; and

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion; and

wherein the expiration criterion depends on a degree of incompleteness of the first incomplete set of records.

2. The method of claim 1, further including:

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time.

3. The method of claim 2, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

4. The method of claim 2, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

5. The method of claim 2, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer.

6. The method of claim 5, wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes:

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

7. The method of claim 1, wherein the key associated with a set of records includes the identifier associated with the set of records.

8. The method of claim 1, wherein the intervals of time are determined based on a clock at the node.

9. The method of claim 1, wherein the intervals of time are determined based on a number of records received.

10. The method of claim 1, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

11. The method of claim 1, wherein the data store enables random access of records stored in the data store.

12. The method of claim 1, wherein processing a set of records includes sending information in the records in the set to another node in the network.

13. A non-transitory computer-readable medium storing a computer program for processing data received at a node in a network, the computer program including instructions for causing a computing system to:

receive records from one or more sources in the network, at least some of the records each being associated with an identifier;

for each of multiple intervals of time, attempt a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, process at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time; and

for at least some of the intervals of time, process at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion; and

wherein the expiration criterion depends on a degree of incompleteness of the first incomplete set of records.

14. The non-transitory computer-readable medium of claim 13, wherein the processing further includes:

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time.

15. The non-transitory computer-readable medium of claim 14, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

16. The non-transitory computer-readable medium of claim 14, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

17. The non-transitory computer-readable medium of claim 14, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer.

18. The non-transitory computer-readable medium of claim 17, wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes:

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

19. The non-transitory computer-readable medium of claim 13, wherein the key associated with a set of records includes the identifier associated with the set of records.

20. The non-transitory computer-readable medium of claim 13, wherein the intervals of time are determined based on a clock at the node.

21. The non-transitory computer-readable medium of claim 13, wherein the intervals of time are determined based on a number of records received.

22. The non-transitory computer-readable medium of claim 13, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

23. The non-transitory computer-readable medium of claim 13, wherein the data store enables random access of records stored in the data store.

24. The non-transitory computer-readable medium of claim 13, wherein processing a set of records includes sending information in the records in the set to another node in the network.

25. A network node, including:

a network interface configured to receive records from one or more sources in the network, at least some of the records each being associated with an identifier; and

at least one processor configured to process sets of records, the processing including;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time; and

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion; and

wherein the expiration criterion depends on a degree of incompleteness of the first incomplete set of records.

26. The network node of claim 25, wherein the processing further includes:

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time.

27. The network node of claim 26, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

28. The network node of claim 26, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

29. The network node of claim 26, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer.

30. The network node of claim 29, wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes:

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

31. The network node of claim 26, wherein the data structure has a relatively faster access speed than the data store.

32. The network node of claim 31, wherein the data store includes a non-volatile storage device, and the data structure includes a volatile memory.

33. The network node of claim 25, wherein the key associated with a set of records includes the identifier associated with the set of records.

34. The network node of claim 25, wherein the intervals of time are determined based on a clock at the node.

35. The network node of claim 25, wherein the intervals of time are determined based on a number of records received.

36. The network node of claim 25, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

37. The network node of claim 25, wherein the data store enables random access of records stored in the data store.

38. The network node of claim 25, wherein processing a set of records includes sending information in the records in the set to another node in the network.

39. A network node, including:

means for receiving records from one or more sources in the network, at least some of the records each being associated with an identifier; and

means for processing sets of records, the processing including;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time; and

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion; and

wherein the expiration criterion depends on a degree of incompleteness of the first incomplete set of records.

40. The network node of claim 39, wherein the processing further includes:

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time.

41. The network node of claim 40, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

42. The network node of claim 40, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

43. The network node of claim 40, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer.

44. The network node of claim 43, wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes:

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

45. The network node of claim 40, wherein the data structure has a relatively faster access speed than the data store.

46. The network node of claim 45, wherein the data store includes a non-volatile storage device, and the data structure includes a volatile memory.

47. The network node of claim 39, wherein the key associated with a set of records includes the identifier associated with the set of records.

48. The network node of claim 39, wherein the intervals of time are determined based on a clock at the node.

49. The network node of claim 39, wherein the intervals of time are determined based on a number of records received.

50. The network node of claim 39, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

51. The network node of claim 39, wherein the data store enables random access of records stored in the data store.

52. The network node of claim 39, wherein processing a set of records includes sending information in the records in the set to another node in the network.

53. A method for processing data received at a node in a network, the method including:

receiving records from one or more sources in the network, at least some of the records each being associated with an identifier, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time;

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time;

wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes;

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

54. The method of claim 53, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

55. The method of claim 53, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

56. The method of claim 55, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

looking up the distinct identifier in an index associated with the data store.

57. The method of claim 56, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

decompressing a portion of the data store that includes any records associated with the distinct identifier.

58. The method of claim 57, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

scanning the decompressed portion of the data store to locate any records associated with the distinct identifier.

59. The method of claim 55, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining whether or not a stored set of records associated with that distinct identifier has been identified as having been processed.

60. The method of claim 55, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining a hash value based on the distinct identifier.

61. The method of claim 53, further including, for at least some of the intervals of time, processing at least one complete set of records, and storing in the data store information identifying the processed complete set of records as having been processed.

62. The method of claim 53, further including, for at least some of the intervals of time, processing at least one incomplete set of records, and storing in the data store information identifying the processed incomplete set of records as having been processed.

63. The method of claim 53, wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion.

64. The method of claim 53, wherein the key associated with a set of records includes the identifier associated with the set of records.

65. The method of claim 53, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

66. The method of claim 53, wherein the data store enables random access of records stored in the data store.

67. A non-transitory computer-readable medium storing a computer program for processing data received at a node in a network, the computer program including instructions for causing a computing system to:

receive records from one or more sources in the network, at least some of the records each being associated with an identifier, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer;

for each of multiple intervals of time, attempt a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, process at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time;

for at least some of the intervals of time, process at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time;

wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes;

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

68. The non-transitory computer-readable medium of claim 67, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

69. The non-transitory computer-readable medium of claim 67, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

70. The non-transitory computer-readable medium of claim 69, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

looking up the distinct identifier in an index associated with the data store.

71. The non-transitory computer-readable medium of claim 70, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

decompressing a portion of the data store that includes any records associated with the distinct identifier.

72. The non-transitory computer-readable medium of claim 71, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

scanning the decompressed portion of the data store to locate any records associated with the distinct identifier.

73. The non-transitory computer-readable medium of claim 69, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining whether or not a stored set of records associated with that distinct identifier has been identified as having been processed.

74. The non-transitory computer-readable medium of claim 69, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining a hash value based on the distinct identifier.

75. The non-transitory computer-readable medium of claim 67, wherein the processing further includes, for at least some of the intervals of time, processing at least one complete set of records, and storing in the data store information identifying the processed complete set of records as having been processed.

76. The non-transitory computer-readable medium of claim 67, wherein the processing further includes, for at least some of the intervals of time, processing at least one incomplete set of records, and storing in the data store information identifying the processed incomplete set of records as having been processed.

77. The non-transitory computer-readable medium of claim 67, wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion.

78. The non-transitory computer-readable medium of claim 67, wherein the key associated with a set of records includes the identifier associated with the set of records.

79. The non-transitory computer-readable medium of claim 67, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

80. The non-transitory computer-readable medium of claim 67, wherein the data store enables random access of records stored in the data store.

81. A network node, including:

a network interface configured to receive records from one or more sources in the network, at least some of the records each being associated with an identifier, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer; and

at least one processor configured to process sets of records, the processing including;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time;

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time;

wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes;

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

82. The network node of claim 81, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

83. The network node of claim 81, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

84. The network node of claim 83, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

looking up the distinct identifier in an index associated with the data store.

85. The network node of claim 84, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

decompressing a portion of the data store that includes any records associated with the distinct identifier.

86. The network node of claim 85, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

scanning the decompressed portion of the data store to locate any records associated with the distinct identifier.

87. The network node of claim 83, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining whether or not a stored set of records associated with that distinct identifier has been identified as having been processed.

88. The network node of claim 83, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining a hash value based on the distinct identifier.

89. The network node of claim 81, wherein the data structure has a relatively faster access speed than the data store.

90. The network node of claim 89, wherein the data store includes a non-volatile storage device, and the data structure includes a volatile memory.

91. The network node of claim 81, wherein the processing further includes, for at least some of the intervals of time, processing at least one complete set of records, and storing in the data store information identifying the processed complete set of records as having been processed.

92. The network node of claim 81, wherein the processing further includes, for at least some of the intervals of time, processing at least one incomplete set of records, and storing in the data store information identifying the processed incomplete set of records as having been processed.

93. The network node of claim 81, wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion.

94. The network node of claim 81, wherein the key associated with a set of records includes the identifier associated with the set of records.

95. The network node of claim 81, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

96. The network node of claim 81, wherein the data store enables random access of records stored in the data store.

97. A network node, including:

means for receiving records from one or more sources in the network, at least some of the records each being associated with an identifier, wherein receiving records from one or more sources in the network includes storing the received records in an input buffer; and

means for processing sets of records, the processing including;

for each of multiple intervals of time, attempting a matching procedure on sets of one or more records, the matching procedure including comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets, where the completeness criterion for a particular set of one or more records is based at least in part on a result of aggregating at least some records in that particular set;

for at least some of the intervals of time, processing at least one complete set of records, for which the completeness criterion was satisfied during that attempt, consisting of one or more of the received records on which the matching procedure is first attempted during the interval of time and one or more records stored in a data store before the interval of time;

for at least some of the intervals of time, processing at least one incomplete set of records, for which the completeness criterion was not satisfied during that attempt, consisting of one or more records stored in the data store before the interval of time;

after at least some attempts of the matching procedure, storing in the data store at least some incomplete sets of records for which the completeness criterion was not satisfied during that attempt, and storing keys used to uniquely identify the incomplete sets of records in a data structure separate from the data store; and

retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time;

wherein retrieving stored keys from the data structure to attempt the matching procedure on corresponding records during a subsequent interval of time includes;

removing a key from the data structure, and adding the removed key or a record corresponding to the removed key to the input buffer.

98. The network node of claim 97, wherein the intervals of time define a unit of work in terms of a predetermined number of records or a predetermined amount of time, and the stored keys are retrieved from the data structure asynchronously with the intervals of time.

99. The network node of claim 97, wherein comparing identifiers associated with different records to generate the sets and determining whether or not a completeness criterion is satisfied for one or more of the sets of records includes:

determining distinct identifiers associated with the received records,

for each distinct identifier, retrieving any records stored in the data store associated with that distinct identifier, and aggregating a set of records associated with that identifier that includes one or more of the received records and any retrieved records, and

determining whether or not the completeness criterion is satisfied for each aggregated set of records.

100. The network node of claim 99, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

looking up the distinct identifier in an index associated with the data store.

101. The network node of claim 100, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

decompressing a portion of the data store that includes any records associated with the distinct identifier.

102. The network node of claim 101, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

scanning the decompressed portion of the data store to locate any records associated with the distinct identifier.

103. The network node of claim 99, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining whether or not a stored set of records associated with that distinct identifier has been identified as having been processed.

104. The network node of claim 99, wherein retrieving any records stored in the data store associated with that distinct identifier includes:

determining a hash value based on the distinct identifier.

105. The network node of claim 97, wherein the data structure has a relatively faster access speed than the data store.

106. The network node of claim 105, wherein the data store includes a non-volatile storage device, and the data structure includes a volatile memory.

107. The network node of claim 97, wherein the processing further includes, for at least some of the intervals of time, processing at least one complete set of records, and storing in the data store information identifying the processed complete set of records as having been processed.

108. The network node of claim 97, wherein the processing further includes, for at least some of the intervals of time, processing at least one incomplete set of records, and storing in the data store information identifying the processed incomplete set of records as having been processed.

109. The network node of claim 97, wherein after at least a first attempt of the matching procedure is attempted on a first incomplete set of records during a first interval of time, and a second attempt of the matching procedure is attempted on the first incomplete set of records during a second interval of time, comparing a time associated with at least one record in the first incomplete set of records with an expiration criterion, and processing the first incomplete set of records in response to the comparison of the time associated with at least one record in the incomplete set of records with the expiration criterion.

110. The network node of claim 97, wherein the key associated with a set of records includes the identifier associated with the set of records.

111. The network node of claim 97, wherein one or more of the intervals of time are included within a checkpoint interval in which data associated with the matching procedure that has been received or generated since a previous checkpoint interval is persistently stored.

112. The network node of claim 97, wherein the data store enables random access of records stored in the data store.