Batching tuples

• US 10,268,727 B2
• Filed: 03/29/2013
• Issued: 04/23/2019
• Est. Priority Date: 03/29/2013
• Status: Expired due to Fees

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Sax, Matthias J., Castellanos, Maria Guadalupe, Hsu, Meichun
• Primary Examiner(s): Morrison, Jay A
• Assistant Examiner(s): Hoang, Ken
• Application Number:
  US14/780,641
• Publication Number:
  US 20160154853A1
• Time in Patent Office: 2,216 Days
• Field of Search: 707602, 707713, 707714, 707999002, 707999003, 707999005, 707999102, 707E17005, 707E17059, 707706, 707707, 707709, 707711, 707712, 707715, 707716, 707717, 707718, 707719, 707721, 707723, 707736, 707747, 707752, 707754, 707758, 707769, 707781, 707797, 707809, 707966, 707971, 707972, 7079991, 707999101, 707999104, 717104, 717127
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

A technique of batching tuples can include determining a plurality of key-attributes for a plurality of tuples, creating a batch tuple, and calculating a hash value for the batch tuple.

Claims

1. A method for batching tuples comprising:

determining a plurality of key-attributes for each of a plurality of tuples, wherein each key-attribute of each tuple describes a type or classification of data in the tuple;

creating, by a processor of a publisher operator, a batch tuple in a buffer containing a plurality of lists based on the plurality of key-attributes of the plurality of tuples, including;

creating a series of intermediate batch tuples, wherein each of the intermediate batch tuples is created for one of the tuples and contains the plurality of lists, and

adding the plurality of key-attributes of each of the tuples into the plurality of lists of one of the intermediate batch tuples in a sequential order such that a last one of the series of intermediate batch tuples contains the plurality of key-attributes of all of the tuples in the plurality of lists;

calculating, by the processor, a hash value for the batch tuple in the buffer based on the plurality of lists;

when the buffer containing the batch tuple is full, determining, by the processor, a subscriber operator instance having a same hash value as the batch tuple; and

sending, by the processor, the batch tuple in the buffer to the subscriber operator instance that has the same hash value as the batch tuple.

2. The method of claim 1, wherein each of the plurality of lists contains a key-attribute among the plurality of key-attributes from each of the plurality of tuples.

3. The method of claim 1, wherein the plurality of lists in each of the intermediate batch tuples includes a number of intermediate lists, wherein each key-attribute from the plurality of key-attributes for the plurality of tuples is added to the number of intermediate lists.

4. The method of claim 1, wherein creating the batch tuple includes creating a first list in the plurality of lists, wherein each key-attribute in the first list includes a same hash value.

5. The method of claim 1, wherein calculating the hash value for the batch tuple includes calculating a hash value for each of the plurality of lists in the batch tuple.

6. A non-transitory computer-readable medium storing a set of instructions executable by a processing resource to cause a computer to:

determine a plurality of key-attributes for each of a plurality of tuples, wherein each key-attribute of each tuple describes a type or classification of data in the tuple;

create a particular batch tuple in a buffer containing a plurality of lists, by;

creating a series of intermediate batch tuples, wherein each of the intermediate batch tuples contains the plurality of lists, and

adding the plurality of key-attributes of each of the tuples into the plurality of lists of one of the intermediate batch tuples in a sequential order such that a last one of the series of intermediate batch tuples contains the plurality of key-attributes of all of the tuples in the plurality of lists;

calculate a hash value for the batch tuple based on the hash value for the plurality of key-attributes; and

when buffer containing the particular batch tuple is full, determining a corresponding subscriber operator instance having a same hash value as the particular batch tuple and send the particular batch tuple to the corresponding subscriber operator instance.

7. The non-transitory computer-readable medium of claim 6, wherein the set of instructions executable by the processing resource includes instructions to:

in addition to the particular batch tuple, create a first distinct batch tuple in a first output buffer for a first corresponding subscriber operator instance, wherein the first distinct batch tuple includes a key for all tuples in a first group; and

create a second distinct batch tuple in a second output buffer for a second corresponding subscriber operator instance, wherein the second distinct batch tuple includes a key for all tuples in a second group.

8. The non-transitory computer-readable medium of claim 7, wherein the set of instructions executable by the processing resource includes instructions to send each of the first and second distinct batch tuples from one of the first and second output buffers in a producer operator instance to an input buffer in the corresponding subscriber operator instance.

9. The non-transitory computer-readable medium of claim 7, wherein the set of instructions executable by the processing resource includes instructions to unpack and deserialize the tuples in each of the first and second distinct batch tuples upon receipt by the corresponding subscriber operator instance.

12. The system of claim 9, wherein the instructions are to cause the processing resource to send the particular batch tuple to a first subscriber operator instance and a second subscriber operator instance if a hash value of the key-attributes of the first subscriber operator instance equals a hash value of the key-attributes of the second subscriber operator instance.

13. The system of claim 12, wherein the instructions are to cause the processing resource to create a shared batch corresponding to the first subscriber operator instance and the second subscriber operator instance.

10. A system for batching tuples in a distributed streaming system, the system comprising:

a processing resource;

a memory resource storing instructions that when executed cause the processing resource to;

determine a plurality of key-attributes for each of a plurality of tuples, wherein each key-attribute of each tuple describes a type or classification of data in the tuple;

create a particular batch tuple in a buffer containing a plurality of lists, by;

creating a series of intermediate batch tuples, wherein each of the intermediate batch tuples contains the plurality of lists, and

adding the plurality of key-attributes of each of the tuples into the plurality of lists of one of the intermediate batch tuples in a sequential order such that a last one of the series of intermediate batch tuples contains the plurality of key-attributes of all of the tuples in the plurality of lists;

calculate a hash value for the particular batch tuple based on the plurality of key-attributes; and

when the buffer containing the particular batch tuple is full, determine a corresponding subscriber operator instance having a same hash value as the particular batch tuple and send the particular batch tuple to the corresponding subscriber operator instance.

11. The system of claim 10, wherein the instructions are executable to cause the processing resource to create additional batch tuples in output buffers to send to subscriber operator instances, wherein a total number of the output buffers is equal to a total number of the subscriber operator instances.

14. The system of claim 10, wherein the particular batch tuple includes a key portion containing a number of key-attributes, and a payload portion that includes a number of grouped tuples.