Distributing and processing streams over one or more networks

• US 9,239,864 B2
• Filed: 04/17/2013
• Issued: 01/19/2016
• Est. Priority Date: 04/17/2012
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Krishnamurthy, Saileshwar, Kumar, Madhu, Bhat, Amit, Sakrejda, Maciek, Lerche, Robert Allen
• Primary Examiner(s): Choi, Yuk Ting
• Application Number:
  US13/865,054
• Publication Number:
  US 20130275452A1
• Time in Patent Office: 1,007 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

In an embodiment, a method for distributing and processing streams over wide area networks comprises receiving, at a unified data processing node, a continuous query; determining a parallel portion of the continuous query; sending the parallel portion to a plurality of distributed data processing nodes located in a plurality of data centers; at each distributed node in the plurality of distributed nodes, locally executing the parallel portion against independent data partitions, producing a partial summary data, sending the partial summary data to the unified node; continuously receiving, at the unified node, in real-time, the partial summary data.

Claims

1. A method comprising:

receiving, at a unified data processing node, a continuous query;

determining a parallel portion of the continuous query;

sending the parallel portion to a plurality of distributed data processing nodes comprising at least a first distributed data processing node in a first data center and a second distributed data processing node in a second data center, wherein the first data center and the second data center are separate;

the first distributed data processing node locally processing the parallel portion against a first independent data stream to produce a first partial summary data, and sending the first partial summary data to the unified data processing node, wherein the first independent data stream is not duplicated or processed by the unified data processing node or the second distributed data processing node;

the second distributed data processing node locally processing the parallel portion against a second independent data stream to produce a second partial summary data, and sending the second partial summary data to the unified data processing node, wherein the second independent data stream is not duplicated or processed by the unified data processing node or the first distributed data processing node;

continuously receiving, at the unified data processing node, in real-time, streaming data results comprising the first partial summary data and the second partial summary data and combining the first partial summary data and the second partial summary data in preparation for returning combined streaming query results to an application; and

wherein the method is performed by one or more computing devices.

2. The method of claim 1 comprising:

determining a serial portion of the continuous query; and

executing, at the unified node, the serial portion using as input the first partial summary data and the second partial summary data.

3. The method of claim 1, comprising:

receiving, from at least one distributed node of the plurality of distributed nodes, a poll message requesting the parallel portion; and

in response to receiving the poll message, sending the parallel portion to the at least one distributed node.

4. The method of claim 1, comprising pushing the parallel portion to at least one distributed node of the plurality of distributed nodes.

5. The method of claim 1, wherein the distributed nodes are in a hierarchy of distributed nodes.

6. The method of claim 1, wherein the first data center and the second data center are in a hierarchy.

7. The method of claim 1, wherein the unified node is a first unified node in a plurality of unified nodes, and comprising sending a partial summary data for each distributed node in the plurality of distributed nodes to each unified node in the plurality of unified nodes.

8. The method of claim 1 comprising storing an archive of partial summary data of each distributed node.

9. The method of claim 1 comprising:

storing an identifier that identifies a portion of partial summary data received;

receiving an inquiry message, comprising the identifier, about whether the portion of partial summary data was received; and

sending a response indicating that the portion of partial summary data was received.

10. The method of claim 1, comprising receiving the continuous query directed to continuous streaming data.

11. The method of claim 1, wherein the parallel portion is a first parallel portion and further comprising:

determining a second parallel portion that depends on the first parallel portion;

sending the second parallel portion to the distributed nodes; and

at each distributed node of the distributed nodes, locally beginning execution of the first parallel portion before beginning execution of the second parallel portion.

12. The method of claim 11 comprising:

a node, of the distributed nodes, suspends processing; and

subsequently, the node resuming processing, wherein the node resumes execution of the first parallel portion before the node resumes execution of the second parallel portion.

13. A system comprising:

a unified node comprising a processor;

a plurality of distributed data processing nodes, each comprising a processor, the plurality of distributed data processing nodes comprising at least a first distributed data processing node in a first data center and a second distributed data processing node in a second data center, wherein the first data center and the second data center are separate;

wherein the unified node is configured to;

receive a continuous query;

determine a parallel portion of the continuous query;

send the parallel portion to at least the first and second distributed data processing nodes of the plurality of distributed data processing nodes; and

receive streaming data results comprising a first partial summary data and a second partial summary data in real-time and combine the first partial summary data and the second partial summary data in preparation for returning combined streaming query results to an application; and

wherein the first distributed data processing node in the plurality of distributed data processing nodes is configured to;

processing the parallel portion against a first independent data stream to produce the first partial summary data, wherein the first independent data stream is not duplicated or processed by the unified data processing node or the second distributed data processing node; and

send the first partial summary data to the unified node;

wherein the second distributed data processing node in the plurality of distributed data processing nodes is configured to;

process the parallel portion against a second independent data stream to produce the second partial summary data, wherein the second independent data stream is not duplicated or processed by the unified data processing node or the first distributed data processing node; and

send the second partial summary data to the unified node.

14. The system of claim 13 wherein the unified node is configured to:

determine a serial portion of the continuous query; and

execute the serial portion using as input the first partial summary data and the second partial summary data.

15. The system of claim 13 where the unified node is configured to:

receive, from at least one distributed node of the plurality of distributed nodes, a poll message requesting the parallel portion; and

in response to receiving the poll message, send the parallel portion to the at least one distributed node.

16. The system of claim 13 wherein the unified node is configured to push the parallel portion to at least one distributed node of the plurality of distributed nodes.

17. The system of claim 13, wherein the distributed nodes are configured in a hierarchy of distributed nodes.

18. The system of claim 13, wherein the first data center and the second data center are in a hierarchy.

19. The system of claim 13, wherein:

the unified node is a first unified node in a plurality of unified nodes; and

each distributed node of the distributed nodes is configured to send a partial summary data for the distributed node to each unified node in the plurality of unified nodes.

20. The system of claim 13 each distributed node of the distributed nodes is configured to store an archive of partial summary data of the distributed node.

21. The system of claim 13 wherein the unified node is configured to:

store an identifier that identifies a portion of partial summary data received;

receive an inquiry message, comprising the identifier, about whether the portion of the partial summary data was received; and

send a response indicating that the portion of partial summary data was received.

22. The system of claim 13 wherein the unified node is configured to receive the continuous query directed to continuous streaming data.

23. The system of claim 13, wherein:

the parallel portion is a first parallel portion;

the unified node is configured to;

determine a second parallel portion that depends on the first parallel portion; and

send the second parallel portion to the distributed nodes; and

each distributed node of the distributed nodes is configured to;

locally begin executing the first parallel portion before beginning to execute the second parallel portion.

24. The system of claim 23 wherein a node, of the distributed nodes, is configured to:

suspend processing; and

subsequently resume processing, wherein the node resumes execution of the first parallel portion before the node resumes execution of the second parallel portion.

25. One or more non-transitory computer-readable media storing one or more sequences of instructions which, when executed by one or more computing devices, cause:

receiving, at a unified data processing node, a continuous query;

determining a parallel portion of the continuous query;

sending the parallel portion to a plurality of distributed data processing nodes comprising at least a first distributed data processing node in a first data center and a second distributed data processing node in a second data center, wherein the first data center and the second data center are separate;

the first distributed data processing node locally processing the parallel portion against a first independent data stream to produce a first partial summary data, and sending the first partial summary data to the unified data processing node, wherein the first independent data stream is not duplicated or processed by the unified data processing node or the second distributed data processing node;

the second distributed data processing node locally processing the parallel portion against a second independent data stream to produce a second partial summary data, and sending the second partial summary data to the unified data processing node, wherein the second independent data stream is not duplicated or processed by the unified data processing node or the first distributed data processing node;

continuously receiving, at the unified data processing node, in real-time, streaming data results comprising the first partial summary data and the second partial summary data and combining the first partial summary data and the second partial summary data in preparation for returning combined streaming query results to an application.