SELECTING QUERIES FOR EXECUTION ON A STREAM OF REAL-TIME DATA

• US 20170039245A1
• Filed: 08/05/2015
• Published: 02/09/2017
• Est. Priority Date: 08/05/2015
• Status: Active Grant

• Current Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Original Assignee: Ab Initio Technology LLC (Ab Initio Software Corporation)
• Inventors: Gadodia, Rajesh, Wholey, Joseph Skeffington III
• Granted Patent: US 10,657,134 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

A computer-implemented method for executing a query on data items located at different places in a stream of near real-time data to provide near-real time intermediate results for the query, as the query is being executed, the method including: from time to time, executing, by one or more computer systems, the query on two or more of the data items located at different places in the stream, with the two or more data items being accessed in near real-time with respect to each of the two or more data items; generating information indicative of results of executing the query; and as the query continues being executed, generating intermediate results of query execution by aggregating the results with prior results of executing the query on data items that previously appeared in the stream of near real-time data; and transmitting to one or more client devices the intermediate results of query execution, prior to completion of execution of the query.

Claims

1. A computer-implemented method for executing a query on data items located at different places in a stream of near real-time data to provide near-real time intermediate results for the query, as the query is being executed, the method including:

from time to time,

executing, by one or more computer systems, the query on two or more of the data items located at different places in the stream, with the two or more data items being accessed in near real-time with respect to each of the two or more data items;

generating information indicative of results of executing the query; and

as the query continues being executed,

generating intermediate results of query execution by aggregating the results with prior results of executing the query on data items that previously appeared in the stream of near real-time data; and

transmitting to one or more client devices the intermediate results of query execution, prior to completion of execution of the query.

2. The computer-implemented method of claim 1, further including:

at a subsequent point in time, aggregating the intermediate results with results for executing the query at the subsequent point in time to generate final results.

3. The computer-implemented method of claim 2, further including:

storing the final results and discarding the intermediate results.

4. The computer-implemented method of claim 1, wherein executing the query from time to time includes:

executing the query on (i) one or more first data items in the stream of near real-time data, with the one or more first data items being located in a first portion of the stream, and (ii) one or more second data items in the stream of near real-time data, with the one or more second data items being located in a second portion of the stream.

5. The computer-implemented method of claim 1, wherein executing the query includes:

periodically executing a dataflow graph that represents the query, with the dataflow graph including executable computer code to implement the query, and with the dataflow graph receiving as input query specifications for the query.

6. The computer-implemented method of claim 5, wherein the dataflow graph includes components that represent operations to be performed in execution of the first query, and wherein the method further includes:

for a component;

performing a checkpoint operation that saves a local state of the component to enable recoverability of a state of the dataflow graph.

7. The computer-implemented method of claim 1, wherein the query is executed on data items that appear in the stream of near real-time data during a period of time the end of which is unknown at a start of executing the query.

8. The computer-implemented method of claim 1, wherein an amount of data items in the stream on which the query is executed is unknown at a start of executing the query.

9. The computer-implemented method of claim 7, further including:

accessing information indicative of user-defined custom operations for data transformation on the aggregated results;

executing the user-defined custom operations on the aggregated results; and

transforming the aggregated results in accordance with the user-defined custom operations.

10. The computer-implemented method of claim 1, further including:

generating, based on the aggregated results, a near real-time alert to alert a user of detection of a pre-defined condition.

11. The computer-implemented method of claim 1, wherein the stream of near real- time data includes a data stream in which data items are (i) periodically received at different times, or (ii) continuously received at different times.

12. The computer-implemented method of claim 1, further including receiving the stream of near real-time data from a data queue, a data repository, or a data feed.

13. The computer-implemented method of claim 1, wherein the query is a first query and wherein the method further includes:

selecting a second query for execution on two or more of the data items that appear at different locations in the stream; and

executing the first and second queries in near real-time with respect to the data items of the stream.

14. The computer-implemented method of claim 1, further including:

generating information for a user interface that when rendered on a display device includes;

input fields for input of information defining queries to be executed on the stream of near real-time data.

15. The computer-implemented method of claim 1, further including:

accessing, in a control repository, pre-defined queries that are candidates for execution on the stream of near real-time data.

16. A system for executing a query on data items located at different places in a stream of near real-time data to provide near-real time intermediate results for the query, as the query is being executed, the system including:

one or more processing devices; and

one or more machine-readable hardware storage devices storing instructions that are executable by the one or more processing devices to perform operations including;

from time to time,

executing, by one or more computer systems, the query on two or more of the data items located at different places in the stream, with the two or more data items being accessed in near real-time with respect to each of the two or more data items;

generating information indicative of results of executing the query; and

as the query continues being executed,

generating intermediate results of query execution by aggregating the results with prior results of executing the query on data items that previously appeared in the stream of near real-time data; and

transmitting to one or more client devices the intermediate results of query execution, prior to completion of execution of the query.

17. The system of claim 16, wherein the operations further include:

at a subsequent point in time, aggregating the intermediate results with results for executing the query at the subsequent point in time to generate final results.

18. The system of claim 17, wherein the operations further include:

storing the final results and discarding the intermediate results.

19. The system of claim 16, wherein executing the query from time to time includes:

executing the query on (i) one or more first data items in the stream of near real-time data, with the one or more first data items being located in a first portion of the stream, and (ii) one or more second data items in the stream of near real-time data, with the one or more second data items being located in a second portion of the stream.

20. The system of claim 16, wherein executing the query includes:

periodically executing a dataflow graph that represents the query, with the dataflow graph including executable computer code to implement the query, and with the dataflow graph receiving as input query specifications for the query.

21. The system of claim 20, wherein the dataflow graph includes components that represent operations to be performed in execution of the first query, and wherein the operations further include:

for a component;

performing a checkpoint operation that saves a local state of the component to enable recoverability of a state of the dataflow graph.

22. The system of claim 16, wherein the query is executed on data items that appear in the stream of near real-time data during a period of time the end of which is unknown at a start of executing the query.

23. The system of claim 16, wherein an amount of data items in the stream on which the query is executed is unknown at a start of executing the query.

24. One or more machine-readable hardware storages for executing a query on data items located at different places in a stream of near real-time data to provide near-real time intermediate results for the query, as the query is being executed, the one or more machine-readable hardware storages storing instructions that are executable by one or more processing devices to perform operations including:

from time to time,

executing, by one or more computer systems, the query on two or more of the data items located at different places in the stream, with the two or more data items being accessed in near real-time with respect to each of the two or more data items;

generating information indicative of results of executing the query; and

as the query continues being executed,

generating intermediate results of query execution by aggregating the results with prior results of executing the query on data items that previously appeared in the stream of near real- time data; and

transmitting to one or more client devices the intermediate results of query execution, prior to completion of execution of the query.