SYSTEM AND METHOD FOR CONTINUOUS ANALYTICS RUN AGAINST A COMBINATION OF STATIC AND REAL-TIME DATA

• US 20140351233A1
• Filed: 05/23/2014
• Published: 11/27/2014
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: Software AG USA Incorporated (Software AG)
• Original Assignee: Software AG USA Incorporated (Software AG)
• Inventors: Crupi, John, Alur, Deepak, Thope, Karthic, Krishnamurthy, Raj
• Granted Patent: US 8,977,600 B2
• Time in Patent Office: Days
• US Class Current: 707/706
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system for continuous analytics comprises an in-memory storage and a processor. The processor receives a query that indicates an analytic function to be performed on a combination of static data and real-time data. The processor loads the in-memory storage from a first source external to the processor with data that was stored as the static data. At the same time that the in-memory storage is already loaded with the static data, the processor continuously receives ephemeral real-time data as it is being generated by a second source external to the processor. The processor runs the analytic function from the query against the in-memory storage loaded with the first data and the real-time data that is being continuously received, in combination, to produce a result of the analytic function. The result is stored and time-stamped in the in-memory storage as an analytic cube.

Claims

1. A system for continuous analytics run against a combination of static and real-time data, comprising:

an in-memory storage; and

a processor, the processor being operably connected to the in-memory storage, the processor being programmed to;

receive, in a query engine, from a client, a query that indicates an analytic function to be performed on a combination of static data and real-time data;

in response to receipt of the query that indicates the analytic function to be performed on the combination of static data and real-time data;

load the in-memory storage from a first source external to the processor with data which was stored as the static data;

at the same time that the in-memory storage is already loaded with the static data, continuously receive real-time data as it is being generated by a second source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to receipt of the query;

temporally correlate the real-time data with existing static data in the in-memory storage, and

continuously run the analytic function from the query against, in combination, both the real-time data which is being continuously received and the existing static data which is loaded in the in-memory storage and which is temporally correlated to the real-time data, to continuously produce a result of the analytic function.

2. The system of claim 1, wherein

the first source of the static data is from a different source than the second source of the real-time data,

the combination of static data and real-time data is referenced in the query as a single variable, and

the processor is further configured to determine a source of the data, the data being referenced in the single variable, and to process the data differently based on the source of the data.

3. The system of claim 1, wherein the static data has a data structure different from the real-time data.

4. The system of claim 1, wherein at least one of the static data and the real-time data is originated by the first or second sources respectively in a hierarchical format.

5. The system of claim 1, wherein the processor is further configured to:

analyze the real-time data and the existing static data which are temporally correlated, in a moving window of the real-time data, as streaming data; and

discard the real-time data which has been analyzed and temporally correlated to make room for the real-time data which continues to be received.

6. The system of claim 1, wherein the processor is further configured to store information about a running calculation being made by the analytic function, to re-use the information about the running calculation in continuously performing the analytic function with the real-time data which is newly received.

7. The system of claim 1,

wherein the combination of static and real-time data indicated in the query which is received and on which the analytic function is performed, indicates at least one of:

(1) a combination of different types of static data, (2) combinations of different types of real-time data, and (3) combinations of different types of static data and real-time data; and

wherein the processor is further configured to;

normalize the at least one of;

(1) the combination of different types of static data, (2) the combinations of different types of real-time data, and (3) the combinations of different types of static data and real-time data, into normalized data which is stored in the in-memory storage in the form of a tuple,

wherein the analytic function which is run is applied against the normalized data stored in the in-memory storage.

8. The system of claim 1, wherein the processor is further configured to

store the result of the analytic function in the in-memory storage as an analytic cube for use in subsequent queries together with a time stamp in association with the result of the analytic function, as part of the analytic cube;

and

in response to a subsequent analytic function for the combination of static data and real-time data having a time period which includes the time stamp, re-use the result of the analytic function from the in-memory storage based on the time stamp associated therewith, as a source of data for the subsequent analytic function.

9. A method for continuous analytics run against a combination of static and real-time data, comprising:

receiving, in a query engine, from a client, a query that indicates an analytic function to be performed on a combination of static data and real-time data;

in response to receiving the query that indicates the analytic function to be performed on the combination of static data and real-time data;

loading, by a processor, an in-memory storage from a first source external to the processor with data which was stored as the static data;

at the same time that the in-memory storage is already loaded with the static data, continuously receiving, by the processor, real-time data as it is being generated by a second source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to receipt of the query;

temporally correlating, by the processor, the real-time data with existing static data in the in-memory storage; and

continuously running, by the processor, the analytic function from the query against, in combination, both the real-time data which is being continuously received and the existing static data which is loaded in the in-memory storage and which is temporally correlated to the real-time data, to continuously produce a result of the analytic function.

10. The method of claim 9, wherein

the first source of the static data is from a different source than the second source of the real-time data,

the combination of static data and real-time data is referenced in the query as a single variable, and

further comprising determining, by the processor, a source of the data, the data being referenced in the single variable;

and processing, by the processor, the data differently based on the source of the data.

11. The method of claim 9, wherein the static data has a data structure different from the real-time data.

12. The method of claim 9, wherein at least one of the static data and the real-time data is originated by the first or second sources respectively in a hierarchical format.

13. The method of claim 9, further comprising

analyzing the real-time data and the existing static data which are temporally correlated, in a moving window of the real-time data, as streaming data;

and

discarding the real-time data which has been analyzed and temporally correlated to make room for the real-time data which continues to be received.

14. The method of claim 9, further comprising storing information about a running calculation being made by the analytic function, and re-using the information about the running calculation in continuously performing the analytic function with the real-time data which is newly received.

15. The method of claim 9,

wherein the combination of static and real-time data indicated in the query which is received and on which the analytic function is performed, indicates at least one of:

(1) a combination of different types of static data, (2) combinations of different types of real-time data, and (3) combinations of different types of static data and real-time data; and

further comprising;

normalizing, by the processor, the at least one of;

(1) the combination of different types of static data, (2) the combinations of different types of real-time data, and (3) the combinations of different types of static data and real-time data, into normalized data which is stored in the in-memory storage in the form of a tuple,

wherein the analytic function which is run is applied against the normalized data stored in the in-memory storage.

16. The method of claim 9, further comprising

storing the result of the analytic function in the in-memory storage as an analytic cube for use in subsequent queries together with a time stamp in association with the result of the analytic function, as part of the analytic cube;

and

in response to a subsequent analytic function for the combination of static data and real-time data having a time period which includes the time stamp, re-using the result of the analytic function from the in-memory storage based on the time stamp associated therewith, as a source of data for the subsequent analytic function.

17. A method for continuous analytics run against a combination of static and real-time data, comprising:

by a processor, in response to a command that indicates an analytic function to be performed on at least real-time data;

continuously receiving, by the processor, real-time data as it is being generated by a source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to the command;

continuously running, by the processor, the analytic function from the command against the at least real-time data which is being continuously received, to continuously produce a result of the analytic function;

storing, by the processor, the result of the analytic function in an in-memory storage as an analytic cube for use in subsequent calculations together with a time stamp in association with the result of the analytic function, as part of the analytic cube; and

in response to a subsequent calculation for the real-time data having a time period which includes the time stamp, re-using, by the processor, the result of the analytic function from the in-memory storage based on the time stamp associated therewith, together with the real-time data which is newly received, as a source of data for the subsequent calculation.

18. The method of claim 17, wherein

the command that indicates both the analytic function to be performed, and a combination of static data and real-time data on which to perform the analytic function;

a source of the static data is different from a source of the real-time data,

the combination of static data and real-time data is referenced in the command as a single variable, and

further comprising determining, by the processor, a source of the data being analyzed by the analytic function, the data being referenced in the single variable; and

processing, by the processor, the data differently based on the source of the data.

19. The method of claim 17, wherein the subsequent calculation is a running calculation being made by the analytic function, further comprising continuously performing the analytic function.

20. The method of claim 17, wherein the subsequent calculation is made in response to a subsequent command that indicates a second analytic function to be performed on the at least real-time data.