System and method for continuous analytics run against a combination of static and real-time data

• US 8,977,600 B2
• Filed: 05/23/2014
• Issued: 03/10/2015
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: Software AG USA Incorporated (Software AG)
• Original Assignee: Software AG USA Incorporated (Software AG)
• Inventors: Crupi, John, Krishnamurthy, Raj, Alur, Deepak, Thope, Karthic
• Primary Examiner(s): BURKE, JEFF A
• Application Number:
  US14/286,266
• Publication Number:
  US 20140351233A1
• Time in Patent Office: 291 Days
• Field of Search: 707/600, 707/607, 707/688
• US Class Current: 707/688
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A system for continuous analytics comprises an in-memory storage and a processor. The processor receives a query that indicates an analytic function to be performed on a combination of static data and real-time data. The processor loads the in-memory storage from a first source external to the processor with data that was stored as the static data. At the same time that the in-memory storage is already loaded with the static data, the processor continuously receives ephemeral real-time data as it is being generated by a second source external to the processor. The processor runs the analytic function from the query against the in-memory storage loaded with the first data and the real-time data that is being continuously received, in combination, to produce a result of the analytic function. The result is stored and time-stamped in the in-memory storage as an analytic cube.

Claims

1. A system for continuous analytics run against a combination of static and real-time data, comprising:

an in-memory storage; and

a processor, the processor being operably connected to the in-memory storage, the processor being programmed to;

receive, in a query engine, from a client, a query that indicates an analytic function to be performed on a combination of static data and real-time data;

in response to receipt of the query that indicates the analytic function to be performed on the combination of static data and real-time data;

load the in-memory storage from a first source external to the processor with data which was stored as the static data;

at the same time that the in-memory storage is already loaded with the static data, continuously receive real-time data as it is being generated by a second source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to receipt of the query;

temporally correlate the real-time data with existing static data in the in-memory storage, and

continuously run the analytic function from the query against, in combination, both the real-time data which is being continuously received and the existing static data which is loaded in the in-memory storage and which is temporally correlated to the real-time data, to continuously produce a result of the analytic function, wherein

the first source of the static data is from a different source than the second source of the real-time data,

the combination of each of the static data and the real-time data is referenced in the query as a single variable, and

the processor is further configured to determine a source of the data, the data being referenced in the single variable, and to process the data differently based on the source of the data;

wherein the combination of static and real-time data indicated in the query which is received and on which the analytic function is performed, indicates at least one of;

(1) a combination of different types of static data, (2) combinations of different types of real-time data, and (3) combinations of different types of static data and real-time data;

wherein the processor is further configured to;

normalize the at least one of;

(1) the combination of different types of static data, (2) the combinations of different types of real-time data, and (3) the combinations of different types of static data and real-time data, into normalized data which is stored in the in-memory storage in the form of a tuple; and

wherein the analytic function which is run is applied against the normalized data stored in the in-memory storage.

2. The system of claim 1, wherein the static data has a data structure different from the real-time data.

3. The system of claim 1, wherein at least one of the static data and the real-time data is originated by the first or second sources respectively in a hierarchical format.

4. The system of claim 1, wherein the processor is further configured to:

analyze the real-time data and the existing static data which are temporally correlated, in a moving window of the real-time data, as streaming data; and

discard the real-time data which has been analyzed and temporally correlated to make room for the real-time data which continues to be received.

5. The system of claim 1, wherein the processor is further configured to store information about a running calculation being made by the analytic function, to re-use the information about the running calculation in continuously performing the analytic function with the real-time data which is newly received.

6. The system of claim 1, wherein the processor is further configured to

store the result of the analytic function in the in-memory storage as an analytic cube for use in subsequent queries together with a time stamp in association with the result of the analytic function, as part of the analytic cube;

and

in response to a subsequent analytic function for the combination of static data and real-time data having a time period which includes the time stamp, re-use the result of the analytic function from the in-memory storage based on the time stamp associated therewith, as a source of data for the subsequent analytic function.

7. A method for continuous analytics run against a combination of static and real-time data, comprising:

receiving, in a query engine, from a client, a query that indicates an analytic function to be performed on a combination of static data and real-time data;

in response to receiving the query that indicates the analytic function to be performed on the combination of static data and real-time data;

loading, by a processor, an in-memory storage from a first source external to the processor with data which was stored as the static data;

at the same time that the in-memory storage is already loaded with the static data, continuously receiving, by the processor, real-time data as it is being generated by a second source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to receipt of the query;

temporally correlating, by the processor, the real-time data with existing static data in the in-memory storage; and

continuously running, by the processor, the analytic function from the query against, in combination, both the real-time data which is being continuously received and the existing static data which is loaded in the in-memory storage and which is temporally correlated to the real-time data, to continuously produce a result of the analytic function, wherein

the first source of the static data is from a different source than the second source of the real-time data,

the combination of each of the static data and the real-time data is referenced in the query as a single variable, and

further comprising determining, by the processor, a source of the data, the data being referenced in the single variable; and

processing, by the processor, the data differently based on the source of the data;

wherein the combination of static and real-time data indicated in the query which is received and on which the analytic function is performed, indicates at least one of;

(1) a combination of different types of static data, (2) combinations of different types of real-time data, and (3) combinations of different types of static data and real-time data; and

further comprising;

normalizing, by the processor, the at least one of;

(1) the combination of different types of static data, (2) the combinations of different types of real-time data, and (3) the combinations of different types of static data and real-time data, into normalized data which is stored in the in-memory storage in the form of a tuple; and

wherein the analytic function which is run is applied against the normalized data stored in the in-memory storage.

8. The method of claim 7, wherein the static data has a data structure different from the real-time data.

9. The method of claim 7, wherein at least one of the static data and the real-time data is originated by the first or second sources respectively in a hierarchical format.

10. The method of claim 7, further comprising

analyzing the real-time data and the existing static data which are temporally correlated, in a moving window of the real-time data, as streaming data;

and

discarding the real-time data which has been analyzed and temporally correlated to make room for the real-time data which continues to be received.

11. The method of claim 7, further comprising storing information about a running calculation being made by the analytic function, and re-using the information about the running calculation in continuously performing the analytic function with the real-time data which is newly received.

12. The method of claim 7, further comprising

storing the result of the analytic function in the in-memory storage as an analytic cube for use in subsequent queries together with a time stamp in association with the result of the analytic function, as part of the analytic cube;

and

in response to a subsequent analytic function for the combination of static data and real-time data having a time period which includes the time stamp, re-using the result of the analytic function from the in-memory storage based on the time stamp associated therewith, as a source of data for the subsequent analytic function.

13. A method for continuous analytics run against a combination of static and real-time data, comprising:

by a processor, in response to a command that indicates an analytic function to be performed on at least real-time data;

continuously receiving, by the processor, real-time data as it is being generated by a source external to the processor, the real-time data being ephemeral;

in response to the real-time data which is continuously received subsequent to the command;

continuously running, by the processor, the analytic function from the command against the at least real-time data which is being continuously received, to continuously produce a result of the analytic function;

storing, by the processor, the result of the analytic function in an in-memory storage as an analytic cube for use in subsequent calculations together with a time stamp in association with the result of the analytic function, as part of the analytic cube; and

in response to a subsequent calculation for the real-time data having a time period which includes the time stamp, re-using, by the processor, the result of the analytic function from the in-memory storage based on the time stamp associated therewith, together with the real-time data which is newly received, as a source of data for the subsequent calculation, wherein

the command indicates both the analytic function to be performed, and a combination of static data and real-time data on which to perform the analytic function,

a source of the static data is different from a source of the real-time data,

the combination of each of the static data and the real-time data is referenced in the command as a single variable, and

further comprising determining, by the processor, a source of the data being analyzed by the analytic function, the data being referenced in the single variable; and

processing, by the processor, the data differently based on the source of the data;

wherein the combination of static and real-time data indicated in the query which is received and on which the analytic function is performed, indicates at least one of;

(1) a combination of different types of static data, (2) combinations of different types of real-time data, and (3) combinations of different types of static data and real-time data;

further comprising, by the processor, normalizing the at least one of;

(1) the combination of different types of static data, (2) the combinations of different types of real-time data, and (3) the combinations of different types of static data and real-time data, into normalized data which is stored in the in-memory storage in the form of a tuple; and

wherein the analytic function which is run is applied against the normalized data stored in the in-memory storage.

14. The method of claim 13, wherein the subsequent calculation is a running calculation being made by the analytic function, further comprising continuously performing the analytic function.

15. The method of claim 13, wherein the subsequent calculation is made in response to a subsequent command that indicates a second analytic function to be performed on the at least real-time data.