Distributed processing of streaming data records

• US 8,738,649 B2
• Filed: 03/15/2013
• Issued: 05/27/2014
• Est. Priority Date: 05/22/2012
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: Guavus, Inc. (Thales SA)
• Inventors: Bawa, Jaskaran Singh, Bisht, Bijay Singh, Srivastava, Anand Vivek, Bhowmik, Sumanta Kumar, Saraf, Atul Kumar
• Primary Examiner(s): Singh, Amresh
• Application Number:
  US13/836,757
• Publication Number:
  US 20130318031A1
• Time in Patent Office: 438 Days
• Field of Search: 707/603, 707/770
• US Class Current: 707/770
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Representative embodiments of a distributed processing method of facilitating interactive analytics of streaming data records by receiving the data records at a plurality of distributed computational nodes, establishing and storing dimensions corresponding to attributes of the data records, parsing the streaming data records to identify matches to at least one of the dimensions and based thereon, reducing the number of data records to create a targeted subset of the data, re-distributing the targeted subsets of the streaming data records among the distributed computational nodes in accordance with the dimensions stored on the nodes, updating a database storing measures of the dimensions in accordance with the targeted subsets of the streaming data records, and using the database to respond to a query based on measures associated with one or more of the dimensions.

Claims

1. A method of providing interactive information related to streaming data records using an OLAP cube, the method comprising steps of:

(i) sending, to a client in response to a request from said client, information related to a first set of predetermined dimensions stored in the OLAP cube, the first set of predetermined dimensions being derived from information in a first subset of the streaming data records;

(ii) receiving a new request from the client, the new request comprising a second set of predetermined dimensions having at least one dimension not present in the first set of predetermined dimensions;

(iii) parsing information in the streaming data records to identify a second subset of information relevant to the second set of predetermined dimensions;

(iv) receiving only the second subset of the information in the streaming data records at a plurality of distributed computational nodes, each node comprising a processor and a storage element;

(v) converting, at each computational node, a portion of the subset of the information in the received streaming data records into key-value pairs;

(vi) parsing, at each computational node, the converted key-value pairs of the subset of the information in the received streaming data records received at each said computational node to (i) identify matches of the keys to at least one predetermined dimension of the second set of predetermined dimensions and (ii) based thereon, combine the key-value pairs having identical keys;

(vii) re-distributing the keys of the converted subset of the received streaming data records among the distributed computational nodes in accordance with the second set of predetermined dimensions stored on the computational nodes, wherein each distributed computational node receives the key corresponding to one of the second set of predetermined dimensions, thereby reducing a size of the portion of the subset of information in the received streaming data records received at each computational node;

(viii) updating the OLAP cube to delete measures associated with only the first set of predetermined dimensions and to store measures associated with the second set of predetermined dimensions by collecting data from the computational nodes; and

(ix) sending, to the client in response to a request from said client, information related to the second set of predetermined dimensions.

2. The method of claim 1, further comprising sending a pull request for the streaming data records or receiving a push notification associated with the streaming data records.

3. The method of claim 1, further comprising labeling the received streaming data records with a time period.

4. The method of claim 1, wherein the database comprises an OLAP cube or a plurality of redundant OLAP cubes.

5. The method of claim 1, wherein a cell in the OLAP cube comprises information derived from the received streaming data records.

6. The method of claim 1, wherein at least one predetermined dimension in the first set of predetermined dimensions is the same as one predetermined dimension in the second set of predetermined dimensions.

7. The method of claim 1, wherein no predetermined dimensions in the first set of predetermined dimensions are the same as any predetermined dimensions in the second set of predetermined dimensions.

8. The method of claim 1, wherein the client is a workstation.

9. The method of claim 1, wherein information related to the first set of predetermined dimensions is sent to the client within sub-second in response to the request from said client.

10. The method of claim 1, wherein information related to the second set of predetermined dimensions is sent to the client within a time period of five minutes to one hour in response to the request from said client.

11. The method of claim 1, wherein the streaming data records are telecommunication data records.

12. A system for providing interactive information related to streaming data records using an OLAP cube, the system comprising:

an interactive cube manipulator for (i) sending, to a client in response to a request from said client, information related to a first set of predetermined dimensions stored in the OLAP cube, the first set of predetermined dimensions being derived from information in a first subset of the streaming data records and (ii) receiving a new request from the client, the new request comprising a second set of predetermined dimensions having at least one dimension not present in the first set of predetermined dimensions;

a data reorganizer for;

parsing information in the streaming data records to identify a second subset of information relevant to the second set of predetermined dimensions;

converting, at each computational node, a portion of the subset of the information in the received streaming data records into key-value pairs;

parsing, at each computational node, the converted key-value pairs of the subset of the information in the received streaming data records received at each said computational node to (i) identify matches of the keys to at least one predetermined dimension of the second set of predetermined dimensions and (ii) based thereon, combine the key-value pairs having identical keys; and

re-distributing the keys of the converted subset of the received streaming data records among the distributed computational nodes in accordance with the second set of predetermined dimensions stored on the computational nodes, wherein each distributed computational node receives the key corresponding to one of the second set of predetermined dimensions, thereby reducing a size of the portion of the subset of information in the received streaming data records received at each computational node;

a data collector for receiving only the second subset of the information in the streaming data records at a plurality of distributed computational nodes, each node comprising a processor and a storage element; and

a cube constructor for updating the OLAP cube to delete measures associated with only the first set of predetermined dimensions and to store measures associated with the second set of predetermined dimensions by collecting data from the computational nodes, wherein

the interactive cube manipulator is further configured for sending, to the client in response to a request from said client, information related to the second set of predetermined dimensions.

13. The system of claim 12, wherein a key in the key-value pairs comprises OLAP values used to identify the value in the key-value pair.

14. The system of claim 12, wherein the interactive cube manipulator is configured to send information related to the first set of predetermined dimensions to the client within one second in response to the request from said client.

15. The system of claim 12, wherein the interactive cube manipulator is configured to send information related to the second set of predetermined dimensions to the client within a time period of five minutes to one hour in response to the request from said client.

16. The system of claim 12, wherein user interface is configured to send information related to the second set of predetermined dimensions to the client within a time period of five minutes to one hour in response to the request from said client.

17. A system for providing interactive information related to streaming data records using an OLAP, the system comprising:

a user interface for (i) sending, to a client in response to a request from said client, information related to a first set of predetermined dimensions stored in the OLAP cube, the first set of predetermined dimensions being derived from information in a first subset of the streaming data records and (ii) receiving a new request from the client, the new request comprising a second set of predetermined dimensions having at least one dimension not present in the first set of predetermined dimensions;

a processor for;

parsing information in the streaming data records to identify a second subset of information relevant to the second set of predetermined dimensions;

converting, at each computational node, a portion of the subset of the information in the received streaming data records into key-value pairs;

parsing, at each computational node, the converted key-value pairs of the subset of the information in the received streaming data records received at each said computational node to (i) identify matches of the keys to at least one predetermined dimension of the second set of predetermined dimensions and (ii) based thereon, combine the key-value pairs having identical keys; and

re-distributing the keys of the converted subset of the received streaming data records among the distributed computational nodes in accordance with the second set of predetermined dimensions stored on the computational nodes, wherein each distributed computational node receives the key corresponding to one of the second set of predetermined dimensions, thereby reducing a size of the portion of the subset of information in the received streaming data records received at each computational node;

a plurality of computing nodes, each node comprising a processor and a local storage device, wherein each node receives only the second subset of the information in the streaming data records; and

a network connecting the plurality of computing nodes, the network distributing and re-distributing the data records amongst the plurality of computing nodes, wherein

the processor is further configured to update the OLAP cube to delete measures associated with only the first set of predetermined dimensions and to store measures associated with the second set of predetermined dimensions by collecting data from the computational nodes; and

the user interface is further configured to send, to the client in response to a request from said client, information related to the second set of predetermined dimensions.

18. The system of claim 17, wherein the plurality of computing nodes comprises a Hadoop cluster.

19. The system of claim 17, wherein the user interface is configured to send information related to the first set of predetermined dimensions to the client within one second in response to the request from said client.