Device for processing large-scale data streams

• US 10,657,155 B2
• Filed: 12/21/2017
• Issued: 05/19/2020
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Bull Sas (Atos SE)
• Original Assignee: Atos Integration
• Inventors: Kepeklian, Gabriel
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US15/850,060
• Publication Number:
  US 20180218061A1
• Time in Patent Office: 880 Days
• Field of Search: 707600-899
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/81 Indexing, e.g. XML tags; Data structures therefor; Storage structures

Abstract

The present invention relates to a device for processing large-scale data streams (big data), comprising a knowledge base, a hardware and software assembly constituting a front-end communication device for capturing the streams of the external medium and if needed restoring data to this medium, the front-end delivering the streams to a platform where it undergoes different processings, traces are collected and stored in a storage and memory architecture during execution of processing operations, the platform producing data which supply a decision-making device comprising a hardware and software assembly defining decision rules for either triggering actions or initiating retroactions to the front-end communication device or to the knowledge base of said processing device.

Claims

1. A device for processing largescale data streams, comprising a knowledge base, a hardware and software assembly constituting a front-end communication device, for capturing the streams of the external medium and if needed restoring data to this medium, the front-end delivering the streams to a platform where said data streams undergo different processings, traces being collected and stored in a storage and memory architecture during execution of the processing operations, the platform produces data which supply a decision-making device comprising a hardware and software assembly defining decision rules for either triggering actions or initiating retroactions to the front-end communication device or to the knowledge base of said processing device,

wherein a SPARQL request language is extended to perform summaries for data stream or implement oblivion functions for archiving data streams with a variable granularity as a function of time and ensuring limited storage space,

wherein the SPARQL language is extended by introducing the notion of adaptable time window and operators specific to the stream processing, the requests processing data being adapted to the arrival rate of dynamic data and being evaluated continually to consider the scalable character of the stream, the semantics of SPARQL requests enabling processings based on time or the order of arrival of data.

2. The device according to claim 1, wherein the different processings comprise semantisation of the stream, or creation of a summary, or crossing of at least two streams or interconnection of several streams.

3. The device according to claim 2, wherein semantisation of streams conforms to an ontology of description of streams.

4. The device according to claim 2, wherein the interconnection of streams can include semantic reasoning.

5. The device according to claim 1, wherein the platform comprises a plurality of “

smart′

op”

which are processing stream processes, these processes being scripted via scripting means to produce data supplying the decision-making device.

6. The device according to claim 1, wherein storing traces in the storage architecture lets the platform access these traces and perform repetitions of any sets or non-real-time processings.

7. The device according to claim 1, wherein all operations to which the streams are subjected contribute to the production of data.

8. The device according to claim 1, wherein the data can be alerts or singularities resulting from reasoning conducted on the streams.

9. The device according to claim 1, wherein the knowledge base contains information representing the knowledge of the external medium and comprises the description of sensors and infrastructure which supports said sensors.

10. The device according to claim 9, wherein this information constituting the knowledge can be modified by a retroaction triggered by decision rules.

11. The device according to claim 1, wherein the platform has a modular architecture for supporting the addition or replacement of components without recompilation, and enables deployment on a park of trivialized virtual machines, said platform being modularized by means of abstraction layers for encapsulating all access to services of the operating, storage or exchange system.

12. The device according to claim 11, wherein replacing implementations of said abstractions passes from simple implementations adapted to development to others focused on large-scale deployment.

13. The device according to claim 1, wherein the architecture of the platform adapts deployment of the system to the target volumetry and has it evolved in case of evolution, by simple addition of virtual machines if the selected components allow this or by replacing the components limiting scalability.

14. The device according to claim 1, wherein the extended SPARQL language involves data which can be either static or dynamic and temporal by interconnecting them, irrespective of their number, their sources or their qualities.